rarias/jungle
rarias/jungle
2
0
Fork 3
Code Issues 53 Pull Requests 10 Actions Packages Projects Releases Wiki Activity

tent: BUG: unable to handle page fault for address: ffffffffecb277fd #117

New Issue
Open
opened 2025-06-17 12:52:36 +02:00 by rarias · 2 comments
rarias commented 2025-06-17 12:52:36 +02:00
Owner
Copy Link

This failure to retrieve a page caused a hard lockup:

tent% sudo journalctl -b -1 -k -n 300
Jun 16 10:09:26 tent kernel: docker0: port 1(vetha33cb68) entered disabled state
Jun 16 10:09:26 tent kernel: vetha33cb68 (unregistering): left allmulticast mode
Jun 16 10:09:26 tent kernel: vetha33cb68 (unregistering): left promiscuous mode
Jun 16 10:09:26 tent kernel: docker0: port 1(vetha33cb68) entered disabled state
Jun 16 13:07:46 tent kernel: BUG: unable to handle page fault for address: ffffffffecb277fd
Jun 16 13:07:46 tent kernel: #PF: supervisor instruction fetch in kernel mode
Jun 16 13:07:46 tent kernel: #PF: error_code(0x0010) - not-present page
Jun 16 13:07:46 tent kernel: PGD 802427067 P4D 802427067 PUD 802429067 PMD 0 
Jun 16 13:07:46 tent kernel: Oops: Oops: 0010 [#1] PREEMPT SMP PTI
Jun 16 13:07:46 tent kernel: CPU: 54 UID: 30012 PID: 909169 Comm: clang++ Not tainted 6.12.9 #1-NixOS
Jun 16 13:07:46 tent kernel: Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0016.033120161139 03/31/2016
Jun 16 13:07:46 tent kernel: RIP: 0010:update_load_avg+0x348/0x7f0
Jun 16 13:07:46 tent kernel: Code: 89 86 88 00 00 00 0f 1f 44 00 00 0f 1f 44 00 00 48 83 bd c0 00 00 00 00 75 0a 41 f6 c5 04 0f 85 19 03 00 00 41 f6 c5 08 75 33 <48> 8b ab 38 01 00 00 48 8d 85 00 01 00 00 48 39 c3 0f 84 04 04 00
Jun 16 13:07:46 tent kernel: RSP: 0000:ffffa2fa87108e18 EFLAGS: 00010002
Jun 16 13:07:46 tent kernel: RAX: 0000000000000001 RBX: ffff9212bfd35900 RCX: 0000000000000000
Jun 16 13:07:46 tent kernel: RDX: ffff92030aa64200 RSI: 0000000000000000 RDI: 0000000000000000
Jun 16 13:07:46 tent kernel: RBP: ffff92030aa67600 R08: 0000000000000000 R09: 0000000000000000
Jun 16 13:07:46 tent kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0003dba34a654f09
Jun 16 13:07:46 tent kernel: R13: 0000000000000001 R14: 0000000000000001 R15: ffff9212bfd25900
Jun 16 13:07:46 tent kernel: FS:  00007ffff37a0780(0000) GS:ffff9212bfd00000(0000) knlGS:0000000000000000
Jun 16 13:07:46 tent kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 16 13:07:46 tent kernel: CR2: ffffffffecb277fd CR3: 00000003db266001 CR4: 00000000003706f0
Jun 16 13:07:46 tent kernel: Call Trace:
Jun 16 13:07:46 tent kernel:  <IRQ>
Jun 16 13:07:46 tent kernel:  ? __die+0x23/0x80
Jun 16 13:07:46 tent kernel:  ? page_fault_oops+0x173/0x5b0
Jun 16 13:07:46 tent kernel:  ? exc_page_fault+0x155/0x160
Jun 16 13:07:46 tent kernel:  ? asm_exc_page_fault+0x26/0x30
Jun 16 13:07:46 tent kernel:  ? update_load_avg+0x348/0x7f0
Jun 16 13:07:46 tent kernel:  ? update_load_avg+0x7e/0x7f0
Jun 16 13:07:46 tent kernel:  ? update_curr+0x98/0x250
Jun 16 13:07:46 tent kernel:  task_tick_fair+0x6b/0x4e0
Jun 16 13:07:46 tent kernel:  sched_tick+0xb0/0x2d0
Jun 16 13:07:46 tent kernel:  update_process_times+0x96/0xb0
Jun 16 13:07:46 tent kernel:  tick_nohz_handler+0x8f/0x150
Jun 16 13:07:46 tent kernel:  ? __pfx_tick_nohz_handler+0x10/0x10
Jun 16 13:07:46 tent kernel:  __hrtimer_run_queues+0x112/0x2b0
Jun 16 13:07:46 tent kernel:  hrtimer_interrupt+0xfa/0x250
Jun 16 13:07:46 tent kernel:  __sysvec_apic_timer_interrupt+0x58/0x120
Jun 16 13:07:46 tent kernel:  sysvec_apic_timer_interrupt+0x6e/0x80
Jun 16 13:07:46 tent kernel:  </IRQ>
Jun 16 13:07:46 tent kernel:  <TASK>
Jun 16 13:07:46 tent kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
Jun 16 13:07:46 tent kernel: RIP: 0010:vm_normal_folio+0x17/0x80
Jun 16 13:07:46 tent kernel: Code: 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 e8 02 ff ff ff 48 85 c0 74 1f 48 8b 50 08 <f6> c2 01 75 47 66 90 31 d2 31 c9 31 f6 31 ff c3 cc cc cc cc a9 ff
Jun 16 13:07:46 tent kernel: RSP: 0000:ffffa2faa7fe3bd8 EFLAGS: 00000286
Jun 16 13:07:46 tent kernel: RAX: ffffebaea68b8ac0 RBX: 000000000b23a000 RCX: 0000000000000000
Jun 16 13:07:46 tent kernel: RDX: ffffebaeb1c25788 RSI: 0000000000000000 RDI: 0000000000000000
Jun 16 13:07:46 tent kernel: RBP: ffff9203e55581d0 R08: 0000000000000000 R09: 0000000000000000
Jun 16 13:07:46 tent kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffebaeb1c25780
Jun 16 13:07:46 tent kernel: R13: ffff91f53f7e69a0 R14: 000000000b400000 R15: ffffa2faa7fe3d88
Jun 16 13:07:46 tent kernel:  ? vm_normal_folio+0xe/0x80
Jun 16 13:07:46 tent kernel:  change_pte_range+0x12b/0x900
Jun 16 13:07:46 tent kernel:  change_protection+0x716/0xbc0
Jun 16 13:07:46 tent kernel:  change_prot_numa+0x64/0x110
Jun 16 13:07:46 tent kernel:  task_numa_work+0x3c1/0x960
Jun 16 13:07:46 tent kernel:  ? __note_gp_changes+0x221/0x280
Jun 16 13:07:46 tent kernel:  task_work_run+0x5c/0x90
Jun 16 13:07:46 tent kernel:  irqentry_exit_to_user_mode+0x221/0x230
Jun 16 13:07:46 tent kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
Jun 16 13:07:46 tent kernel: RIP: 0033:0x7ffff4c1e73c
Jun 16 13:07:46 tent kernel: Code: ec 18 64 48 8b 04 25 28 00 00 00 48 89 44 24 08 48 8b 07 48 85 c0 74 76 48 89 c2 83 e2 06 75 36 48 83 e0 f8 74 68 0f b6 50 1c <83> e2 7f 83 ea 32 83 fa 01 77 04 48 8b 40 40 48 8b 54 24 08 64 48
Jun 16 13:07:46 tent kernel: RSP: 002b:00007ffffffe6530 EFLAGS: 00000206
Jun 16 13:07:46 tent kernel: RAX: 00000000005d3928 RBX: 00007ffffffe7460 RCX: 000000000002328f
Jun 16 13:07:46 tent kernel: RDX: 0000000000000045 RSI: 00007ffffffe68e0 RDI: 00007ffffffe6550
Jun 16 13:07:46 tent kernel: RBP: 000000000002328f R08: 0000000000000000 R09: 0000000000000001
Jun 16 13:07:46 tent kernel: R10: 00000000005eb2a8 R11: 0000000000000000 R12: 00007ffffffe68e0
Jun 16 13:07:46 tent kernel: R13: 0000000000000000 R14: 0000000000000001 R15: 00007ffffffe65a8
Jun 16 13:07:46 tent kernel:  </TASK>
Jun 16 13:07:46 tent kernel: Modules linked in: nft_chain_nat xt_MASQUERADE nf_conntrack_netlink xfrm_user xt_addrtype overlay xt_nat nf_nat br_netfilter veth tls cmac algif_hash bluetooth rfkill ecdh_generic ecc qrtr tcp_diag inet_diag nhpoly1305_avx2 nhpoly1305_sse2 nhpoly1305 chacha_generic chacha_x86_64 libchacha adiantum libpoly1305 camellia_generic camellia_aesni_avx2 camellia_aesni_avx_x86_64 camellia_x86_64 cast5_avx_x86_64 cast5_generic cast_common des_generic des3_ede_x86_64 libdes blowfish_generic blowfish_x86_64 blowfish_common cbc serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic xts twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common lrw algif_skcipher af_alg msr sb_edac edac_core intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel gf128mul hfi1 crypto_simd
Jun 16 13:07:46 tent kernel:  cryptd ixgbe rdmavt xfrm_algo mdio_devres ib_uverbs joydev iTCO_wdt libphy intel_pmc_bxt watchdog rapl intel_cstate mxm_wmi evdev ptp ib_core intel_uncore ipmi_si mgag200 mei_me pps_core i2c_i801 i2c_algo_bit mdio mei lpc_ich i2c_mux i2c_smbus ioatdma dca input_leds mousedev led_class mac_hid wmi acpi_power_meter tiny_power_button acpi_ipmi acpi_pad button xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipt_rpfilter xt_pkttype xt_LOG nf_log_syslog xt_tcpudp nft_compat nf_tables sch_fq_codel libcrc32c atkbd libps2 serio vivaldi_fmap loop cpufreq_powersave tun tap macvlan bridge stp llc kvm ipmi_watchdog ipmi_devintf ipmi_msghandler fuse efi_pstore configfs nfnetlink dmi_sysfs ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache jbd2 raid1 md_mod hid_generic sd_mod usbhid hid ahci libahci libata xhci_pci xhci_hcd crc32c_intel scsi_mod ehci_pci ehci_hcd scsi_common rtc_cmos dm_mod dax
Jun 16 13:07:46 tent kernel: CR2: ffffffffecb277fd
Jun 16 13:07:46 tent kernel: ---[ end trace 0000000000000000 ]---
Jun 16 13:07:46 tent kernel: ixgbe 0000:03:00.0 eno1: NETDEV WATCHDOG: CPU: 1: transmit queue 10 timed out 5085 ms
Jun 16 13:07:46 tent kernel: watchdog: Watchdog detected hard LOCKUP on cpu 39
Jun 16 13:07:46 tent kernel: Modules linked in: nft_chain_nat xt_MASQUERADE nf_conntrack_netlink xfrm_user xt_addrtype overlay xt_nat nf_nat br_netfilter veth tls cmac algif_hash bluetooth rfkill ecdh_generic ecc qrtr tcp_diag inet_diag nhpoly1305_avx2 nhpoly1305_sse2 nhpoly1305 chacha_generic chacha_x86_64 libchacha adiantum libpoly1305 camellia_generic camellia_aesni_avx2 camellia_aesni_avx_x86_64 camellia_x86_64 cast5_avx_x86_64 cast5_generic cast_common des_generic des3_ede_x86_64 libdes blowfish_generic blowfish_x86_64 blowfish_common cbc serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic xts twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common lrw algif_skcipher af_alg msr sb_edac edac_core intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel gf128mul hfi1 crypto_simd
Jun 16 13:07:46 tent kernel:  cryptd ixgbe rdmavt xfrm_algo mdio_devres ib_uverbs joydev iTCO_wdt libphy intel_pmc_bxt watchdog rapl intel_cstate mxm_wmi evdev ptp ib_core intel_uncore ipmi_si mgag200 mei_me pps_core i2c_i801 i2c_algo_bit mdio mei lpc_ich i2c_mux i2c_smbus ioatdma dca input_leds mousedev led_class mac_hid wmi acpi_power_meter tiny_power_button acpi_ipmi acpi_pad button xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipt_rpfilter xt_pkttype xt_LOG nf_log_syslog xt_tcpudp nft_compat nf_tables sch_fq_codel libcrc32c atkbd libps2 serio vivaldi_fmap loop cpufreq_powersave tun tap macvlan bridge stp llc kvm ipmi_watchdog ipmi_devintf ipmi_msghandler fuse efi_pstore configfs nfnetlink dmi_sysfs ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache jbd2 raid1 md_mod hid_generic sd_mod usbhid hid ahci libahci libata xhci_pci xhci_hcd crc32c_intel scsi_mod ehci_pci ehci_hcd scsi_common rtc_cmos dm_mod dax
Jun 16 13:07:46 tent kernel: CPU: 39 UID: 30012 PID: 909795 Comm: make Tainted: G      D            6.12.9 #1-NixOS
Jun 16 13:07:46 tent kernel: Tainted: [D]=DIE
Jun 16 13:07:46 tent kernel: Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0016.033120161139 03/31/2016
Jun 16 13:07:46 tent kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x79/0x2c0
Jun 16 13:07:46 tent kernel: Code: 0f ba 2b 08 0f 92 c2 8b 03 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 77 64 85 c0 74 10 0f b6 03 84 c0 74 09 f3 90 0f b6 03 <84> c0 75 f7 b8 01 00 00 00 66 89 03 5b 5d 41 5c 41 5d 31 c0 31 d2
Jun 16 13:07:46 tent kernel: RSP: 0018:ffffa2fab0a2bce0 EFLAGS: 00000002
Jun 16 13:07:46 tent kernel: RAX: 0000000000000001 RBX: ffff9212bfd35800 RCX: 0000000000000000
Jun 16 13:07:46 tent kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9212bfd35800
Jun 16 13:07:46 tent kernel: RBP: ffff91f3e03d9280 R08: 0000000000000000 R09: 0000000000000000
Jun 16 13:07:46 tent kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000035800
Jun 16 13:07:46 tent kernel: R13: ffffa2fab0a2bd50 R14: 0000000000000084 R15: ffff91f39168b200
Jun 16 13:07:46 tent kernel: FS:  00007ffff7dc2740(0000) GS:ffff9202bfc80000(0000) knlGS:0000000000000000
Jun 16 13:07:46 tent kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 16 13:07:46 tent kernel: CR2: 000000000049a018 CR3: 00000010a9008004 CR4: 00000000003706f0
Jun 16 13:07:46 tent kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 16 13:07:46 tent kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jun 16 13:07:46 tent kernel: Call Trace:
Jun 16 13:07:46 tent kernel:  <NMI>
Jun 16 13:07:46 tent kernel:  ? watchdog_hardlockup_check+0x106/0x1f0
Jun 16 13:07:46 tent kernel:  ? __perf_event_overflow+0x111/0x320
Jun 16 13:07:46 tent kernel:  ? handle_pmi_common+0x16d/0x3d0
Jun 16 13:07:46 tent kernel:  ? set_pte_vaddr_p4d+0x4e/0x60
Jun 16 13:07:46 tent kernel:  ? flush_tlb_one_kernel+0xe/0x40
Jun 16 13:07:46 tent kernel:  ? ghes_copy_tofrom_phys+0x7f/0x120
Jun 16 13:07:46 tent kernel:  ? intel_pmu_handle_irq+0x10a/0x520
Jun 16 13:07:46 tent kernel:  ? ghes_notify_nmi+0x238/0x390
Jun 16 13:07:46 tent kernel:  ? perf_event_nmi_handler+0x2a/0x50
Jun 16 13:07:46 tent kernel:  ? nmi_handle+0x61/0x160
Jun 16 13:07:46 tent kernel:  ? default_do_nmi+0x43/0x100
Jun 16 13:07:46 tent kernel:  ? exc_nmi+0x138/0x1d0
Jun 16 13:07:46 tent kernel:  ? end_repeat_nmi+0xf/0x53
Jun 16 13:07:46 tent kernel:  ? native_queued_spin_lock_slowpath+0x79/0x2c0
Jun 16 13:07:46 tent kernel:  ? native_queued_spin_lock_slowpath+0x79/0x2c0
Jun 16 13:07:46 tent kernel:  ? native_queued_spin_lock_slowpath+0x79/0x2c0
Jun 16 13:07:46 tent kernel:  </NMI>
Jun 16 13:07:46 tent kernel:  <TASK>
Jun 16 13:07:46 tent kernel:  _raw_spin_lock+0x3f/0x60
Jun 16 13:07:46 tent kernel:  raw_spin_rq_lock_nested+0x1c/0x90
Jun 16 13:07:46 tent kernel:  __task_rq_lock+0x34/0xf0
Jun 16 13:07:46 tent kernel:  wake_up_new_task+0x160/0x320
Jun 16 13:07:46 tent kernel:  kernel_clone+0x2a4/0x430
Jun 16 13:07:46 tent kernel:  __do_sys_clone3+0xef/0x140
Jun 16 13:07:46 tent kernel:  do_syscall_64+0xb7/0x210
Jun 16 13:07:46 tent kernel:  entry_SYSCALL_64_after_hwframe+0x77/0x7f
Jun 16 13:07:46 tent kernel: RIP: 0033:0x7ffff7ed539d
Jun 16 13:07:46 tent kernel: Code: 31 f6 31 ff 45 31 d2 45 31 db c3 66 90 f3 0f 1e fa b8 ea ff ff ff 48 85 ff 74 28 48 85 d2 74 23 49 89 c8 b8 b3 01 00 00 0f 05 <48> 85 c0 7c 14 74 01 c3 31 ed 4c 89 c7 ff d2 48 89 c7 b8 3c 00 00
Jun 16 13:07:46 tent kernel: RSP: 002b:00007fffffff8838 EFLAGS: 00000206 ORIG_RAX: 00000000000001b3
Jun 16 13:07:46 tent kernel: RAX: ffffffffffffffda RBX: 00007ffff7db9000 RCX: 00007ffff7ed539d
Jun 16 13:07:46 tent kernel: RDX: 00007ffff7ebcc50 RSI: 0000000000000058 RDI: 00007fffffff8880
Jun 16 13:07:46 tent kernel: RBP: 0000000000009000 R08: 00007fffffff88e0 R09: 0000000000000000
Jun 16 13:07:46 tent kernel: R10: 0000000000000008 R11: 0000000000000206 R12: 00007fffffff8c10
Jun 16 13:07:46 tent kernel: R13: 00007fffffff88e0 R14: 0000000000000000 R15: 00007ffff7ebcc50
Jun 16 13:07:46 tent kernel:  </TASK>
Jun 16 13:07:46 tent kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Jun 16 13:07:46 tent kernel: rcu:         54-...0: (8 ticks this GP) idle=7684/1/0x4000000000000000 softirq=37445173/37445173 fqs=2100
Jun 16 13:07:46 tent kernel: rcu:         (detected by 3, t=21002 jiffies, g=64380521, q=74746 ncpus=56)
Jun 16 13:07:46 tent kernel: Sending NMI from CPU 3 to CPUs 54:
Jun 16 13:07:46 tent kernel: NMI backtrace for cpu 54
Jun 16 13:07:46 tent kernel: CPU: 54 UID: 30012 PID: 909169 Comm: clang++ Tainted: G      D            6.12.9 #1-NixOS
Jun 16 13:07:46 tent kernel: Tainted: [D]=DIE
Jun 16 13:07:46 tent kernel: Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0016.033120161139 03/31/2016
Jun 16 13:07:46 tent kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x28f/0x2c0
Jun 16 13:07:46 tent kernel: Code: 83 e0 03 83 e9 01 48 c1 e0 05 48 63 c9 48 05 40 69 03 00 48 03 04 cd 20 54 07 aa 48 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 <85> c0 74 f7 48 8b 0a 48 85 c9 74 81 0f 0d 09 e9 79 ff ff ff be 01
Jun 16 13:07:46 tent kernel: RSP: 0000:ffffa2fa87108890 EFLAGS: 00000046
Jun 16 13:07:46 tent kernel: RAX: 0000000000000000 RBX: ffff9212bfd35800 RCX: 0000000000000019
Jun 16 13:07:46 tent kernel: RDX: ffff9212bfd36940 RSI: 0000000000680101 RDI: ffff9212bfd35800
Jun 16 13:07:46 tent kernel: RBP: ffff9212bfd36940 R08: 0000000000000000 R09: 0000000000000000
Jun 16 13:07:46 tent kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000dc0000
Jun 16 13:07:46 tent kernel: R13: 0000000000dc0000 R14: 0000000000000008 R15: ffff92030ba2af24
Jun 16 13:07:46 tent kernel: FS:  00007ffff37a0780(0000) GS:ffff9212bfd00000(0000) knlGS:0000000000000000
Jun 16 13:07:46 tent kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 16 13:07:46 tent kernel: CR2: ffffffffecb277fd CR3: 00000003db266001 CR4: 00000000003706f0
Jun 16 13:07:46 tent kernel: Call Trace:
Jun 16 13:07:46 tent kernel:  <NMI>
Jun 16 13:07:46 tent kernel:  ? nmi_cpu_backtrace+0x9f/0x120
Jun 16 13:07:46 tent kernel:  ? nmi_cpu_backtrace_handler+0x11/0x20
Jun 16 13:07:46 tent kernel:  ? nmi_handle+0x61/0x160
Jun 16 13:07:46 tent kernel:  ? default_do_nmi+0x43/0x100
Jun 16 13:07:46 tent kernel:  ? exc_nmi+0x138/0x1d0
Jun 16 13:07:46 tent kernel:  ? end_repeat_nmi+0xf/0x53
Jun 16 13:07:46 tent kernel:  ? native_queued_spin_lock_slowpath+0x28f/0x2c0
Jun 16 13:07:46 tent kernel:  ? native_queued_spin_lock_slowpath+0x28f/0x2c0
Jun 16 13:07:46 tent kernel:  ? native_queued_spin_lock_slowpath+0x28f/0x2c0
Jun 16 13:07:46 tent kernel:  </NMI>
Jun 16 13:07:46 tent kernel:  <IRQ>
Jun 16 13:07:46 tent kernel:  _raw_spin_lock+0x3f/0x60
Jun 16 13:07:46 tent kernel:  raw_spin_rq_lock_nested+0x1c/0x90
Jun 16 13:07:46 tent kernel:  try_to_wake_up+0x218/0x6c0
Jun 16 13:07:46 tent kernel:  kick_pool+0x66/0x160
Jun 16 13:07:46 tent kernel:  __queue_work+0x2df/0x4e0
Jun 16 13:07:46 tent kernel:  queue_work_on+0x6b/0x80
Jun 16 13:07:46 tent kernel:  soft_cursor+0x1a0/0x250
Jun 16 13:07:46 tent kernel:  ? info_print_prefix+0xb3/0xe0
Jun 16 13:07:46 tent kernel:  bit_cursor+0x383/0x600
Jun 16 13:07:46 tent kernel:  hide_cursor+0x2b/0xb0
Jun 16 13:07:46 tent kernel:  vt_console_print+0x44b/0x460
Jun 16 13:07:46 tent kernel:  console_flush_all+0x291/0x490
Jun 16 13:07:46 tent kernel:  console_unlock+0x73/0x130
Jun 16 13:07:46 tent kernel:  vprintk_emit+0x174/0x2c0
Jun 16 13:07:46 tent kernel:  _printk+0x64/0x90
Jun 16 13:07:46 tent kernel:  oops_exit+0x26/0x40
Jun 16 13:07:46 tent kernel:  oops_end+0x4c/0xc0
Jun 16 13:07:46 tent kernel:  page_fault_oops+0x197/0x5b0
Jun 16 13:07:46 tent kernel:  exc_page_fault+0x155/0x160
Jun 16 13:07:46 tent kernel:  asm_exc_page_fault+0x26/0x30
Jun 16 13:07:46 tent kernel: RIP: 0010:update_load_avg+0x348/0x7f0
Jun 16 13:07:46 tent kernel: Code: 89 86 88 00 00 00 0f 1f 44 00 00 0f 1f 44 00 00 48 83 bd c0 00 00 00 00 75 0a 41 f6 c5 04 0f 85 19 03 00 00 41 f6 c5 08 75 33 <48> 8b ab 38 01 00 00 48 8d 85 00 01 00 00 48 39 c3 0f 84 04 04 00
Jun 16 13:07:46 tent kernel: RSP: 0000:ffffa2fa87108e18 EFLAGS: 00010002
Jun 16 13:07:46 tent kernel: RAX: 0000000000000001 RBX: ffff9212bfd35900 RCX: 0000000000000000
Jun 16 13:07:46 tent kernel: RDX: ffff92030aa64200 RSI: 0000000000000000 RDI: 0000000000000000
Jun 16 13:07:46 tent kernel: RBP: ffff92030aa67600 R08: 0000000000000000 R09: 0000000000000000
Jun 16 13:07:46 tent kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0003dba34a654f09
Jun 16 13:07:46 tent kernel: R13: 0000000000000001 R14: 0000000000000001 R15: ffff9212bfd25900
Jun 16 13:07:46 tent kernel:  ? update_load_avg+0x7e/0x7f0
Jun 16 13:07:46 tent kernel:  ? update_curr+0x98/0x250
Jun 16 13:07:46 tent kernel:  task_tick_fair+0x6b/0x4e0
Jun 16 13:07:46 tent kernel:  sched_tick+0xb0/0x2d0
Jun 16 13:07:46 tent kernel:  update_process_times+0x96/0xb0
Jun 16 13:07:46 tent kernel:  tick_nohz_handler+0x8f/0x150
Jun 16 13:07:46 tent kernel:  ? __pfx_tick_nohz_handler+0x10/0x10
Jun 16 13:07:46 tent kernel:  __hrtimer_run_queues+0x112/0x2b0
Jun 16 13:07:46 tent kernel:  hrtimer_interrupt+0xfa/0x250
Jun 16 13:07:46 tent kernel:  __sysvec_apic_timer_interrupt+0x58/0x120
Jun 16 13:07:46 tent kernel:  sysvec_apic_timer_interrupt+0x6e/0x80
Jun 16 13:07:46 tent kernel:  </IRQ>
Jun 16 13:07:46 tent kernel:  <TASK>
Jun 16 13:07:46 tent kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
Jun 16 13:07:46 tent kernel: RIP: 0010:vm_normal_folio+0x17/0x80
Jun 16 13:07:46 tent kernel: Code: 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 e8 02 ff ff ff 48 85 c0 74 1f 48 8b 50 08 <f6> c2 01 75 47 66 90 31 d2 31 c9 31 f6 31 ff c3 cc cc cc cc a9 ff
Jun 16 13:07:46 tent kernel: RSP: 0000:ffffa2faa7fe3bd8 EFLAGS: 00000286
Jun 16 13:07:46 tent kernel: RAX: ffffebaea68b8ac0 RBX: 000000000b23a000 RCX: 0000000000000000
Jun 16 13:07:46 tent kernel: RDX: ffffebaeb1c25788 RSI: 0000000000000000 RDI: 0000000000000000
Jun 16 13:07:46 tent kernel: RBP: ffff9203e55581d0 R08: 0000000000000000 R09: 0000000000000000
Jun 16 13:07:46 tent kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffebaeb1c25780
Jun 16 13:07:46 tent kernel: R13: ffff91f53f7e69a0 R14: 000000000b400000 R15: ffffa2faa7fe3d88
Jun 16 13:07:46 tent kernel:  ? vm_normal_folio+0xe/0x80
Jun 16 13:07:46 tent kernel:  change_pte_range+0x12b/0x900
Jun 16 13:07:46 tent kernel:  change_protection+0x716/0xbc0
Jun 16 13:07:46 tent kernel:  change_prot_numa+0x64/0x110
Jun 16 13:07:46 tent kernel:  task_numa_work+0x3c1/0x960
Jun 16 13:07:46 tent kernel:  ? __note_gp_changes+0x221/0x280
Jun 16 13:07:46 tent kernel:  task_work_run+0x5c/0x90
Jun 16 13:07:46 tent kernel:  irqentry_exit_to_user_mode+0x221/0x230
Jun 16 13:07:46 tent kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
Jun 16 13:07:46 tent kernel: RIP: 0033:0x7ffff4c1e73c
Jun 16 13:07:46 tent kernel: Code: ec 18 64 48 8b 04 25 28 00 00 00 48 89 44 24 08 48 8b 07 48 85 c0 74 76 48 89 c2 83 e2 06 75 36 48 83 e0 f8 74 68 0f b6 50 1c <83> e2 7f 83 ea 32 83 fa 01 77 04 48 8b 40 40 48 8b 54 24 08 64 48
Jun 16 13:07:46 tent kernel: RSP: 002b:00007ffffffe6530 EFLAGS: 00000206
Jun 16 13:07:46 tent kernel: RAX: 00000000005d3928 RBX: 00007ffffffe7460 RCX: 000000000002328f
Jun 16 13:07:46 tent kernel: RDX: 0000000000000045 RSI: 00007ffffffe68e0 RDI: 00007ffffffe6550
Jun 16 13:07:46 tent kernel: RBP: 000000000002328f R08: 0000000000000000 R09: 0000000000000001
Jun 16 13:07:46 tent kernel: R10: 00000000005eb2a8 R11: 0000000000000000 R12: 00007ffffffe68e0
Jun 16 13:07:46 tent kernel: R13: 0000000000000000 R14: 0000000000000001 R15: 00007ffffffe65a8
Jun 16 13:07:46 tent kernel:  </TASK>
Jun 16 13:07:46 tent kernel: rcu: rcu_preempt kthread starved for 10500 jiffies! g64380521 f0x0 RCU_GP_DOING_FQS(6) ->state=0x0 ->cpu=44
Jun 16 13:07:46 tent kernel: rcu:         Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Jun 16 13:07:46 tent kernel: rcu: RCU grace-period kthread stack dump:
Jun 16 13:07:46 tent kernel: task:rcu_preempt     state:R  running task     stack:0     pid:18    tgid:18    ppid:2      flags:0x00004008
Jun 16 13:07:46 tent kernel: Call Trace:
Jun 16 13:07:46 tent kernel:  <TASK>
Jun 16 13:07:46 tent kernel:  ? __schedule+0x3d8/0x12d0
Jun 16 13:07:46 tent kernel:  ? lock_timer_base+0x76/0xa0
Jun 16 13:07:46 tent kernel:  ? _raw_spin_lock+0x3f/0x60
Jun 16 13:07:46 tent kernel:  ? raw_spin_rq_lock_nested+0x1c/0x90
Jun 16 13:07:46 tent kernel:  ? _raw_spin_rq_lock_irqsave+0x17/0x30
Jun 16 13:07:46 tent kernel:  ? resched_cpu+0x2a/0x90
Jun 16 13:07:46 tent kernel:  ? force_qs_rnp+0x271/0x310
Jun 16 13:07:46 tent kernel:  ? __pfx_rcu_watching_snap_recheck+0x10/0x10
Jun 16 13:07:46 tent kernel:  ? rcu_gp_fqs_loop+0x4c6/0x6e0
Jun 16 13:07:46 tent kernel:  ? __pfx_rcu_gp_kthread+0x10/0x10
Jun 16 13:07:46 tent kernel:  ? rcu_gp_kthread+0x1ac/0x280
Jun 16 13:07:46 tent kernel:  ? kthread+0xd0/0x100
Jun 16 13:07:46 tent kernel:  ? __pfx_kthread+0x10/0x10
Jun 16 13:07:46 tent kernel:  ? ret_from_fork+0x34/0x50
Jun 16 13:07:46 tent kernel:  ? __pfx_kthread+0x10/0x10
Jun 16 13:07:46 tent kernel:  ? ret_from_fork_asm+0x1a/0x30
Jun 16 13:07:46 tent kernel:  </TASK>
Jun 16 13:07:46 tent kernel: rcu: Stack dump where RCU GP kthread last ran:
Jun 16 13:07:46 tent kernel: Sending NMI from CPU 3 to CPUs 44:
Jun 16 13:07:46 tent kernel: NMI backtrace for cpu 44
Jun 16 13:07:46 tent kernel: CPU: 44 UID: 0 PID: 18 Comm: rcu_preempt Tainted: G      D            6.12.9 #1-NixOS
Jun 16 13:07:46 tent kernel: Tainted: [D]=DIE
Jun 16 13:07:46 tent kernel: Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0016.033120161139 03/31/2016
Jun 16 13:07:46 tent kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x28f/0x2c0
Jun 16 13:07:46 tent kernel: Code: 83 e0 03 83 e9 01 48 c1 e0 05 48 63 c9 48 05 40 69 03 00 48 03 04 cd 20 54 07 aa 48 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 <85> c0 74 f7 48 8b 0a 48 85 c9 74 81 0f 0d 09 e9 79 ff ff ff be 01
Jun 16 13:07:46 tent kernel: RSP: 0018:ffffa2fa84317da0 EFLAGS: 00000046
Jun 16 13:07:46 tent kernel: RAX: 0000000000000000 RBX: ffff9212bfd35800 RCX: 0000000000000036
Jun 16 13:07:46 tent kernel: RDX: ffff9212bf836940 RSI: 0000000000dc0101 RDI: ffff9212bfd35800
Jun 16 13:07:46 tent kernel: RBP: ffff9212bf836940 R08: 0000000000000000 R09: 0000000000000000
Jun 16 13:07:46 tent kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000b40000
Jun 16 13:07:46 tent kernel: R13: 0000000000b40000 R14: ffff9212bfd369c0 R15: 0000000000000036
Jun 16 13:07:46 tent kernel: FS:  0000000000000000(0000) GS:ffff9212bf800000(0000) knlGS:0000000000000000
Jun 16 13:07:46 tent kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 16 13:07:46 tent kernel: CR2: 0000000000483368 CR3: 0000000802422002 CR4: 00000000003706f0
Jun 16 13:07:46 tent kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 16 13:07:46 tent kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jun 16 13:07:46 tent kernel: Call Trace:
Jun 16 13:07:46 tent kernel:  <NMI>
Jun 16 13:07:46 tent kernel:  ? nmi_cpu_backtrace+0x9f/0x120
Jun 16 13:07:46 tent kernel:  ? nmi_cpu_backtrace_handler+0x11/0x20
Jun 16 13:07:46 tent kernel:  ? nmi_handle+0x61/0x160
Jun 16 13:07:46 tent kernel:  ? default_do_nmi+0x43/0x100
Jun 16 13:07:46 tent kernel:  ? exc_nmi+0x138/0x1d0
Jun 16 13:07:46 tent kernel:  ? end_repeat_nmi+0xf/0x53
Jun 16 13:07:46 tent kernel:  ? native_queued_spin_lock_slowpath+0x28f/0x2c0
Jun 16 13:07:46 tent kernel:  ? native_queued_spin_lock_slowpath+0x28f/0x2c0
Jun 16 13:07:46 tent kernel:  ? native_queued_spin_lock_slowpath+0x28f/0x2c0
Jun 16 13:07:46 tent kernel:  </NMI>
Jun 16 13:07:46 tent kernel:  <TASK>
Jun 16 13:07:46 tent kernel:  _raw_spin_lock+0x3f/0x60
Jun 16 13:07:46 tent kernel:  raw_spin_rq_lock_nested+0x1c/0x90
Jun 16 13:07:46 tent kernel:  _raw_spin_rq_lock_irqsave+0x17/0x30
Jun 16 13:07:46 tent kernel:  resched_cpu+0x2a/0x90
Jun 16 13:07:46 tent kernel:  force_qs_rnp+0x271/0x310
Jun 16 13:07:46 tent kernel:  ? __pfx_rcu_watching_snap_recheck+0x10/0x10
Jun 16 13:07:46 tent kernel:  rcu_gp_fqs_loop+0x4c6/0x6e0
Jun 16 13:07:46 tent kernel:  ? __pfx_rcu_gp_kthread+0x10/0x10
Jun 16 13:07:46 tent kernel:  rcu_gp_kthread+0x1ac/0x280
Jun 16 13:07:46 tent kernel:  kthread+0xd0/0x100
Jun 16 13:07:46 tent kernel:  ? __pfx_kthread+0x10/0x10
Jun 16 13:07:46 tent kernel:  ret_from_fork+0x34/0x50
Jun 16 13:07:46 tent kernel:  ? __pfx_kthread+0x10/0x10
Jun 16 13:07:46 tent kernel:  ret_from_fork_asm+0x1a/0x30
Jun 16 13:07:46 tent kernel:  </TASK>

It is likely associated with a bad DIMM:

tent% sudo ipmitool sel get 0x264
SEL Record ID          : 0264
 Record Type           : 02
 Timestamp             : 2025-06-06 2025-06-06
 Generator ID          : 0033
 EvM Revision          : 04
 Sensor Type           : Memory
 Sensor Number         : 02
 Event Type            : Sensor-specific Discrete
 Event Direction       : Assertion Event
 Event Data (RAW)      : a00038
 Event Interpretation  : Missing
 Description           : Correctable ECC

Sensor ID              : Mmry ECC Sensor (0x2)
Entity ID              : 32.1 (Memory Device)
Sensor Type            : Memory (0x0c)

Not sure which one.

This failure to retrieve a page caused a hard lockup: ``` tent% sudo journalctl -b -1 -k -n 300 Jun 16 10:09:26 tent kernel: docker0: port 1(vetha33cb68) entered disabled state Jun 16 10:09:26 tent kernel: vetha33cb68 (unregistering): left allmulticast mode Jun 16 10:09:26 tent kernel: vetha33cb68 (unregistering): left promiscuous mode Jun 16 10:09:26 tent kernel: docker0: port 1(vetha33cb68) entered disabled state Jun 16 13:07:46 tent kernel: BUG: unable to handle page fault for address: ffffffffecb277fd Jun 16 13:07:46 tent kernel: #PF: supervisor instruction fetch in kernel mode Jun 16 13:07:46 tent kernel: #PF: error_code(0x0010) - not-present page Jun 16 13:07:46 tent kernel: PGD 802427067 P4D 802427067 PUD 802429067 PMD 0 Jun 16 13:07:46 tent kernel: Oops: Oops: 0010 [#1] PREEMPT SMP PTI Jun 16 13:07:46 tent kernel: CPU: 54 UID: 30012 PID: 909169 Comm: clang++ Not tainted 6.12.9 #1-NixOS Jun 16 13:07:46 tent kernel: Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0016.033120161139 03/31/2016 Jun 16 13:07:46 tent kernel: RIP: 0010:update_load_avg+0x348/0x7f0 Jun 16 13:07:46 tent kernel: Code: 89 86 88 00 00 00 0f 1f 44 00 00 0f 1f 44 00 00 48 83 bd c0 00 00 00 00 75 0a 41 f6 c5 04 0f 85 19 03 00 00 41 f6 c5 08 75 33 <48> 8b ab 38 01 00 00 48 8d 85 00 01 00 00 48 39 c3 0f 84 04 04 00 Jun 16 13:07:46 tent kernel: RSP: 0000:ffffa2fa87108e18 EFLAGS: 00010002 Jun 16 13:07:46 tent kernel: RAX: 0000000000000001 RBX: ffff9212bfd35900 RCX: 0000000000000000 Jun 16 13:07:46 tent kernel: RDX: ffff92030aa64200 RSI: 0000000000000000 RDI: 0000000000000000 Jun 16 13:07:46 tent kernel: RBP: ffff92030aa67600 R08: 0000000000000000 R09: 0000000000000000 Jun 16 13:07:46 tent kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0003dba34a654f09 Jun 16 13:07:46 tent kernel: R13: 0000000000000001 R14: 0000000000000001 R15: ffff9212bfd25900 Jun 16 13:07:46 tent kernel: FS: 00007ffff37a0780(0000) GS:ffff9212bfd00000(0000) knlGS:0000000000000000 Jun 16 13:07:46 tent kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 16 13:07:46 tent kernel: CR2: ffffffffecb277fd CR3: 00000003db266001 CR4: 00000000003706f0 Jun 16 13:07:46 tent kernel: Call Trace: Jun 16 13:07:46 tent kernel: <IRQ> Jun 16 13:07:46 tent kernel: ? __die+0x23/0x80 Jun 16 13:07:46 tent kernel: ? page_fault_oops+0x173/0x5b0 Jun 16 13:07:46 tent kernel: ? exc_page_fault+0x155/0x160 Jun 16 13:07:46 tent kernel: ? asm_exc_page_fault+0x26/0x30 Jun 16 13:07:46 tent kernel: ? update_load_avg+0x348/0x7f0 Jun 16 13:07:46 tent kernel: ? update_load_avg+0x7e/0x7f0 Jun 16 13:07:46 tent kernel: ? update_curr+0x98/0x250 Jun 16 13:07:46 tent kernel: task_tick_fair+0x6b/0x4e0 Jun 16 13:07:46 tent kernel: sched_tick+0xb0/0x2d0 Jun 16 13:07:46 tent kernel: update_process_times+0x96/0xb0 Jun 16 13:07:46 tent kernel: tick_nohz_handler+0x8f/0x150 Jun 16 13:07:46 tent kernel: ? __pfx_tick_nohz_handler+0x10/0x10 Jun 16 13:07:46 tent kernel: __hrtimer_run_queues+0x112/0x2b0 Jun 16 13:07:46 tent kernel: hrtimer_interrupt+0xfa/0x250 Jun 16 13:07:46 tent kernel: __sysvec_apic_timer_interrupt+0x58/0x120 Jun 16 13:07:46 tent kernel: sysvec_apic_timer_interrupt+0x6e/0x80 Jun 16 13:07:46 tent kernel: </IRQ> Jun 16 13:07:46 tent kernel: <TASK> Jun 16 13:07:46 tent kernel: asm_sysvec_apic_timer_interrupt+0x1a/0x20 Jun 16 13:07:46 tent kernel: RIP: 0010:vm_normal_folio+0x17/0x80 Jun 16 13:07:46 tent kernel: Code: 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 e8 02 ff ff ff 48 85 c0 74 1f 48 8b 50 08 <f6> c2 01 75 47 66 90 31 d2 31 c9 31 f6 31 ff c3 cc cc cc cc a9 ff Jun 16 13:07:46 tent kernel: RSP: 0000:ffffa2faa7fe3bd8 EFLAGS: 00000286 Jun 16 13:07:46 tent kernel: RAX: ffffebaea68b8ac0 RBX: 000000000b23a000 RCX: 0000000000000000 Jun 16 13:07:46 tent kernel: RDX: ffffebaeb1c25788 RSI: 0000000000000000 RDI: 0000000000000000 Jun 16 13:07:46 tent kernel: RBP: ffff9203e55581d0 R08: 0000000000000000 R09: 0000000000000000 Jun 16 13:07:46 tent kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffebaeb1c25780 Jun 16 13:07:46 tent kernel: R13: ffff91f53f7e69a0 R14: 000000000b400000 R15: ffffa2faa7fe3d88 Jun 16 13:07:46 tent kernel: ? vm_normal_folio+0xe/0x80 Jun 16 13:07:46 tent kernel: change_pte_range+0x12b/0x900 Jun 16 13:07:46 tent kernel: change_protection+0x716/0xbc0 Jun 16 13:07:46 tent kernel: change_prot_numa+0x64/0x110 Jun 16 13:07:46 tent kernel: task_numa_work+0x3c1/0x960 Jun 16 13:07:46 tent kernel: ? __note_gp_changes+0x221/0x280 Jun 16 13:07:46 tent kernel: task_work_run+0x5c/0x90 Jun 16 13:07:46 tent kernel: irqentry_exit_to_user_mode+0x221/0x230 Jun 16 13:07:46 tent kernel: asm_sysvec_apic_timer_interrupt+0x1a/0x20 Jun 16 13:07:46 tent kernel: RIP: 0033:0x7ffff4c1e73c Jun 16 13:07:46 tent kernel: Code: ec 18 64 48 8b 04 25 28 00 00 00 48 89 44 24 08 48 8b 07 48 85 c0 74 76 48 89 c2 83 e2 06 75 36 48 83 e0 f8 74 68 0f b6 50 1c <83> e2 7f 83 ea 32 83 fa 01 77 04 48 8b 40 40 48 8b 54 24 08 64 48 Jun 16 13:07:46 tent kernel: RSP: 002b:00007ffffffe6530 EFLAGS: 00000206 Jun 16 13:07:46 tent kernel: RAX: 00000000005d3928 RBX: 00007ffffffe7460 RCX: 000000000002328f Jun 16 13:07:46 tent kernel: RDX: 0000000000000045 RSI: 00007ffffffe68e0 RDI: 00007ffffffe6550 Jun 16 13:07:46 tent kernel: RBP: 000000000002328f R08: 0000000000000000 R09: 0000000000000001 Jun 16 13:07:46 tent kernel: R10: 00000000005eb2a8 R11: 0000000000000000 R12: 00007ffffffe68e0 Jun 16 13:07:46 tent kernel: R13: 0000000000000000 R14: 0000000000000001 R15: 00007ffffffe65a8 Jun 16 13:07:46 tent kernel: </TASK> Jun 16 13:07:46 tent kernel: Modules linked in: nft_chain_nat xt_MASQUERADE nf_conntrack_netlink xfrm_user xt_addrtype overlay xt_nat nf_nat br_netfilter veth tls cmac algif_hash bluetooth rfkill ecdh_generic ecc qrtr tcp_diag inet_diag nhpoly1305_avx2 nhpoly1305_sse2 nhpoly1305 chacha_generic chacha_x86_64 libchacha adiantum libpoly1305 camellia_generic camellia_aesni_avx2 camellia_aesni_avx_x86_64 camellia_x86_64 cast5_avx_x86_64 cast5_generic cast_common des_generic des3_ede_x86_64 libdes blowfish_generic blowfish_x86_64 blowfish_common cbc serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic xts twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common lrw algif_skcipher af_alg msr sb_edac edac_core intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel gf128mul hfi1 crypto_simd Jun 16 13:07:46 tent kernel: cryptd ixgbe rdmavt xfrm_algo mdio_devres ib_uverbs joydev iTCO_wdt libphy intel_pmc_bxt watchdog rapl intel_cstate mxm_wmi evdev ptp ib_core intel_uncore ipmi_si mgag200 mei_me pps_core i2c_i801 i2c_algo_bit mdio mei lpc_ich i2c_mux i2c_smbus ioatdma dca input_leds mousedev led_class mac_hid wmi acpi_power_meter tiny_power_button acpi_ipmi acpi_pad button xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipt_rpfilter xt_pkttype xt_LOG nf_log_syslog xt_tcpudp nft_compat nf_tables sch_fq_codel libcrc32c atkbd libps2 serio vivaldi_fmap loop cpufreq_powersave tun tap macvlan bridge stp llc kvm ipmi_watchdog ipmi_devintf ipmi_msghandler fuse efi_pstore configfs nfnetlink dmi_sysfs ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache jbd2 raid1 md_mod hid_generic sd_mod usbhid hid ahci libahci libata xhci_pci xhci_hcd crc32c_intel scsi_mod ehci_pci ehci_hcd scsi_common rtc_cmos dm_mod dax Jun 16 13:07:46 tent kernel: CR2: ffffffffecb277fd Jun 16 13:07:46 tent kernel: ---[ end trace 0000000000000000 ]--- Jun 16 13:07:46 tent kernel: ixgbe 0000:03:00.0 eno1: NETDEV WATCHDOG: CPU: 1: transmit queue 10 timed out 5085 ms Jun 16 13:07:46 tent kernel: watchdog: Watchdog detected hard LOCKUP on cpu 39 Jun 16 13:07:46 tent kernel: Modules linked in: nft_chain_nat xt_MASQUERADE nf_conntrack_netlink xfrm_user xt_addrtype overlay xt_nat nf_nat br_netfilter veth tls cmac algif_hash bluetooth rfkill ecdh_generic ecc qrtr tcp_diag inet_diag nhpoly1305_avx2 nhpoly1305_sse2 nhpoly1305 chacha_generic chacha_x86_64 libchacha adiantum libpoly1305 camellia_generic camellia_aesni_avx2 camellia_aesni_avx_x86_64 camellia_x86_64 cast5_avx_x86_64 cast5_generic cast_common des_generic des3_ede_x86_64 libdes blowfish_generic blowfish_x86_64 blowfish_common cbc serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic xts twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common lrw algif_skcipher af_alg msr sb_edac edac_core intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel gf128mul hfi1 crypto_simd Jun 16 13:07:46 tent kernel: cryptd ixgbe rdmavt xfrm_algo mdio_devres ib_uverbs joydev iTCO_wdt libphy intel_pmc_bxt watchdog rapl intel_cstate mxm_wmi evdev ptp ib_core intel_uncore ipmi_si mgag200 mei_me pps_core i2c_i801 i2c_algo_bit mdio mei lpc_ich i2c_mux i2c_smbus ioatdma dca input_leds mousedev led_class mac_hid wmi acpi_power_meter tiny_power_button acpi_ipmi acpi_pad button xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipt_rpfilter xt_pkttype xt_LOG nf_log_syslog xt_tcpudp nft_compat nf_tables sch_fq_codel libcrc32c atkbd libps2 serio vivaldi_fmap loop cpufreq_powersave tun tap macvlan bridge stp llc kvm ipmi_watchdog ipmi_devintf ipmi_msghandler fuse efi_pstore configfs nfnetlink dmi_sysfs ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache jbd2 raid1 md_mod hid_generic sd_mod usbhid hid ahci libahci libata xhci_pci xhci_hcd crc32c_intel scsi_mod ehci_pci ehci_hcd scsi_common rtc_cmos dm_mod dax Jun 16 13:07:46 tent kernel: CPU: 39 UID: 30012 PID: 909795 Comm: make Tainted: G D 6.12.9 #1-NixOS Jun 16 13:07:46 tent kernel: Tainted: [D]=DIE Jun 16 13:07:46 tent kernel: Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0016.033120161139 03/31/2016 Jun 16 13:07:46 tent kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x79/0x2c0 Jun 16 13:07:46 tent kernel: Code: 0f ba 2b 08 0f 92 c2 8b 03 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 77 64 85 c0 74 10 0f b6 03 84 c0 74 09 f3 90 0f b6 03 <84> c0 75 f7 b8 01 00 00 00 66 89 03 5b 5d 41 5c 41 5d 31 c0 31 d2 Jun 16 13:07:46 tent kernel: RSP: 0018:ffffa2fab0a2bce0 EFLAGS: 00000002 Jun 16 13:07:46 tent kernel: RAX: 0000000000000001 RBX: ffff9212bfd35800 RCX: 0000000000000000 Jun 16 13:07:46 tent kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9212bfd35800 Jun 16 13:07:46 tent kernel: RBP: ffff91f3e03d9280 R08: 0000000000000000 R09: 0000000000000000 Jun 16 13:07:46 tent kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000035800 Jun 16 13:07:46 tent kernel: R13: ffffa2fab0a2bd50 R14: 0000000000000084 R15: ffff91f39168b200 Jun 16 13:07:46 tent kernel: FS: 00007ffff7dc2740(0000) GS:ffff9202bfc80000(0000) knlGS:0000000000000000 Jun 16 13:07:46 tent kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 16 13:07:46 tent kernel: CR2: 000000000049a018 CR3: 00000010a9008004 CR4: 00000000003706f0 Jun 16 13:07:46 tent kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jun 16 13:07:46 tent kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Jun 16 13:07:46 tent kernel: Call Trace: Jun 16 13:07:46 tent kernel: <NMI> Jun 16 13:07:46 tent kernel: ? watchdog_hardlockup_check+0x106/0x1f0 Jun 16 13:07:46 tent kernel: ? __perf_event_overflow+0x111/0x320 Jun 16 13:07:46 tent kernel: ? handle_pmi_common+0x16d/0x3d0 Jun 16 13:07:46 tent kernel: ? set_pte_vaddr_p4d+0x4e/0x60 Jun 16 13:07:46 tent kernel: ? flush_tlb_one_kernel+0xe/0x40 Jun 16 13:07:46 tent kernel: ? ghes_copy_tofrom_phys+0x7f/0x120 Jun 16 13:07:46 tent kernel: ? intel_pmu_handle_irq+0x10a/0x520 Jun 16 13:07:46 tent kernel: ? ghes_notify_nmi+0x238/0x390 Jun 16 13:07:46 tent kernel: ? perf_event_nmi_handler+0x2a/0x50 Jun 16 13:07:46 tent kernel: ? nmi_handle+0x61/0x160 Jun 16 13:07:46 tent kernel: ? default_do_nmi+0x43/0x100 Jun 16 13:07:46 tent kernel: ? exc_nmi+0x138/0x1d0 Jun 16 13:07:46 tent kernel: ? end_repeat_nmi+0xf/0x53 Jun 16 13:07:46 tent kernel: ? native_queued_spin_lock_slowpath+0x79/0x2c0 Jun 16 13:07:46 tent kernel: ? native_queued_spin_lock_slowpath+0x79/0x2c0 Jun 16 13:07:46 tent kernel: ? native_queued_spin_lock_slowpath+0x79/0x2c0 Jun 16 13:07:46 tent kernel: </NMI> Jun 16 13:07:46 tent kernel: <TASK> Jun 16 13:07:46 tent kernel: _raw_spin_lock+0x3f/0x60 Jun 16 13:07:46 tent kernel: raw_spin_rq_lock_nested+0x1c/0x90 Jun 16 13:07:46 tent kernel: __task_rq_lock+0x34/0xf0 Jun 16 13:07:46 tent kernel: wake_up_new_task+0x160/0x320 Jun 16 13:07:46 tent kernel: kernel_clone+0x2a4/0x430 Jun 16 13:07:46 tent kernel: __do_sys_clone3+0xef/0x140 Jun 16 13:07:46 tent kernel: do_syscall_64+0xb7/0x210 Jun 16 13:07:46 tent kernel: entry_SYSCALL_64_after_hwframe+0x77/0x7f Jun 16 13:07:46 tent kernel: RIP: 0033:0x7ffff7ed539d Jun 16 13:07:46 tent kernel: Code: 31 f6 31 ff 45 31 d2 45 31 db c3 66 90 f3 0f 1e fa b8 ea ff ff ff 48 85 ff 74 28 48 85 d2 74 23 49 89 c8 b8 b3 01 00 00 0f 05 <48> 85 c0 7c 14 74 01 c3 31 ed 4c 89 c7 ff d2 48 89 c7 b8 3c 00 00 Jun 16 13:07:46 tent kernel: RSP: 002b:00007fffffff8838 EFLAGS: 00000206 ORIG_RAX: 00000000000001b3 Jun 16 13:07:46 tent kernel: RAX: ffffffffffffffda RBX: 00007ffff7db9000 RCX: 00007ffff7ed539d Jun 16 13:07:46 tent kernel: RDX: 00007ffff7ebcc50 RSI: 0000000000000058 RDI: 00007fffffff8880 Jun 16 13:07:46 tent kernel: RBP: 0000000000009000 R08: 00007fffffff88e0 R09: 0000000000000000 Jun 16 13:07:46 tent kernel: R10: 0000000000000008 R11: 0000000000000206 R12: 00007fffffff8c10 Jun 16 13:07:46 tent kernel: R13: 00007fffffff88e0 R14: 0000000000000000 R15: 00007ffff7ebcc50 Jun 16 13:07:46 tent kernel: </TASK> Jun 16 13:07:46 tent kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Jun 16 13:07:46 tent kernel: rcu: 54-...0: (8 ticks this GP) idle=7684/1/0x4000000000000000 softirq=37445173/37445173 fqs=2100 Jun 16 13:07:46 tent kernel: rcu: (detected by 3, t=21002 jiffies, g=64380521, q=74746 ncpus=56) Jun 16 13:07:46 tent kernel: Sending NMI from CPU 3 to CPUs 54: Jun 16 13:07:46 tent kernel: NMI backtrace for cpu 54 Jun 16 13:07:46 tent kernel: CPU: 54 UID: 30012 PID: 909169 Comm: clang++ Tainted: G D 6.12.9 #1-NixOS Jun 16 13:07:46 tent kernel: Tainted: [D]=DIE Jun 16 13:07:46 tent kernel: Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0016.033120161139 03/31/2016 Jun 16 13:07:46 tent kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x28f/0x2c0 Jun 16 13:07:46 tent kernel: Code: 83 e0 03 83 e9 01 48 c1 e0 05 48 63 c9 48 05 40 69 03 00 48 03 04 cd 20 54 07 aa 48 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 <85> c0 74 f7 48 8b 0a 48 85 c9 74 81 0f 0d 09 e9 79 ff ff ff be 01 Jun 16 13:07:46 tent kernel: RSP: 0000:ffffa2fa87108890 EFLAGS: 00000046 Jun 16 13:07:46 tent kernel: RAX: 0000000000000000 RBX: ffff9212bfd35800 RCX: 0000000000000019 Jun 16 13:07:46 tent kernel: RDX: ffff9212bfd36940 RSI: 0000000000680101 RDI: ffff9212bfd35800 Jun 16 13:07:46 tent kernel: RBP: ffff9212bfd36940 R08: 0000000000000000 R09: 0000000000000000 Jun 16 13:07:46 tent kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000dc0000 Jun 16 13:07:46 tent kernel: R13: 0000000000dc0000 R14: 0000000000000008 R15: ffff92030ba2af24 Jun 16 13:07:46 tent kernel: FS: 00007ffff37a0780(0000) GS:ffff9212bfd00000(0000) knlGS:0000000000000000 Jun 16 13:07:46 tent kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 16 13:07:46 tent kernel: CR2: ffffffffecb277fd CR3: 00000003db266001 CR4: 00000000003706f0 Jun 16 13:07:46 tent kernel: Call Trace: Jun 16 13:07:46 tent kernel: <NMI> Jun 16 13:07:46 tent kernel: ? nmi_cpu_backtrace+0x9f/0x120 Jun 16 13:07:46 tent kernel: ? nmi_cpu_backtrace_handler+0x11/0x20 Jun 16 13:07:46 tent kernel: ? nmi_handle+0x61/0x160 Jun 16 13:07:46 tent kernel: ? default_do_nmi+0x43/0x100 Jun 16 13:07:46 tent kernel: ? exc_nmi+0x138/0x1d0 Jun 16 13:07:46 tent kernel: ? end_repeat_nmi+0xf/0x53 Jun 16 13:07:46 tent kernel: ? native_queued_spin_lock_slowpath+0x28f/0x2c0 Jun 16 13:07:46 tent kernel: ? native_queued_spin_lock_slowpath+0x28f/0x2c0 Jun 16 13:07:46 tent kernel: ? native_queued_spin_lock_slowpath+0x28f/0x2c0 Jun 16 13:07:46 tent kernel: </NMI> Jun 16 13:07:46 tent kernel: <IRQ> Jun 16 13:07:46 tent kernel: _raw_spin_lock+0x3f/0x60 Jun 16 13:07:46 tent kernel: raw_spin_rq_lock_nested+0x1c/0x90 Jun 16 13:07:46 tent kernel: try_to_wake_up+0x218/0x6c0 Jun 16 13:07:46 tent kernel: kick_pool+0x66/0x160 Jun 16 13:07:46 tent kernel: __queue_work+0x2df/0x4e0 Jun 16 13:07:46 tent kernel: queue_work_on+0x6b/0x80 Jun 16 13:07:46 tent kernel: soft_cursor+0x1a0/0x250 Jun 16 13:07:46 tent kernel: ? info_print_prefix+0xb3/0xe0 Jun 16 13:07:46 tent kernel: bit_cursor+0x383/0x600 Jun 16 13:07:46 tent kernel: hide_cursor+0x2b/0xb0 Jun 16 13:07:46 tent kernel: vt_console_print+0x44b/0x460 Jun 16 13:07:46 tent kernel: console_flush_all+0x291/0x490 Jun 16 13:07:46 tent kernel: console_unlock+0x73/0x130 Jun 16 13:07:46 tent kernel: vprintk_emit+0x174/0x2c0 Jun 16 13:07:46 tent kernel: _printk+0x64/0x90 Jun 16 13:07:46 tent kernel: oops_exit+0x26/0x40 Jun 16 13:07:46 tent kernel: oops_end+0x4c/0xc0 Jun 16 13:07:46 tent kernel: page_fault_oops+0x197/0x5b0 Jun 16 13:07:46 tent kernel: exc_page_fault+0x155/0x160 Jun 16 13:07:46 tent kernel: asm_exc_page_fault+0x26/0x30 Jun 16 13:07:46 tent kernel: RIP: 0010:update_load_avg+0x348/0x7f0 Jun 16 13:07:46 tent kernel: Code: 89 86 88 00 00 00 0f 1f 44 00 00 0f 1f 44 00 00 48 83 bd c0 00 00 00 00 75 0a 41 f6 c5 04 0f 85 19 03 00 00 41 f6 c5 08 75 33 <48> 8b ab 38 01 00 00 48 8d 85 00 01 00 00 48 39 c3 0f 84 04 04 00 Jun 16 13:07:46 tent kernel: RSP: 0000:ffffa2fa87108e18 EFLAGS: 00010002 Jun 16 13:07:46 tent kernel: RAX: 0000000000000001 RBX: ffff9212bfd35900 RCX: 0000000000000000 Jun 16 13:07:46 tent kernel: RDX: ffff92030aa64200 RSI: 0000000000000000 RDI: 0000000000000000 Jun 16 13:07:46 tent kernel: RBP: ffff92030aa67600 R08: 0000000000000000 R09: 0000000000000000 Jun 16 13:07:46 tent kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0003dba34a654f09 Jun 16 13:07:46 tent kernel: R13: 0000000000000001 R14: 0000000000000001 R15: ffff9212bfd25900 Jun 16 13:07:46 tent kernel: ? update_load_avg+0x7e/0x7f0 Jun 16 13:07:46 tent kernel: ? update_curr+0x98/0x250 Jun 16 13:07:46 tent kernel: task_tick_fair+0x6b/0x4e0 Jun 16 13:07:46 tent kernel: sched_tick+0xb0/0x2d0 Jun 16 13:07:46 tent kernel: update_process_times+0x96/0xb0 Jun 16 13:07:46 tent kernel: tick_nohz_handler+0x8f/0x150 Jun 16 13:07:46 tent kernel: ? __pfx_tick_nohz_handler+0x10/0x10 Jun 16 13:07:46 tent kernel: __hrtimer_run_queues+0x112/0x2b0 Jun 16 13:07:46 tent kernel: hrtimer_interrupt+0xfa/0x250 Jun 16 13:07:46 tent kernel: __sysvec_apic_timer_interrupt+0x58/0x120 Jun 16 13:07:46 tent kernel: sysvec_apic_timer_interrupt+0x6e/0x80 Jun 16 13:07:46 tent kernel: </IRQ> Jun 16 13:07:46 tent kernel: <TASK> Jun 16 13:07:46 tent kernel: asm_sysvec_apic_timer_interrupt+0x1a/0x20 Jun 16 13:07:46 tent kernel: RIP: 0010:vm_normal_folio+0x17/0x80 Jun 16 13:07:46 tent kernel: Code: 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 e8 02 ff ff ff 48 85 c0 74 1f 48 8b 50 08 <f6> c2 01 75 47 66 90 31 d2 31 c9 31 f6 31 ff c3 cc cc cc cc a9 ff Jun 16 13:07:46 tent kernel: RSP: 0000:ffffa2faa7fe3bd8 EFLAGS: 00000286 Jun 16 13:07:46 tent kernel: RAX: ffffebaea68b8ac0 RBX: 000000000b23a000 RCX: 0000000000000000 Jun 16 13:07:46 tent kernel: RDX: ffffebaeb1c25788 RSI: 0000000000000000 RDI: 0000000000000000 Jun 16 13:07:46 tent kernel: RBP: ffff9203e55581d0 R08: 0000000000000000 R09: 0000000000000000 Jun 16 13:07:46 tent kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffebaeb1c25780 Jun 16 13:07:46 tent kernel: R13: ffff91f53f7e69a0 R14: 000000000b400000 R15: ffffa2faa7fe3d88 Jun 16 13:07:46 tent kernel: ? vm_normal_folio+0xe/0x80 Jun 16 13:07:46 tent kernel: change_pte_range+0x12b/0x900 Jun 16 13:07:46 tent kernel: change_protection+0x716/0xbc0 Jun 16 13:07:46 tent kernel: change_prot_numa+0x64/0x110 Jun 16 13:07:46 tent kernel: task_numa_work+0x3c1/0x960 Jun 16 13:07:46 tent kernel: ? __note_gp_changes+0x221/0x280 Jun 16 13:07:46 tent kernel: task_work_run+0x5c/0x90 Jun 16 13:07:46 tent kernel: irqentry_exit_to_user_mode+0x221/0x230 Jun 16 13:07:46 tent kernel: asm_sysvec_apic_timer_interrupt+0x1a/0x20 Jun 16 13:07:46 tent kernel: RIP: 0033:0x7ffff4c1e73c Jun 16 13:07:46 tent kernel: Code: ec 18 64 48 8b 04 25 28 00 00 00 48 89 44 24 08 48 8b 07 48 85 c0 74 76 48 89 c2 83 e2 06 75 36 48 83 e0 f8 74 68 0f b6 50 1c <83> e2 7f 83 ea 32 83 fa 01 77 04 48 8b 40 40 48 8b 54 24 08 64 48 Jun 16 13:07:46 tent kernel: RSP: 002b:00007ffffffe6530 EFLAGS: 00000206 Jun 16 13:07:46 tent kernel: RAX: 00000000005d3928 RBX: 00007ffffffe7460 RCX: 000000000002328f Jun 16 13:07:46 tent kernel: RDX: 0000000000000045 RSI: 00007ffffffe68e0 RDI: 00007ffffffe6550 Jun 16 13:07:46 tent kernel: RBP: 000000000002328f R08: 0000000000000000 R09: 0000000000000001 Jun 16 13:07:46 tent kernel: R10: 00000000005eb2a8 R11: 0000000000000000 R12: 00007ffffffe68e0 Jun 16 13:07:46 tent kernel: R13: 0000000000000000 R14: 0000000000000001 R15: 00007ffffffe65a8 Jun 16 13:07:46 tent kernel: </TASK> Jun 16 13:07:46 tent kernel: rcu: rcu_preempt kthread starved for 10500 jiffies! g64380521 f0x0 RCU_GP_DOING_FQS(6) ->state=0x0 ->cpu=44 Jun 16 13:07:46 tent kernel: rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior. Jun 16 13:07:46 tent kernel: rcu: RCU grace-period kthread stack dump: Jun 16 13:07:46 tent kernel: task:rcu_preempt state:R running task stack:0 pid:18 tgid:18 ppid:2 flags:0x00004008 Jun 16 13:07:46 tent kernel: Call Trace: Jun 16 13:07:46 tent kernel: <TASK> Jun 16 13:07:46 tent kernel: ? __schedule+0x3d8/0x12d0 Jun 16 13:07:46 tent kernel: ? lock_timer_base+0x76/0xa0 Jun 16 13:07:46 tent kernel: ? _raw_spin_lock+0x3f/0x60 Jun 16 13:07:46 tent kernel: ? raw_spin_rq_lock_nested+0x1c/0x90 Jun 16 13:07:46 tent kernel: ? _raw_spin_rq_lock_irqsave+0x17/0x30 Jun 16 13:07:46 tent kernel: ? resched_cpu+0x2a/0x90 Jun 16 13:07:46 tent kernel: ? force_qs_rnp+0x271/0x310 Jun 16 13:07:46 tent kernel: ? __pfx_rcu_watching_snap_recheck+0x10/0x10 Jun 16 13:07:46 tent kernel: ? rcu_gp_fqs_loop+0x4c6/0x6e0 Jun 16 13:07:46 tent kernel: ? __pfx_rcu_gp_kthread+0x10/0x10 Jun 16 13:07:46 tent kernel: ? rcu_gp_kthread+0x1ac/0x280 Jun 16 13:07:46 tent kernel: ? kthread+0xd0/0x100 Jun 16 13:07:46 tent kernel: ? __pfx_kthread+0x10/0x10 Jun 16 13:07:46 tent kernel: ? ret_from_fork+0x34/0x50 Jun 16 13:07:46 tent kernel: ? __pfx_kthread+0x10/0x10 Jun 16 13:07:46 tent kernel: ? ret_from_fork_asm+0x1a/0x30 Jun 16 13:07:46 tent kernel: </TASK> Jun 16 13:07:46 tent kernel: rcu: Stack dump where RCU GP kthread last ran: Jun 16 13:07:46 tent kernel: Sending NMI from CPU 3 to CPUs 44: Jun 16 13:07:46 tent kernel: NMI backtrace for cpu 44 Jun 16 13:07:46 tent kernel: CPU: 44 UID: 0 PID: 18 Comm: rcu_preempt Tainted: G D 6.12.9 #1-NixOS Jun 16 13:07:46 tent kernel: Tainted: [D]=DIE Jun 16 13:07:46 tent kernel: Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0016.033120161139 03/31/2016 Jun 16 13:07:46 tent kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x28f/0x2c0 Jun 16 13:07:46 tent kernel: Code: 83 e0 03 83 e9 01 48 c1 e0 05 48 63 c9 48 05 40 69 03 00 48 03 04 cd 20 54 07 aa 48 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 <85> c0 74 f7 48 8b 0a 48 85 c9 74 81 0f 0d 09 e9 79 ff ff ff be 01 Jun 16 13:07:46 tent kernel: RSP: 0018:ffffa2fa84317da0 EFLAGS: 00000046 Jun 16 13:07:46 tent kernel: RAX: 0000000000000000 RBX: ffff9212bfd35800 RCX: 0000000000000036 Jun 16 13:07:46 tent kernel: RDX: ffff9212bf836940 RSI: 0000000000dc0101 RDI: ffff9212bfd35800 Jun 16 13:07:46 tent kernel: RBP: ffff9212bf836940 R08: 0000000000000000 R09: 0000000000000000 Jun 16 13:07:46 tent kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000b40000 Jun 16 13:07:46 tent kernel: R13: 0000000000b40000 R14: ffff9212bfd369c0 R15: 0000000000000036 Jun 16 13:07:46 tent kernel: FS: 0000000000000000(0000) GS:ffff9212bf800000(0000) knlGS:0000000000000000 Jun 16 13:07:46 tent kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 16 13:07:46 tent kernel: CR2: 0000000000483368 CR3: 0000000802422002 CR4: 00000000003706f0 Jun 16 13:07:46 tent kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jun 16 13:07:46 tent kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Jun 16 13:07:46 tent kernel: Call Trace: Jun 16 13:07:46 tent kernel: <NMI> Jun 16 13:07:46 tent kernel: ? nmi_cpu_backtrace+0x9f/0x120 Jun 16 13:07:46 tent kernel: ? nmi_cpu_backtrace_handler+0x11/0x20 Jun 16 13:07:46 tent kernel: ? nmi_handle+0x61/0x160 Jun 16 13:07:46 tent kernel: ? default_do_nmi+0x43/0x100 Jun 16 13:07:46 tent kernel: ? exc_nmi+0x138/0x1d0 Jun 16 13:07:46 tent kernel: ? end_repeat_nmi+0xf/0x53 Jun 16 13:07:46 tent kernel: ? native_queued_spin_lock_slowpath+0x28f/0x2c0 Jun 16 13:07:46 tent kernel: ? native_queued_spin_lock_slowpath+0x28f/0x2c0 Jun 16 13:07:46 tent kernel: ? native_queued_spin_lock_slowpath+0x28f/0x2c0 Jun 16 13:07:46 tent kernel: </NMI> Jun 16 13:07:46 tent kernel: <TASK> Jun 16 13:07:46 tent kernel: _raw_spin_lock+0x3f/0x60 Jun 16 13:07:46 tent kernel: raw_spin_rq_lock_nested+0x1c/0x90 Jun 16 13:07:46 tent kernel: _raw_spin_rq_lock_irqsave+0x17/0x30 Jun 16 13:07:46 tent kernel: resched_cpu+0x2a/0x90 Jun 16 13:07:46 tent kernel: force_qs_rnp+0x271/0x310 Jun 16 13:07:46 tent kernel: ? __pfx_rcu_watching_snap_recheck+0x10/0x10 Jun 16 13:07:46 tent kernel: rcu_gp_fqs_loop+0x4c6/0x6e0 Jun 16 13:07:46 tent kernel: ? __pfx_rcu_gp_kthread+0x10/0x10 Jun 16 13:07:46 tent kernel: rcu_gp_kthread+0x1ac/0x280 Jun 16 13:07:46 tent kernel: kthread+0xd0/0x100 Jun 16 13:07:46 tent kernel: ? __pfx_kthread+0x10/0x10 Jun 16 13:07:46 tent kernel: ret_from_fork+0x34/0x50 Jun 16 13:07:46 tent kernel: ? __pfx_kthread+0x10/0x10 Jun 16 13:07:46 tent kernel: ret_from_fork_asm+0x1a/0x30 Jun 16 13:07:46 tent kernel: </TASK> ``` It is likely associated with a bad DIMM: ``` tent% sudo ipmitool sel get 0x264 SEL Record ID : 0264 Record Type : 02 Timestamp : 2025-06-06 2025-06-06 Generator ID : 0033 EvM Revision : 04 Sensor Type : Memory Sensor Number : 02 Event Type : Sensor-specific Discrete Event Direction : Assertion Event Event Data (RAW) : a00038 Event Interpretation : Missing Description : Correctable ECC Sensor ID : Mmry ECC Sensor (0x2) Entity ID : 32.1 (Memory Device) Sensor Type : Memory (0x0c) ``` Not sure which one.
rarias added the
hw
kernel
 labels 2025-06-17 12:52:45 +02:00
rarias commented 2025-06-17 12:57:35 +02:00
Author
Owner
Copy Link
tent% sudo journalctl -b -1 -k | grep EDAC
Jun 03 19:04:48 tent kernel: EDAC MC: Ver: 3.0.0
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fa0
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fa0
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fa0
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f60
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f60
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f60
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fa8
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fa8
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fa8
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f71
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f71
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f71
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faa
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faa
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faa
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fab
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fab
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fab
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fac
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fad
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f68
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f68
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f68
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f79
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f79
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f79
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6a
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6a
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6a
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6b
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6b
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6b
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6c
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6d
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffc
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffc
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffc
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffd
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffd
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffd
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faf
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faf
Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faf
Jun 03 19:04:48 tent kernel: EDAC MC0: Giving out device to module sb_edac controller Broadwell SrcID#1_Ha#0: DEV 0000:ff:12.0 (INTERRUPT)
Jun 03 19:04:48 tent kernel: EDAC MC1: Giving out device to module sb_edac controller Broadwell SrcID#0_Ha#0: DEV 0000:7f:12.0 (INTERRUPT)
Jun 03 19:04:48 tent kernel: EDAC MC2: Giving out device to module sb_edac controller Broadwell SrcID#1_Ha#1: DEV 0000:ff:12.4 (INTERRUPT)
Jun 03 19:04:48 tent kernel: EDAC MC3: Giving out device to module sb_edac controller Broadwell SrcID#0_Ha#1: DEV 0000:7f:12.4 (INTERRUPT)
Jun 03 19:04:48 tent kernel: EDAC sbridge:  Ver: 1.1.2 
Jun 05 03:48:05 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
Jun 05 03:48:05 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 8: 8c00004000010091
Jun 05 03:48:05 tent kernel: EDAC sbridge MC0: TSC 12baec7cc6440 
Jun 05 03:48:05 tent kernel: EDAC sbridge MC0: ADDR 1cb5134e00 
Jun 05 03:48:05 tent kernel: EDAC sbridge MC0: MISC 1404ae486 
Jun 05 03:48:05 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749088085 SOCKET 1 APIC 20
Jun 05 03:48:05 tent kernel: EDAC MC2: 1 CE memory read error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 slot:0 page:0x1cb5134 offset:0xe00 grain:32 syndrome:0x0 -  area:DRAM err_code:0001:0091 socket:1 ha:1 channel_mask:2 rank:0 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0)
Jun 05 11:33:57 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
Jun 05 11:33:57 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1
Jun 05 11:33:57 tent kernel: EDAC sbridge MC0: TSC 16da0f3a7830e 
Jun 05 11:33:57 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 
Jun 05 11:33:57 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c 
Jun 05 11:33:57 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749116037 SOCKET 1 APIC 20
Jun 05 11:33:57 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 -  area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0)
Jun 05 19:58:01 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
Jun 05 19:58:01 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1
Jun 05 19:58:01 tent kernel: EDAC sbridge MC0: TSC 1b4fb614b8584 
Jun 05 19:58:01 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 
Jun 05 19:58:01 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c 
Jun 05 19:58:01 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749146281 SOCKET 1 APIC 20
Jun 05 19:58:01 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 -  area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0)
Jun 06 01:00:55 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
Jun 06 01:00:55 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1
Jun 06 01:00:55 tent kernel: EDAC sbridge MC0: TSC 1dfdbef448a1a 
Jun 06 01:00:55 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 
Jun 06 01:00:55 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c 
Jun 06 01:00:55 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749164455 SOCKET 1 APIC 20
Jun 06 01:00:55 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 -  area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0)
Jun 06 16:22:25 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
Jun 06 16:22:25 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1
Jun 06 16:22:25 tent kernel: EDAC sbridge MC0: TSC 2624d21be03ac 
Jun 06 16:22:25 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 
Jun 06 16:22:25 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c 
Jun 06 16:22:25 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749219745 SOCKET 1 APIC 20
Jun 06 16:22:25 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 -  area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0)
Jun 09 20:13:27 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
Jun 09 20:13:27 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1
Jun 09 20:13:27 tent kernel: EDAC sbridge MC0: TSC 4e6852128bd38 
Jun 09 20:13:27 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 
Jun 09 20:13:27 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c 
Jun 09 20:13:27 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749492807 SOCKET 1 APIC 20
Jun 09 20:13:27 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 -  area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0)
Jun 10 10:31:29 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
Jun 10 10:31:29 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1
Jun 10 10:31:29 tent kernel: EDAC sbridge MC0: TSC 55ffa639ae164 
Jun 10 10:31:29 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 
Jun 10 10:31:29 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c 
Jun 10 10:31:29 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749544289 SOCKET 1 APIC 20
Jun 10 10:31:29 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 -  area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0)
Jun 12 15:07:25 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
Jun 12 15:07:25 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1
Jun 12 15:07:25 tent kernel: EDAC sbridge MC0: TSC 71eb6cabf7714 
Jun 12 15:07:25 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 
Jun 12 15:07:25 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c 
Jun 12 15:07:25 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749733645 SOCKET 1 APIC 20
Jun 12 15:07:25 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 -  area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0)
Jun 13 14:50:35 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
Jun 13 14:50:35 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1
Jun 13 14:50:35 tent kernel: EDAC sbridge MC0: TSC 7e82b0ae1b256 
Jun 13 14:50:35 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 
Jun 13 14:50:35 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c 
Jun 13 14:50:35 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749819035 SOCKET 1 APIC 20
Jun 13 14:50:35 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 -  area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0)
Jun 14 23:47:43 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
Jun 14 23:47:43 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1
Jun 14 23:47:43 tent kernel: EDAC sbridge MC0: TSC 9000a6fa450ea 
Jun 14 23:47:43 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 
Jun 14 23:47:43 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c 
Jun 14 23:47:43 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749937663 SOCKET 1 APIC 20
Jun 14 23:47:43 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 -  area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0)
``` tent% sudo journalctl -b -1 -k | grep EDAC Jun 03 19:04:48 tent kernel: EDAC MC: Ver: 3.0.0 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fa0 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fa0 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fa0 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f60 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f60 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f60 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fa8 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fa8 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fa8 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f71 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f71 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f71 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faa Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faa Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faa Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fab Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fab Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fab Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fac Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6fad Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f68 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f68 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f68 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f79 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f79 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f79 Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6a Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6a Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6a Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6b Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6b Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6b Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6c Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6f6d Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffc Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffc Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffc Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffd Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffd Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6ffd Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faf Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faf Jun 03 19:04:48 tent kernel: EDAC sbridge: Seeking for: PCI ID 8086:6faf Jun 03 19:04:48 tent kernel: EDAC MC0: Giving out device to module sb_edac controller Broadwell SrcID#1_Ha#0: DEV 0000:ff:12.0 (INTERRUPT) Jun 03 19:04:48 tent kernel: EDAC MC1: Giving out device to module sb_edac controller Broadwell SrcID#0_Ha#0: DEV 0000:7f:12.0 (INTERRUPT) Jun 03 19:04:48 tent kernel: EDAC MC2: Giving out device to module sb_edac controller Broadwell SrcID#1_Ha#1: DEV 0000:ff:12.4 (INTERRUPT) Jun 03 19:04:48 tent kernel: EDAC MC3: Giving out device to module sb_edac controller Broadwell SrcID#0_Ha#1: DEV 0000:7f:12.4 (INTERRUPT) Jun 03 19:04:48 tent kernel: EDAC sbridge: Ver: 1.1.2 Jun 05 03:48:05 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR Jun 05 03:48:05 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 8: 8c00004000010091 Jun 05 03:48:05 tent kernel: EDAC sbridge MC0: TSC 12baec7cc6440 Jun 05 03:48:05 tent kernel: EDAC sbridge MC0: ADDR 1cb5134e00 Jun 05 03:48:05 tent kernel: EDAC sbridge MC0: MISC 1404ae486 Jun 05 03:48:05 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749088085 SOCKET 1 APIC 20 Jun 05 03:48:05 tent kernel: EDAC MC2: 1 CE memory read error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 slot:0 page:0x1cb5134 offset:0xe00 grain:32 syndrome:0x0 - area:DRAM err_code:0001:0091 socket:1 ha:1 channel_mask:2 rank:0 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0) Jun 05 11:33:57 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR Jun 05 11:33:57 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1 Jun 05 11:33:57 tent kernel: EDAC sbridge MC0: TSC 16da0f3a7830e Jun 05 11:33:57 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 Jun 05 11:33:57 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c Jun 05 11:33:57 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749116037 SOCKET 1 APIC 20 Jun 05 11:33:57 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 - area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0) Jun 05 19:58:01 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR Jun 05 19:58:01 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1 Jun 05 19:58:01 tent kernel: EDAC sbridge MC0: TSC 1b4fb614b8584 Jun 05 19:58:01 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 Jun 05 19:58:01 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c Jun 05 19:58:01 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749146281 SOCKET 1 APIC 20 Jun 05 19:58:01 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 - area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0) Jun 06 01:00:55 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR Jun 06 01:00:55 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1 Jun 06 01:00:55 tent kernel: EDAC sbridge MC0: TSC 1dfdbef448a1a Jun 06 01:00:55 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 Jun 06 01:00:55 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c Jun 06 01:00:55 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749164455 SOCKET 1 APIC 20 Jun 06 01:00:55 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 - area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0) Jun 06 16:22:25 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR Jun 06 16:22:25 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1 Jun 06 16:22:25 tent kernel: EDAC sbridge MC0: TSC 2624d21be03ac Jun 06 16:22:25 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 Jun 06 16:22:25 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c Jun 06 16:22:25 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749219745 SOCKET 1 APIC 20 Jun 06 16:22:25 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 - area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0) Jun 09 20:13:27 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR Jun 09 20:13:27 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1 Jun 09 20:13:27 tent kernel: EDAC sbridge MC0: TSC 4e6852128bd38 Jun 09 20:13:27 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 Jun 09 20:13:27 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c Jun 09 20:13:27 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749492807 SOCKET 1 APIC 20 Jun 09 20:13:27 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 - area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0) Jun 10 10:31:29 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR Jun 10 10:31:29 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1 Jun 10 10:31:29 tent kernel: EDAC sbridge MC0: TSC 55ffa639ae164 Jun 10 10:31:29 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 Jun 10 10:31:29 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c Jun 10 10:31:29 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749544289 SOCKET 1 APIC 20 Jun 10 10:31:29 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 - area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0) Jun 12 15:07:25 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR Jun 12 15:07:25 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1 Jun 12 15:07:25 tent kernel: EDAC sbridge MC0: TSC 71eb6cabf7714 Jun 12 15:07:25 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 Jun 12 15:07:25 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c Jun 12 15:07:25 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749733645 SOCKET 1 APIC 20 Jun 12 15:07:25 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 - area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0) Jun 13 14:50:35 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR Jun 13 14:50:35 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1 Jun 13 14:50:35 tent kernel: EDAC sbridge MC0: TSC 7e82b0ae1b256 Jun 13 14:50:35 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 Jun 13 14:50:35 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c Jun 13 14:50:35 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749819035 SOCKET 1 APIC 20 Jun 13 14:50:35 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 - area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0) Jun 14 23:47:43 tent kernel: EDAC sbridge MC0: HANDLING MCE MEMORY ERROR Jun 14 23:47:43 tent kernel: EDAC sbridge MC0: CPU 14: Machine Check Event: 0 Bank 14: 8c000045000800c1 Jun 14 23:47:43 tent kernel: EDAC sbridge MC0: TSC 9000a6fa450ea Jun 14 23:47:43 tent kernel: EDAC sbridge MC0: ADDR 1cb5134000 Jun 14 23:47:43 tent kernel: EDAC sbridge MC0: MISC 900010001000a8c Jun 14 23:47:43 tent kernel: EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1749937663 SOCKET 1 APIC 20 Jun 14 23:47:43 tent kernel: EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 (channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 - area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0) ```
rarias commented 2025-06-17 13:00:01 +02:00
Author
Owner
Copy Link

Here is the info we have:

CPU_SrcID#1_Ha#1_Chan#1_DIMM#0

channel:1
page:0x1cb5134
offset:0x0
grain:32
syndrome:0x0
area:DRAM
err_code:0008:00c1
socket:1
ha:1
channel_mask:2
rank:255
row:0xc6a1
col:0x338
bank_addr:2
bank_group:0

Here is the info we have: CPU_SrcID#1_Ha#1_Chan#1_DIMM#0 channel:1 page:0x1cb5134 offset:0x0 grain:32 syndrome:0x0 area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:255 row:0xc6a1 col:0x338 bank_addr:2 bank_group:0
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
ceph
ci
config
cross
hw
io
kernel
mpi
net
nix
ops
repair
slurm
No Label
hw
kernel
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
abonerib arocanon varcila rarias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rarias/jungle#117
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?