Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

handle_bdev_mount_event has a null pointer。kernel panic #365

Open
hemuke opened this issue Mar 22, 2024 · 8 comments
Open

handle_bdev_mount_event has a null pointer。kernel panic #365

hemuke opened this issue Mar 22, 2024 · 8 comments

Comments

@hemuke
Copy link

hemuke commented Mar 22, 2024

[ 126.460209] TECH PREVIEW: Overlay filesystem may not be fully supported.
Please review provided documentation for limitations.
[ 126.637468] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 126.637520] IP: [] handle_bdev_mount_event+0x67/0x1b0 [dattobd]
[ 126.637566] PGD 0
[ 126.637582] Oops: 0000 [#1] SMP
[ 126.637604] Modules linked in: overlay(T) fuse btrfs raid6_pq xor vfat msdos fat ext4 mbcache jbd2 nls_utf8 isofs dm_mod kvm_intel kvm irqbypass aesni_intel lrw gf128mul glue_helper ablk_helper sg ppdev cryptd virtio_balloon pcspkr joydev parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci virtio_ring drm_panel_orientation_quirks serio_raw virtio floppy dattobd(OE)
[ 126.637989] CPU: 0 PID: 2629 Comm: dockerd Kdump: loaded Tainted: G OE ------------ T 3.10.0-957.el7.x86_64 #1
[ 126.638057] Hardware name: Huayun ArStack, BIOS 1.11.0-2.el7 04/01/2014
[ 126.638093] task: ffff8b6592a49040 ti: ffff8b65a3608000 task.ti: ffff8b65a3608000
[ 126.638133] RIP: 0010:[] [] handle_bdev_mount_event+0x67/0x1b0 [dattobd]
[ 126.638189] RSP: 0018:ffff8b65a360bed8 EFLAGS: 00010246
[ 126.638218] RAX: 00000000fffffffe RBX: 0000000000000000 RCX: 0000000000000000
[ 126.638256] RDX: 0000000000003dc9 RSI: 0000000000000000 RDI: 0000000000000000
[ 126.638293] RBP: ffff8b65a360bf10 R08: 000000000001f1a0 R09: ffffffff88452abd
[ 126.638331] R10: ffff8b65ae61f1a0 R11: fffff8a400d75600 R12: ffff8b65a360bf24
[ 126.638368] R13: 000000c000a24380 R14: 0000000000000000 R15: 0000000000000000
[ 126.638405] FS: 00007f01cf7fe700(0000) GS:ffff8b65ae600000(0000) knlGS:0000000000000000
[ 126.638447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 126.638478] CR2: 0000000000000000 CR3: 0000000035fb4000 CR4: 00000000001206f0
[ 126.638519] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 126.638558] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 126.638595] Call Trace:
[ 126.638615] [] ftrace_sys_umount+0x39/0x90 [dattobd]
[ 126.638654] [] system_call_fastpath+0x22/0x27
[ 126.638685] Code: 31 d2 48 8d 4d c8 83 e3 08 0f 94 c2 4c 89 ee bf 9c ff ff ff e8 6b 1a 33 c8 8b 35 2d 91 00 00 85 f6 0f 85 00 01 00 00 48 8b 7d c8 <48> 8b 07 48 39 45 d0 74 50 8b 0d 12 91 00 00 85 c9 0f 85 0b 01
[ 126.638936] RIP [] handle_bdev_mount_event+0x67/0x1b0 [dattobd]
[ 126.638981] RSP
[ 126.639009] CR2: 0000000000000000

datto may not support backing up docker on VMs
@Swistusmen
Copy link
Contributor

Hi,
Honestly speaking I have never tested backing VM with docker running on it, gonna check it in spare time. Can you pls provide more details like OS, kernel version.

@hemuke
Copy link
Author

hemuke commented Mar 22, 2024

[root@host-192-168-126-181 ~]# uname -r
3.10.0-957.el7.x86_64
[root@host-192-168-126-181 ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)

Hi, Honestly speaking I have never tested backing VM with docker running on it, gonna check it in spare time. Can you pls provide more details like OS, kernel version.

yes thank you swistusmen
CentOS Linux release 7.9.2009 (Core)
3.10.0-957.el7.x86_64

@ScottRagen
Copy link

ScottRagen commented Mar 26, 2024 •
edited
Loading

Hi, We have been getting similar errors since upgrading Ubuntu 22.04 LTS to the newest released kernel.
The server still operates, but any Docker queries like docker container list, or even shutting down the server results in unresponsive processes. The containers also do not operate.

Dmesg below:
[ 23.357126] kauditd_printk_skb: 22 callbacks suppressed
[ 23.357127] audit: type=1400 audit(1711417444.953:34): apparmor="STATUS" operation="profile_load" profile="unconfined" name="docker-default" pid=1830 comm="apparmor_parser"
[ 24.391150] BUG: kernel NULL pointer dereference, address: 0000000000000002
[ 24.391686] #PF: supervisor read access in kernel mode
[ 24.392176] #PF: error_code(0x0000) - not-present page
[ 24.392680] PGD 0 P4D 0
[ 24.393180] Oops: 0000 [#1] SMP PTI
[ 24.393654] CPU: 0 PID: 2080 Comm: dockerd Tainted: G OE 5.4.0-174-generic #193-Ubuntu
[ 24.393835] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v1.0 11/26/2012
[ 24.393835] RIP: 0010:handle_bdev_mount_event+0x6c/0x100 [dattobd]
[ 24.393835] Code: 08 48 8d 4d c8 4c 89 e6 0f 94 c2 bf 9c ff ff ff 45 31 c0 e8 56 81 13 e6 8b 15 90 9e 00 00 85 d2 0f 85 67 39 00 00 48 8b 7d c8 <48> 8b 07 48 39 45 d0 74 34 41 bd ed ff ff ff 48 8d 7d c8 e8 3c 09
[ 24.393835] RSP: 0018:ffffabe3c27afea8 EFLAGS: 00010246
[ 24.393835] RAX: 00000000fffffffe RBX: ffffabe3c27afef4 RCX: 000000000010b230
[ 24.393835] RDX: 0000000000000000 RSI: 6c904e326328fd5a RDI: 0000000000000002
[ 24.393835] RBP: ffffabe3c27afee0 R08: ffffabe3c27afd80 R09: ffff9b85ef95f500
[ 24.393835] R10: 00000000fffffffe R11: 00000000006d6873 R12: 000000c0009d6fc0
[ 24.393835] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 24.393835] FS: 00007fb67f7fe700(0000) GS:ffff9b85fc000000(0000) knlGS:0000000000000000
[ 24.393835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 24.393835] CR2: 0000000000000002 CR3: 00000001f0096003 CR4: 00000000003606f0
[ 24.393835] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 24.393835] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 24.393835] Call Trace:
[ 24.393835] ? show_regs.cold+0x1a/0x1f
[ 24.393835] ? __die+0x90/0xd9
[ 24.393835] ? no_context+0x196/0x380
[ 24.393835] ? __bad_area_nosemaphore+0x50/0x1a0
[ 24.393835] ? bad_area_nosemaphore+0x16/0x20
[ 24.393835] ? do_user_addr_fault+0x267/0x450
[ 24.393835] ? putname+0x4a/0x50
[ 24.393835] ? __do_page_fault+0x58/0x90
[ 24.393835] ? do_page_fault+0x2c/0xe0
[ 24.393835] ? page_fault+0x34/0x40
[ 24.393835] ? handle_bdev_mount_event+0x6c/0x100 [dattobd]
[ 24.393835] ? handle_bdev_mount_event+0x5a/0x100 [dattobd]
[ 24.393835] ftrace_ksys_umount+0x3a/0x90 [dattobd]
[ 24.393835] __x64_sys_umount+0x16/0x20
[ 24.393835] do_syscall_64+0x57/0x190
[ 24.393835] entry_SYSCALL_64_after_hwframe+0x5c/0xc1
[ 24.393835] RIP: 0033:0x564a8fbd500e
[ 24.393835] Code: 48 83 ec 38 e8 13 00 00 00 48 83 c4 38 5d c3 cc cc cc cc cc cc cc cc cc cc cc cc cc 49 89 f2 48 89 fa 48 89 ce 48 89 df 0f 05 <48> 3d 01 f0 ff ff 76 15 48 f7 d8 48 89 c1 48 c7 c0 ff ff ff ff 48
[ 24.393835] RSP: 002b:000000c0008b02a8 EFLAGS: 00000202 ORIG_RAX: 00000000000000a6
[ 24.393835] RAX: ffffffffffffffda RBX: 000000c0009d6fc0 RCX: 0000564a8fbd500e
[ 24.393835] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 000000c0009d6fc0
[ 24.393835] RBP: 000000c0008b02e8 R08: 0000000000000000 R09: 0000000000000000
[ 24.393835] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
[ 24.393835] R13: 000000c0005ec800 R14: 000000c000be6ea0 R15: 0000000000000024
[ 24.393835] Modules linked in: aufs overlay nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua binfmt_misc intel_rapl_msr intel_rapl_common rapl serio_raw hyperv_fb hv_balloon joydev mac_hid nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt nf_log_ipv4 nf_log_common ipt_REJECT nf_reject_ipv4
xt_LOG xt_MASQUERADE iptable_nat nf_nat xt_limit xt_addrtype xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 sch_fq_codel ramoops reed_solomon ip6table_filter drm ip6_tables iptable_filter bpfilter msr efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress dattobd(OE) raid10 raid456 async_ra
id6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel hid_generic hid_hyperv hv_netvsc hv_storvsc scsi_transport_fc hv_utils hyperv_keyboard hid aesni_intel crypto_simd cryptd glue_helper hv_vmbus
[ 24.393835] CR2: 0000000000000002
[ 24.393835] ---[ end trace 9a072ae20af90325 ]---
[ 24.393835] RIP: 0010:handle_bdev_mount_event+0x6c/0x100 [dattobd]
[ 24.393835] Code: 08 48 8d 4d c8 4c 89 e6 0f 94 c2 bf 9c ff ff ff 45 31 c0 e8 56 81 13 e6 8b 15 90 9e 00 00 85 d2 0f 85 67 39 00 00 48 8b 7d c8 <48> 8b 07 48 39 45 d0 74 34 41 bd ed ff ff ff 48 8d 7d c8 e8 3c 09
[ 24.393835] RSP: 0018:ffffabe3c27afea8 EFLAGS: 00010246
[ 24.393835] RAX: 00000000fffffffe RBX: ffffabe3c27afef4 RCX: 000000000010b230
[ 24.393835] RDX: 0000000000000000 RSI: 6c904e326328fd5a RDI: 0000000000000002
[ 24.393835] RBP: ffffabe3c27afee0 R08: ffffabe3c27afd80 R09: ffff9b85ef95f500
[ 24.393835] R10: 00000000fffffffe R11: 00000000006d6873 R12: 000000c0009d6fc0
[ 24.393835] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 24.393835] FS: 00007fb67f7fe700(0000) GS:ffff9b85fc000000(0000) knlGS:0000000000000000
[ 24.393835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 24.393835] CR2: 0000000000000002 CR3: 00000001f0096003 CR4: 00000000003606f0
[ 24.393835] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 24.393835] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 24.458683] BUG: kernel NULL pointer dereference, address: 0000000000000002
[ 24.459385] #PF: supervisor read access in kernel mode
[ 24.460064] #PF: error_code(0x0000) - not-present page
[ 24.460715] PGD 0 P4D 0
[ 24.461398] Oops: 0000 [#2] SMP PTI
[ 24.461836] CPU: 0 PID: 1749 Comm: dockerd Tainted: G D OE 5.4.0-174-generic #193-Ubuntu
[ 24.461836] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v1.0 11/26/2012
[ 24.461836] RIP: 0010:handle_bdev_mount_event+0x6c/0x100 [dattobd]
[ 24.461836] Code: 08 48 8d 4d c8 4c 89 e6 0f 94 c2 bf 9c ff ff ff 45 31 c0 e8 56 81 13 e6 8b 15 90 9e 00 00 85 d2 0f 85 67 39 00 00 48 8b 7d c8 <48> 8b 07 48 39 45 d0 74 34 41 bd ed ff ff ff 48 8d 7d c8 e8 3c 09
[ 24.461836] RSP: 0018:ffffabe3c3607ea8 EFLAGS: 00010246
[ 24.461836] RAX: 00000000fffffffe RBX: ffffabe3c3607ef4 RCX: 000000000010b33d
[ 24.461836] RDX: 0000000000000000 RSI: 6cc04e3263288d5a RDI: 0000000000000002
[ 24.461836] RBP: ffffabe3c3607ee0 R08: ffffabe3c3607d80 R09: ffff9b85ef95e840
[ 24.461836] R10: 00000000fffffffe R11: 00000000006d6873 R12: 000000c0009d7260
[ 24.461836] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 24.461836] FS: 00007fb69d5b9700(0000) GS:ffff9b85fc000000(0000) knlGS:0000000000000000
[ 24.461836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 24.461836] CR2: 0000000000000002 CR3: 00000001f0096001 CR4: 00000000003606f0
[ 24.461836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 24.461836] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 24.461836] Call Trace:
[ 24.461836] ? show_regs.cold+0x1a/0x1f
[ 24.461836] ? __die+0x90/0xd9
[ 24.461836] ? no_context+0x196/0x380
[ 24.461836] ? __bad_area_nosemaphore+0x50/0x1a0
[ 24.461836] ? bad_area_nosemaphore+0x16/0x20
[ 24.461836] ? do_user_addr_fault+0x267/0x450
[ 24.461836] ? putname+0x4a/0x50
[ 24.461836] ? __do_page_fault+0x58/0x90
[ 24.461836] ? do_page_fault+0x2c/0xe0
[ 24.461836] ? page_fault+0x34/0x40
[ 24.461836] ? handle_bdev_mount_event+0x6c/0x100 [dattobd]
[ 24.461836] ? handle_bdev_mount_event+0x5a/0x100 [dattobd]
[ 24.461836] ftrace_ksys_umount+0x3a/0x90 [dattobd]
[ 24.461836] __x64_sys_umount+0x16/0x20
[ 24.461836] do_syscall_64+0x57/0x190
[ 24.461836] entry_SYSCALL_64_after_hwframe+0x5c/0xc1
[ 24.461836] RIP: 0033:0x564a8fbd500e
[ 24.461836] Code: 48 83 ec 38 e8 13 00 00 00 48 83 c4 38 5d c3 cc cc cc cc cc cc cc cc cc cc cc cc cc 49 89 f2 48 89 fa 48 89 ce 48 89 df 0f 05 <48> 3d 01 f0 ff ff 76 15 48 f7 d8 48 89 c1 48 c7 c0 ff ff ff ff 48
[ 24.461836] RSP: 002b:000000c000bac2a8 EFLAGS: 00000202 ORIG_RAX: 00000000000000a6
[ 24.461836] RAX: ffffffffffffffda RBX: 000000c0009d7260 RCX: 0000564a8fbd500e
[ 24.461836] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 000000c0009d7260
[ 24.461836] RBP: 000000c000bac2e8 R08: 0000000000000000 R09: 0000000000000000
[ 24.461836] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
[ 24.461836] R13: 000000c000ad9400 R14: 000000c000be7040 R15: 000000000000002a
[ 24.461836] Modules linked in: aufs overlay nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua binfmt_misc intel_rapl_msr intel_rapl_common rapl serio_raw hyperv_fb hv_balloon joydev mac_hid nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt nf_log_ipv4 nf_log_common ipt_REJECT nf_reject_ipv4
xt_LOG xt_MASQUERADE iptable_nat nf_nat xt_limit xt_addrtype xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 sch_fq_codel ramoops reed_solomon ip6table_filter drm ip6_tables iptable_filter bpfilter msr efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress dattobd(OE) raid10 raid456 async_ra
id6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel hid_generic hid_hyperv hv_netvsc hv_storvsc scsi_transport_fc hv_utils hyperv_keyboard hid aesni_intel crypto_simd cryptd glue_helper hv_vmbus
[ 24.461836] CR2: 0000000000000002
[ 24.461836] ---[ end trace 9a072ae20af90326 ]---
[ 24.461836] RIP: 0010:handle_bdev_mount_event+0x6c/0x100 [dattobd]
[ 24.461836] Code: 08 48 8d 4d c8 4c 89 e6 0f 94 c2 bf 9c ff ff ff 45 31 c0 e8 56 81 13 e6 8b 15 90 9e 00 00 85 d2 0f 85 67 39 00 00 48 8b 7d c8 <48> 8b 07 48 39 45 d0 74 34 41 bd ed ff ff ff 48 8d 7d c8 e8 3c 09
[ 24.461836] RSP: 0018:ffffabe3c27afea8 EFLAGS: 00010246
[ 24.461836] RAX: 00000000fffffffe RBX: ffffabe3c27afef4 RCX: 000000000010b230
[ 24.461836] RDX: 0000000000000000 RSI: 6c904e326328fd5a RDI: 0000000000000002
[ 24.461836] RBP: ffffabe3c27afee0 R08: ffffabe3c27afd80 R09: ffff9b85ef95f500
[ 24.461836] R10: 00000000fffffffe R11: 00000000006d6873 R12: 000000c0009d6fc0
[ 24.461836] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 24.461836] FS: 00007fb69d5b9700(0000) GS:ffff9b85fc000000(0000) knlGS:0000000000000000
[ 24.461836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 24.461836] CR2: 0000000000000002 CR3: 00000001f0096001 CR4: 00000000003606f0
[ 24.461836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 24.461836] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 26.166586] BUG: kernel NULL pointer dereference, address: 0000000000000002
[ 26.166613] #PF: supervisor read access in kernel mode
[ 26.166628] #PF: error_code(0x0000) - not-present page
[ 26.166643] PGD 0 P4D 0
[ 26.166652] Oops: 0000 [#3] SMP PTI
[ 26.166665] CPU: 0 PID: 2079 Comm: dockerd Tainted: G D OE 5.4.0-174-generic #193-Ubuntu
[ 26.166686] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v1.0 11/26/2012
[ 26.166742] RIP: 0010:handle_bdev_mount_event+0x6c/0x100 [dattobd]
[ 26.166761] Code: 08 48 8d 4d c8 4c 89 e6 0f 94 c2 bf 9c ff ff ff 45 31 c0 e8 56 81 13 e6 8b 15 90 9e 00 00 85 d2 0f 85 67 39 00 00 48 8b 7d c8 <48> 8b 07 48 39 45 d0 74 34 41 bd ed ff ff ff 48 8d 7d c8 e8 3c 09
[ 26.166812] RSP: 0018:ffffabe3c277fea8 EFLAGS: 00010246
[ 26.166827] RAX: 00000000fffffffe RBX: ffffabe3c277fef4 RCX: 0000000000112350
[ 26.166846] RDX: 0000000000000000 RSI: 6ce04e3263288d5a RDI: 0000000000000002
[ 26.166863] RBP: ffffabe3c277fee0 R08: ffffabe3c277fd80 R09: ffff9b85ef95f200
[ 26.166882] R10: 00000000fffffffe R11: 00000000006d6873 R12: 000000c0009d7ce0
[ 26.166902] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 26.166919] FS: 00007fb67ffff700(0000) GS:ffff9b85fc000000(0000) knlGS:0000000000000000
[ 26.166940] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 26.166956] CR2: 0000000000000002 CR3: 00000001f0096006 CR4: 00000000003606f0
[ 26.166975] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 26.166994] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 26.167014] Call Trace:
[ 26.167028] ? show_regs.cold+0x1a/0x1f
[ 26.167041] ? __die+0x90/0xd9
[ 26.167052] ? no_context+0x196/0x380
[ 26.167065] ? __bad_area_nosemaphore+0x50/0x1a0
[ 26.167078] ? bad_area_nosemaphore+0x16/0x20
[ 26.167091] ? do_user_addr_fault+0x267/0x450
[ 26.167105] ? putname+0x4a/0x50
[ 26.167116] ? __do_page_fault+0x58/0x90
[ 26.167129] ? do_page_fault+0x2c/0xe0
[ 26.167141] ? page_fault+0x34/0x40
[ 26.167153] ? handle_bdev_mount_event+0x6c/0x100 [dattobd]
[ 26.167168] ? handle_bdev_mount_event+0x5a/0x100 [dattobd]
[ 26.167185] ftrace_ksys_umount+0x3a/0x90 [dattobd]
[ 26.167201] __x64_sys_umount+0x16/0x20
[ 26.167214] do_syscall_64+0x57/0x190
[ 26.167224] entry_SYSCALL_64_after_hwframe+0x5c/0xc1
[ 26.167242] RIP: 0033:0x564a8fbd500e
[ 26.167253] Code: 48 83 ec 38 e8 13 00 00 00 48 83 c4 38 5d c3 cc cc cc cc cc cc cc cc cc cc cc cc cc 49 89 f2 48 89 fa 48 89 ce 48 89 df 0f 05 <48> 3d 01 f0 ff ff 76 15 48 f7 d8 48 89 c1 48 c7 c0 ff ff ff ff 48
[ 26.167296] RSP: 002b:000000c0008b42a8 EFLAGS: 00000202 ORIG_RAX: 00000000000000a6
[ 26.167317] RAX: ffffffffffffffda RBX: 000000c0009d7ce0 RCX: 0000564a8fbd500e
[ 26.167334] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 000000c0009d7ce0
[ 26.167353] RBP: 000000c0008b42e8 R08: 0000000000000000 R09: 0000000000000000
[ 26.167372] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
[ 26.167390] R13: 000000c0005ec400 R14: 000000c000be71e0 R15: 0000000000000042
[ 26.167409] Modules linked in: aufs overlay nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua binfmt_misc intel_rapl_msr intel_rapl_common rapl serio_raw hyperv_fb hv_balloon joydev mac_hid nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt nf_log_ipv4 nf_log_common ipt_REJECT nf_reject_ipv4
xt_LOG xt_MASQUERADE iptable_nat nf_nat xt_limit xt_addrtype xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 sch_fq_codel ramoops reed_solomon ip6table_filter drm ip6_tables iptable_filter bpfilter msr efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress dattobd(OE) raid10 raid456 async_ra
id6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel hid_generic hid_hyperv hv_netvsc hv_storvsc scsi_transport_fc hv_utils hyperv_keyboard hid aesni_intel crypto_simd cryptd glue_helper hv_vmbus
[ 26.170395] CR2: 0000000000000002
[ 26.170560] ---[ end trace 9a072ae20af90327 ]---
[ 26.170560] RIP: 0010:handle_bdev_mount_event+0x6c/0x100 [dattobd]
[ 26.170560] Code: 08 48 8d 4d c8 4c 89 e6 0f 94 c2 bf 9c ff ff ff 45 31 c0 e8 56 81 13 e6 8b 15 90 9e 00 00 85 d2 0f 85 67 39 00 00 48 8b 7d c8 <48> 8b 07 48 39 45 d0 74 34 41 bd ed ff ff ff 48 8d 7d c8 e8 3c 09
[ 26.170560] RSP: 0018:ffffabe3c27afea8 EFLAGS: 00010246
[ 26.170560] RAX: 00000000fffffffe RBX: ffffabe3c27afef4 RCX: 000000000010b230
[ 26.170560] RDX: 0000000000000000 RSI: 6c904e326328fd5a RDI: 0000000000000002
[ 26.170560] RBP: ffffabe3c27afee0 R08: ffffabe3c27afd80 R09: ffff9b85ef95f500
[ 26.170560] R10: 00000000fffffffe R11: 00000000006d6873 R12: 000000c0009d6fc0
[ 26.170560] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 26.170560] FS: 00007fb67ffff700(0000) GS:ffff9b85fc000000(0000) knlGS:0000000000000000
[ 26.170560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 26.170560] CR2: 0000000000000002 CR3: 00000001f0096006 CR4: 00000000003606f0
[ 26.170560] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 26.170560] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 26.205537] BUG: kernel NULL pointer dereference, address: 0000000000000002
[ 26.205836] #PF: supervisor read access in kernel mode
[ 26.205836] #PF: error_code(0x0000) - not-present page
[ 26.205836] PGD 0 P4D 0
[ 26.205836] Oops: 0000 [#4] SMP PTI
[ 26.205836] CPU: 0 PID: 2078 Comm: dockerd Tainted: G D OE 5.4.0-174-generic #193-Ubuntu
[ 26.205836] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v1.0 11/26/2012
[ 26.205836] RIP: 0010:handle_bdev_mount_event+0x6c/0x100 [dattobd]
[ 26.205836] Code: 08 48 8d 4d c8 4c 89 e6 0f 94 c2 bf 9c ff ff ff 45 31 c0 e8 56 81 13 e6 8b 15 90 9e 00 00 85 d2 0f 85 67 39 00 00 48 8b 7d c8 <48> 8b 07 48 39 45 d0 74 34 41 bd ed ff ff ff 48 8d 7d c8 e8 3c 09
[ 26.205836] RSP: 0018:ffffabe3c272fea8 EFLAGS: 00010246
[ 26.205836] RAX: 00000000fffffffe RBX: ffffabe3c272fef4 RCX: 00000000001123eb
[ 26.205836] RDX: 0000000000000000 RSI: 6cc04e326328fd5a RDI: 0000000000000002
[ 26.205836] RBP: ffffabe3c272fee0 R08: ffffabe3c272fd80 R09: ffff9b85ef95fbc0
[ 26.205836] R10: 00000000fffffffe R11: 00000000006d6873 R12: 000000c0009d7dc0
[ 26.205836] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 26.205836] FS: 00007fb69cd68700(0000) GS:ffff9b85fc000000(0000) knlGS:0000000000000000
[ 26.205836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 26.205836] CR2: 0000000000000002 CR3: 00000001f0096004 CR4: 00000000003606f0
[ 26.205836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 26.205836] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 26.205836] Call Trace:
[ 26.205836] ? show_regs.cold+0x1a/0x1f
[ 26.205836] ? __die+0x90/0xd9
[ 26.205836] ? no_context+0x196/0x380
[ 26.205836] ? __bad_area_nosemaphore+0x50/0x1a0
[ 26.205836] ? bad_area_nosemaphore+0x16/0x20
[ 26.205836] ? do_user_addr_fault+0x267/0x450
[ 26.205836] ? putname+0x4a/0x50
[ 26.205836] ? __do_page_fault+0x58/0x90
[ 26.205836] ? do_page_fault+0x2c/0xe0
[ 26.205836] ? page_fault+0x34/0x40
[ 26.205836] ? handle_bdev_mount_event+0x6c/0x100 [dattobd]
[ 26.205836] ? handle_bdev_mount_event+0x5a/0x100 [dattobd]
[ 26.205836] ftrace_ksys_umount+0x3a/0x90 [dattobd]
[ 26.205836] __x64_sys_umount+0x16/0x20
[ 26.205836] do_syscall_64+0x57/0x190
[ 26.205836] entry_SYSCALL_64_after_hwframe+0x5c/0xc1
[ 26.205836] RIP: 0033:0x564a8fbd500e
[ 26.205836] Code: 48 83 ec 38 e8 13 00 00 00 48 83 c4 38 5d c3 cc cc cc cc cc cc cc cc cc cc cc cc cc 49 89 f2 48 89 fa 48 89 ce 48 89 df 0f 05 <48> 3d 01 f0 ff ff 76 15 48 f7 d8 48 89 c1 48 c7 c0 ff ff ff ff 48
[ 26.205836] RSP: 002b:000000c000a902a8 EFLAGS: 00000202 ORIG_RAX: 00000000000000a6
[ 26.205836] RAX: ffffffffffffffda RBX: 000000c0009d7dc0 RCX: 0000564a8fbd500e
[ 26.205836] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 000000c0009d7dc0
[ 26.205836] RBP: 000000c000a902e8 R08: 0000000000000000 R09: 0000000000000000
[ 26.205836] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
[ 26.205836] R13: 000000c0005ec000 R14: 000000c000be7380 R15: 0000000000000044
[ 26.205836] Modules linked in: aufs overlay nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua binfmt_misc intel_rapl_msr intel_rapl_common rapl serio_raw hyperv_fb hv_balloon joydev mac_hid nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt nf_log_ipv4 nf_log_common ipt_REJECT nf_reject_ipv4
xt_LOG xt_MASQUERADE iptable_nat nf_nat xt_limit xt_addrtype xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 sch_fq_codel ramoops reed_solomon ip6table_filter drm ip6_tables iptable_filter bpfilter msr efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress dattobd(OE) raid10 raid456 async_ra
id6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel hid_generic hid_hyperv hv_netvsc hv_storvsc scsi_transport_fc hv_utils hyperv_keyboard hid aesni_intel crypto_simd cryptd glue_helper hv_vmbus
[ 26.205836] CR2: 0000000000000002
[ 26.205836] ---[ end trace 9a072ae20af90328 ]---
[ 26.205836] RIP: 0010:handle_bdev_mount_event+0x6c/0x100 [dattobd]
[ 26.205836] Code: 08 48 8d 4d c8 4c 89 e6 0f 94 c2 bf 9c ff ff ff 45 31 c0 e8 56 81 13 e6 8b 15 90 9e 00 00 85 d2 0f 85 67 39 00 00 48 8b 7d c8 <48> 8b 07 48 39 45 d0 74 34 41 bd ed ff ff ff 48 8d 7d c8 e8 3c 09
[ 26.205836] RSP: 0018:ffffabe3c27afea8 EFLAGS: 00010246
[ 26.205836] RAX: 00000000fffffffe RBX: ffffabe3c27afef4 RCX: 000000000010b230
[ 26.205836] RDX: 0000000000000000 RSI: 6c904e326328fd5a RDI: 0000000000000002
[ 26.205836] RBP: ffffabe3c27afee0 R08: ffffabe3c27afd80 R09: ffff9b85ef95f500
[ 26.205836] R10: 00000000fffffffe R11: 00000000006d6873 R12: 000000c0009d6fc0
[ 26.205836] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 26.205836] FS: 00007fb69cd68700(0000) GS:ffff9b85fc000000(0000) knlGS:0000000000000000
[ 26.205836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 26.205836] CR2: 0000000000000002 CR3: 00000001f0096004 CR4: 00000000003606f0
[ 26.205836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 26.205836] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

We also have CentOS Stream 8 servers that completely crash but we are migrating away from these servers, so not as concerned.

@natalia-zelazna natalia-zelazna mentioned this issue Mar 27, 2024
Closed
@natalia-zelazna
Copy link
Contributor

Hi, can you provide your scenario? Was it only on shutdown/reboot or also in other scenarios? I have observed such trace in /var/crash in following scenario:

  1. make a backup
  2. reboot system
    And on such reboot I was able to observe a crash and a dump looking like this one. It should be visible on all distros.

@ScottRagen
Copy link

Hi and thank you for your response.

Yes, it's on bootup it occurs.

However it can occur on a clean start without dattobd loaded. If I load dattobd, start docker, then running " docker run --rm hello-world", the panic shows and the docker command never returns, unresponsive to ctrl+c, and ctrl+z, only kill -9 will terminate the process. It repeats the panic any time a docker command becomes unresponsive, for example when I tried starting it again in the below example.

I have found the following:

• When dattobd is inserted and when docker is stopped, it appears to be placed in a “crashed” state. (process is no longer running, but starting it again causes it to be unresponsive if dattobd is loaded at the time)
• Starting docker in this state whilst dattobd is running causes it to freeze, this includes even after clean rebooting.

So whilst the module is loaded and docker is in a “clean” (responsive) state, we can successfully stop docker, but it appears to put it in this “crashed” state and starting it again causes the freeze if the dattobd module is still loaded, this includes only starting it after a clean reboot.
For example: systemctl stop docker && init 6. On reboot, docker does not respond.

However if I remove the dattobd module before stopping docker, we can successfully start it again afterwards, even with dattobd loaded during the startup, for example:

rmmod dattobd && systemctl stop docker && modprobe dattobd && systemctl start docker

Docker remains responsive.

I don’t think it’s the full answer, because whilst docker is responsive and dattobd is loaded, I could cause it to stop responding by running the command “docker run --rm hello-world”, so it’s not strictly on stop/start that causes issues.

@natalia-zelazna
Copy link
Contributor

Hi, thanks for the explanation.

@AtleeRRN
Copy link

I've found any kernel above 5.15.0-94 causes a kernel panic.

I'd be curious about what versions others are running?

@Swistusmen
Copy link
Contributor

It is going to be solved by #377 (review)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@hemuke @Swistusmen @AtleeRRN @ScottRagen @natalia-zelazna