[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: xen-blkfront crash on xl block-detach of not fully attached device


  • To: Jason Andryuk <jandryuk@xxxxxxxxx>
  • From: Roger Pau Monné <roger.pau@xxxxxxxxxx>
  • Date: Thu, 12 May 2022 15:59:42 +0200
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iUVnq91lsYz9R+1UWwnN5ZQ+HeoMQMcVuT0iWoZwXNQ=; b=oV0YNhB8AOFT89Cc3TPwqXb+pFNEgt2kJTD2DSaMfnIC5nlO8HjSqL/V3Cd8W+WY4aJDw9yaKA9ZtO/4krvSOEE0WF+BS1jI8YB4+MMSJxyLV0EjSBvmqZhOf8uGlUaXImBfRCRdu8/vMcDKfVBPJHYc6Q7VLKq/LBOvK9gTs5YETW/OSeLEqZIDFEloWrCpuqyp4ujnDWAXx1aSdwjDNx/RYAm8AqBgguqGwjYpJg/cKm8jiaHhjViB/O7M5uCCBvf7V4tPo+f7q/HJ4SIO7RD6539dC41wO76baqxffUFlYRNr+/6bmbTGbjFqonie7/CYvvMLi/8cK9CGz1YY0A==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Wej1ryreZ6YrlH2+y+pjEcq5vhMETZo8BQ/YxJz3xPTMlpU/62GusYE5pNsMztUH+0IDV/m5nzVShgC/ClvRhkspoSyMe+Wn7Z+anZezPDZjzTPoHXXvMBKE4wkkECm5SpM+FxhsifRCnWiu7rBXatai2UMHMfm3gQkJ1Sh1xqGSgrqpuCYPdAQpiHVuCkS9l+PFAO2WVzsPKyxYKANA9+ITkM5XZSK88Wl39aEboxEKPj3brFEDryhJZkL0IyFNysu7NuB1pXvrv2AAAYv8SBUA9qJ7QirJjsE2oTHjBg7ZqYeIUXMlaMy2E0xLIXGtDWjzr/EVANe3M+yVCB3+6w==
  • Authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=citrix.com;
  • Cc: Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>, xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • Delivery-date: Thu, 12 May 2022 13:59:56 +0000
  • Ironport-data: A9a23:r02z2ajBZdmSlP4wCsU95zd9X161YhEKZh0ujC45NGQN5FlHY01je htvXGCEOq7cYjPwfd5zbd7j/U1QvZDdzdNlTwZprSxkFC4b9cadCdqndUqhZCn6wu8v7a5EA 2fyTvGacajYm1eF/k/F3oDJ9CU6jefSLlbFILas1hpZHGeIcw98z0M68wIFqtQw24LhXlrX4 YmaT/D3YzdJ5RYlagr41IrbwP9flKyaVOQw5wFWiVhj5TcyplFNZH4tDfjZw0jQG+G4KtWSV efbpIxVy0uCl/sb5nFJpZ6gGqECaua60QFjERO6UYD66vRJjnRaPqrWqJPwwKqY4tmEt4kZ9 TlDiXC/YTsGAqzdoeQBaAF/Dn5SN7155L/VI3fq5KR/z2WeG5ft69NHKRhueKc+paNwC2wI8 uEEIjcQaBzFn/ix3L+wVuhrgIIkMdXvO4Qc/HpnyFk1D95/GcyFH/qMuocehW9o7ixNNa+2i 84xcz1gYQ6GexRSElwWFIg/jKGjgXyXnzhw9wvN/vRoszG7IApZ3KrdKevvWYCzStR8l2S9h XOfz0brK0RPXDCY4X/fmp62vcfNlCX3QpMPF5W38/drhBuYwWl7IBgfT16yu/S6ok+4RdNEK kYQ9zYurK4970iiRJ/2WBjQiG6JuFsQVsRdF8U+6RqR0ezE7gCBHG8GQzVdLts8u6cLqScC0 1aIm5bjA2NpubjMEXaFrO7M9XW1JDQfKnIEaWkcVwwZ7tL/oYY1yBXSUtJkF63zhdrwcd3t/ w23QOEFr+17paY2O2+TpzgrXxrESkD1czMI
  • Ironport-hdrordr: A9a23:+IgSCK48nUbP7MzXRgPXwVqBI+orL9Y04lQ7vn2ZFiY5TiXIra qTdaogviMc6Ax/ZJjvo6HkBEClewKlyXcT2/hrAV7CZniehILMFu1fBOTZowEIdxeOldK1kJ 0QCZSWa+eAcmSS7/yKhzVQeuxIqLfnzEnrv5a5854Ed3AXV0gK1XYcNu/0KDwVeOEQbqBJaa Z0q/A37gaISDAyVICWF3MFV+/Mq5nik4/nWwcPA1oC5BOVhT2lxbbmG1zAty1uGA9n8PMHyy zoggb57qKsv7WSzQLd7Xba69BzlMH6wtVOKcSQgow+KynqiCyveIN9Mofy9AwdkaWK0hIHgd PMqxAvM4Ba7G7QRHi8pV/X1wzpwF8Vmgvf4G7dpUGmjd3yRTo8BcYEr5leaAHl500pu8w5+L 5X3kqC3qAnQi/orWDY3ZzlRhtqnk27rT4JiugIlUFSVoMYdft4sZEfxkVIC50NdRiKpLzPKN MeTf002cwmMW9zNxvizypSKZ2XLzkO9y69MwY/Upf/6UkVoJh7p3FosfD30E1wsa7VcKM0lt gsAp4Y6o2mcfVmHZ6VJN1xNvdfWVa9Ny4lDgqpUCfaPZBCHU7xgLjKx5hwzN2WWfUzvekPcd L6IRlliVI=
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On Thu, May 12, 2022 at 08:47:01AM -0400, Jason Andryuk wrote:
> On Wed, May 11, 2022 at 3:25 PM Marek Marczykowski-Górecki
> <marmarek@xxxxxxxxxxxxxxxxxxxxxx> wrote:
> >
> > Hi,
> >
> > The reproducer is trivial:
> >
> > [user@dom0 ~]$ sudo xl block-attach work backend=sys-usb vdev=xvdi 
> > target=/dev/sdz
> > [user@dom0 ~]$ xl block-list work
> > Vdev  BE  handle state evt-ch ring-ref BE-path
> > 51712 0   241    4     -1     -1       /local/domain/0/backend/vbd/241/51712
> > 51728 0   241    4     -1     -1       /local/domain/0/backend/vbd/241/51728
> > 51744 0   241    4     -1     -1       /local/domain/0/backend/vbd/241/51744
> > 51760 0   241    4     -1     -1       /local/domain/0/backend/vbd/241/51760
> > 51840 3   241    3     -1     -1       /local/domain/3/backend/vbd/241/51840
> >                  ^ note state, the /dev/sdz doesn't exist in the backend
> >
> > [user@dom0 ~]$ sudo xl block-detach work xvdi
> > [user@dom0 ~]$ xl block-list work
> > Vdev  BE  handle state evt-ch ring-ref BE-path
> > work is an invalid domain identifier
> >
> > And its console has:
> >
> > BUG: kernel NULL pointer dereference, address: 0000000000000050
> > #PF: supervisor read access in kernel mode
> > #PF: error_code(0x0000) - not-present page
> > PGD 80000000edebb067 P4D 80000000edebb067 PUD edec2067 PMD 0
> > Oops: 0000 [#1] PREEMPT SMP PTI
> > CPU: 1 PID: 52 Comm: xenwatch Not tainted 5.16.18-2.43.fc32.qubes.x86_64 #1
> > RIP: 0010:blk_mq_stop_hw_queues+0x5/0x40
> > Code: 00 48 83 e0 fd 83 c3 01 48 89 85 a8 00 00 00 41 39 5c 24 50 77 c0 5b 
> > 5d 41 5c 41 5d c3 c3 0f 1f 80 00 00 00 00 0f 1f 44 00 00 <8b> 47 50 85 c0 
> > 74 32 41 54 49 89 fc 55 53 31 db 49 8b 44 24 48 48
> > RSP: 0018:ffffc90000bcfe98 EFLAGS: 00010293
> > RAX: ffffffffc0008370 RBX: 0000000000000005 RCX: 0000000000000000
> > RDX: 0000000000000000 RSI: 0000000000000005 RDI: 0000000000000000
> > RBP: ffff88800775f000 R08: 0000000000000001 R09: ffff888006e620b8
> > R10: ffff888006e620b0 R11: f000000000000000 R12: ffff8880bff39000
> > R13: ffff8880bff39000 R14: 0000000000000000 R15: ffff88800604be00
> > FS:  0000000000000000(0000) GS:ffff8880f3300000(0000) knlGS:0000000000000000
> > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 0000000000000050 CR3: 00000000e932e002 CR4: 00000000003706e0
> > Call Trace:
> >  <TASK>
> >  blkback_changed+0x95/0x137 [xen_blkfront]
> >  ? read_reply+0x160/0x160
> >  xenwatch_thread+0xc0/0x1a0
> >  ? do_wait_intr_irq+0xa0/0xa0
> >  kthread+0x16b/0x190
> >  ? set_kthread_struct+0x40/0x40
> >  ret_from_fork+0x22/0x30
> >  </TASK>
> > Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device 
> > snd_timer snd soundcore ipt_REJECT nf_reject_ipv4 xt_state xt_conntrack 
> > nft_counter nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 
> > nf_defrag_ipv4 nft_compat nf_tables nfnetlink intel_rapl_msr 
> > intel_rapl_common crct10dif_pclmul crc32_pclmul crc32c_intel 
> > ghash_clmulni_intel xen_netfront pcspkr xen_scsiback target_core_mod 
> > xen_netback xen_privcmd xen_gntdev xen_gntalloc xen_blkback xen_evtchn 
> > ipmi_devintf ipmi_msghandler fuse bpf_preload ip_tables overlay xen_blkfront
> > CR2: 0000000000000050
> > ---[ end trace 7bc9597fd06ae89d ]---
> > RIP: 0010:blk_mq_stop_hw_queues+0x5/0x40
> > Code: 00 48 83 e0 fd 83 c3 01 48 89 85 a8 00 00 00 41 39 5c 24 50 77 c0 5b 
> > 5d 41 5c 41 5d c3 c3 0f 1f 80 00 00 00 00 0f 1f 44 00 00 <8b> 47 50 85 c0 
> > 74 32 41 54 49 89 fc 55 53 31 db 49 8b 44 24 48 48
> > RSP: 0018:ffffc90000bcfe98 EFLAGS: 00010293
> > RAX: ffffffffc0008370 RBX: 0000000000000005 RCX: 0000000000000000
> > RDX: 0000000000000000 RSI: 0000000000000005 RDI: 0000000000000000
> > RBP: ffff88800775f000 R08: 0000000000000001 R09: ffff888006e620b8
> > R10: ffff888006e620b0 R11: f000000000000000 R12: ffff8880bff39000
> > R13: ffff8880bff39000 R14: 0000000000000000 R15: ffff88800604be00
> > FS:  0000000000000000(0000) GS:ffff8880f3300000(0000) knlGS:0000000000000000
> > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 0000000000000050 CR3: 00000000e932e002 CR4: 00000000003706e0
> > Kernel panic - not syncing: Fatal exception
> > Kernel Offset: disabled
> 
> This looks like it may be blkfront_closing() calling
> blk_mq_stop_hw_queues() with info->rq == NULL.  info->rq is only
> assigned in blkfront_connect(), which is called for state 4, but your
> vbd never made it through there.  It seems like blkfront_closing()
> should NULL check info->rq and info->gd before using them.

Care to send a patch? :)

Thanks, Roger.



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.