[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] RE: XenVif div by zero on Tx path after resume.
-----Original Message----- From: win-pv-devel <win-pv-devel-bounces@xxxxxxxxxxxxxxxxxxxx> On Behalf Of Durrant, Paul Sent: 22 April 2022 13:32 To: paul@xxxxxxx; win-pv-devel@xxxxxxxxxxxxxxxxxxxx Subject: Re: XenVif div by zero on Tx path after resume. > Oh ffs... the call to KeLowerIrql(DISPATCH_LEVEL) is being done at the end of > SyncEnableInterrupts() rather than after the late callbacks at the beginning > of SyncRelease(). That will be the problem... and now I need to dig for why > it is like that. Yes, I thought something was a bit wrong. On the running machine, I notice all the CPU's are at IRQL zero (small sample here of a few CPU's of 32), and a dump of the running threads on the system, one of which points fairly conclusively to SyncRelease: 31: kd> !pcr 0 KPCR for Processor 0 at fffff8061205a000: Major 1 Minor 1 NtTib.ExceptionList: fffff80612169fb0 NtTib.StackBase: fffff80612168000 NtTib.StackLimit: 0000000000000000 NtTib.SubSystemTib: fffff8061205a000 NtTib.Version: 000000001205a180 NtTib.UserPointer: fffff8061205a870 NtTib.SelfTib: 000000cf28cf3000 SelfPcr: 0000000000000000 Prcb: fffff8061205a180 Irql: 0000000000000000 IRR: 0000000000000000 IDR: 0000000000000000 InterruptMode: 0000000000000000 IDT: 0000000000000000 GDT: 0000000000000000 TSS: 0000000000000000 CurrentThread: fffff8061753fb80 NextThread: ffffe084d02d9080 IdleThread: fffff8061753fb80 DpcQueue: Unable to read nt!_KDPC_DATA.DpcListHead.Flink @ fffff8061205d4c0 31: kd> !pcr 1 KPCR for Processor 1 at ffff9781f2beb000: Major 1 Minor 1 NtTib.ExceptionList: ffff9781f2bfcfb0 NtTib.StackBase: ffff9781f2bfb000 NtTib.StackLimit: 0000000000000000 NtTib.SubSystemTib: ffff9781f2beb000 NtTib.Version: 00000000f2beb180 NtTib.UserPointer: ffff9781f2beb870 NtTib.SelfTib: 000000e761342000 SelfPcr: 0000000000000000 Prcb: ffff9781f2beb180 Irql: 0000000000000000 IRR: 0000000000000000 IDR: 0000000000000000 InterruptMode: 0000000000000000 IDT: 0000000000000000 GDT: 0000000000000000 TSS: 0000000000000000 CurrentThread: ffffe084d1164080 NextThread: 0000000000000000 IdleThread: ffff9781f2bf8040 DpcQueue: Unable to read nt!_KDPC_DATA.DpcListHead.Flink @ ffff9781f2bee4c0 31: kd> !pcr 2 KPCR for Processor 2 at ffff9781f2e87000: Major 1 Minor 1 NtTib.ExceptionList: ffff9781f2e98fb0 NtTib.StackBase: ffff9781f2e97000 NtTib.StackLimit: 0000000000000000 NtTib.SubSystemTib: ffff9781f2e87000 NtTib.Version: 00000000f2e87180 NtTib.UserPointer: ffff9781f2e87870 NtTib.SelfTib: 000000855f4a4000 SelfPcr: 0000000000000000 Prcb: ffff9781f2e87180 Irql: 0000000000000000 IRR: 0000000000000000 IDR: 0000000000000000 InterruptMode: 0000000000000000 IDT: 0000000000000000 GDT: 0000000000000000 TSS: 0000000000000000 CurrentThread: ffff9781f2e94040 NextThread: 0000000000000000 IdleThread: ffff9781f2e94040 DpcQueue: Unable to read nt!_KDPC_DATA.DpcListHead.Flink @ ffff9781f2e8a4c0 31: kd> !thread ffffe084d1164080 THREAD ffffe084d1164080 Cid 0ef8.1bec Teb: 000000e761342000 Win32Thread: 0000000000000000 RUNNING on processor 1 Impersonation token: ffffc009fb7100f0 (Level Impersonation) Owning Process ffffe084d00b0240 Image: svchost.exe Attached Process N/A Image: N/A Wait Start TickCount 6384 Ticks: 0 Context Switch Count 207 IdealProcessor: 0 UserTime 00:00:00.031 KernelTime 00:00:00.015 Win32 Start Address 0x00007ff9eb93bfc0 Stack Init ffff958818c67bf0 Current ffff958818c672f0 Base ffff958818c68000 Limit ffff958818c61000 Call 0000000000000000 Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5 Child-SP RetAddr : Args to Child : Call Site ffff9588`18c67520 fffff806`16b0b419 : ffffc009`fb710000 ffffe084`c96c2b01 ffff9588`00000001 ffffe084`d127bea0 : nt!SeCaptureSecurityDescriptor+0x4dd ffff9588`18c67620 fffff806`16b0af33 : 00000000`00000002 ffffc009`fb876060 00000000`00000001 fffff806`16f851d9 : nt!SeAccessCheckByTypeWithAdminlessChecks+0x4c9 ffff9588`18c678e0 fffff806`16b0d177 : 0000020b`6aef23d0 00000000`00000000 00000000`00000cb4 00000000`00000001 : nt!SeAccessCheckByType+0x113 ffff9588`18c67980 fffff806`16c28c35 : 00000000`00000000 00000000`00000000 0000020b`6aef23d0 00000000`00000000 : nt!NtAccessCheck+0x57 ffff9588`18c679f0 00007ff9`eb9ceef4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25 (TrapFrame @ ffff9588`18c67a60) 000000e7`61bfdd48 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ff9`eb9ceef4 31: kd> !thread ffffe084cbb5e040 THREAD ffffe084cbb5e040 Cid 0004.02cc Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 13 Not impersonating DeviceMap ffffc009f5a366bf Owning Process ffffe084c96b4040 Image: System Attached Process N/A Image: N/A Wait Start TickCount 5180 Ticks: 1204 (0:00:00:18.812) Context Switch Count 3 IdealProcessor: 19 UserTime 00:00:00.000 KernelTime 00:00:18.812 Win32 Start Address xenbus!ThreadFunction (0xfffff806187fa850) Stack Init ffff958815267bf0 Current ffff9588152676f0 Base ffff958815268000 Limit ffff958815261000 Call 0000000000000000 Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5 Child-SP RetAddr : Args to Child : Call Site (Inline Function) --------`-------- : --------`-------- --------`-------- --------`-------- --------`-------- : xenbus!SyncRelease+0xc4 (Inline Function @ fffff806`187fa2e1) [E:\jenkins\workspace\win-xenbus_master\local\src\xenbus\sync.c @ 460] ffff9588`152679f0 fffff806`187eb206 : ffffe084`cbb09a00 ffffe084`001f0000 00000000`00989680 00000000`00989680 : xenbus!SuspendTrigger+0x5d1 [E:\jenkins\workspace\win-xenbus_master\local\src\xenbus\suspend.c @ 288] ffff9588`15267aa0 fffff806`187fa86a : 5d158d48`00642464 48244489`48ffa778 ffffe084`cba38ce0 244c8900`00008824 : xenbus!FdoSuspend+0x1a6 [E:\jenkins\workspace\win-xenbus_master\local\src\xenbus\fdo.c @ 1658] ffff9588`15267b40 fffff806`16ae83f5 : ffffe084`cbb5e040 fffff806`187fa850 ffffe084`cba38ce0 20246483`48002824 : xenbus!ThreadFunction+0x1a [E:\jenkins\workspace\win-xenbus_master\local\src\xenbus\thread.c @ 109] ffff9588`15267b70 fffff806`16c1a868 : ffff9781`f3dea180 ffffe084`cbb5e040 fffff806`16ae83a0 3d803745`8948c433 : nt!PspSystemThreadStartup+0x55 ffff9588`15267bc0 00000000`00000000 : ffff9588`15268000 ffff9588`15261000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x28 31: kd> !thread ffffe084d12c4080 THREAD ffffe084d12c4080 Cid 110c.087c Teb: 0000000000d64000 Win32Thread: ffffe084d1a5fbd0 RUNNING on processor 15 Not impersonating DeviceMap ffffc009f5a366bf Owning Process ffffe084d01820c0 Image: xenguestagent.exe Attached Process N/A Image: N/A Wait Start TickCount 6383 Ticks: 1 (0:00:00:00.015) Context Switch Count 7 IdealProcessor: 27 UserTime 00:00:00.000 KernelTime 00:00:00.000 Win32 Start Address 0x00007ff9eb93bfc0 Stack Init ffff958816e2fbf0 Current ffff958816e2f350 Base ffff958816e30000 Limit ffff958816e29000 Call 0000000000000000 Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5 Child-SP RetAddr : Args to Child : Call Site ffff9588`16e2f670 fffff806`17242239 : 00000000`00000000 01000000`00100000 00000000`00000000 00000000`00000000 : nt!ExFreeHeapPool+0x1c8 ffff9588`16e2f770 fffff806`16f121b7 : 00000000`1e1b5180 ffffe084`c9818c60 00000000`1ed2c0e8 ffffc009`fb874be0 : nt!ExFreePool+0x9 ffff9588`16e2f7a0 fffff806`16f701e2 : ffffc009`00000000 00000000`00000000 ffffc009`fb874be0 00000000`1ed2c0e8 : nt!AlpcMessageDestroyProcedure+0x47 ffff9588`16e2f7d0 fffff806`16f6fdc2 : 00000000`1e1b5180 00000000`fe000000 ffffc009`fb874be0 00000000`fe000000 : nt!AlpcpUnlockBlob+0xc2 ffff9588`16e2f810 fffff806`16f6dc46 : ffffe084`c9818c60 ffff9588`00020000 00000000`1e1b5180 00000000`1c614c28 : nt!AlpcpProcessSynchronousRequest+0x562 ffff9588`16e2f930 fffff806`16c28c35 : ffffe084`d12c4080 ffff9588`16e2fae0 00000000`1ed2c0b8 ffff9588`16e2fa08 : nt!NtAlpcSendWaitReceivePort+0x1d6 ffff9588`16e2f9f0 00007ff9`eb9d0064 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25 (TrapFrame @ ffff9588`16e2fa60) 00000000`1ed2c098 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ff9`eb9d0064 31: kd> !thread ffffe084d0843040 THREAD ffffe084d0843040 Cid 15b8.15e8 Teb: 000000f9f5604000 Win32Thread: 0000000000000000 RUNNING on processor 16 Not impersonating DeviceMap ffffc009f5a366bf Owning Process ffffe084d083e280 Image: WmiPrvSE.exe Attached Process N/A Image: N/A Wait Start TickCount 6384 Ticks: 0 Context Switch Count 1836 IdealProcessor: 22 UserTime 00:00:00.093 KernelTime 00:00:00.031 Win32 Start Address 0x00007ff9eb93bfc0 Stack Init ffff9588182b7bf0 Current ffff9588182b7350 Base ffff9588182b8000 Limit ffff9588182b1000 Call 0000000000000000 Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5 Child-SP RetAddr : Args to Child : Call Site 000000f9`f5a7ca38 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ff9`eb9be2e2 31: kd> !thread ffffe084d02e1080 THREAD ffffe084d02e1080 Cid 0ef8.1d0c Teb: 000000e76134e000 Win32Thread: 0000000000000000 RUNNING on processor 19 Not impersonating DeviceMap ffffc009f5a366bf Owning Process ffffe084d00b0240 Image: svchost.exe Attached Process N/A Image: N/A Wait Start TickCount 6383 Ticks: 1 (0:00:00:00.015) Context Switch Count 45 IdealProcessor: 17 UserTime 00:00:00.000 KernelTime 00:00:00.000 Win32 Start Address 0x00007ff9d7310e90 Stack Init ffff95881895fbf0 Current ffff95881895f350 Base ffff958818960000 Limit ffff958818959000 Call 0000000000000000 Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5 Child-SP RetAddr : Args to Child : Call Site 000000e7`624ff778 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ff9`eb9be2e2 31: kd> !thread ffffe084d0411080 THREAD ffffe084d0411080 Cid 1004.129c Teb: 000000855f4a6000 Win32Thread: 0000000000000000 RUNNING on processor 1f Not impersonating DeviceMap ffffc009f5a366bf Owning Process ffffe084d01930c0 Image: windowsclatsdaemon-x64.exe Attached Process N/A Image: N/A Wait Start TickCount 1246 Ticks: 5138 (0:00:01:20.281) Context Switch Count 353 IdealProcessor: 31 UserTime 00:00:00.015 KernelTime 00:00:00.031 Win32 Start Address 0x0000000000c654a0 Stack Init ffff958817ca2bf0 Current ffff958817ca2400 Base ffff958817ca3000 Limit ffff958817c9c000 Call 0000000000000000 Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5 Child-SP RetAddr : Args to Child : Call Site ffff9588`17ca1748 fffff806`19f9084a : 00000000`0000dead fffff806`19fa7c50 fffff806`19fa7c70 00000000`0000144e : nt!KeBugCheckEx (Inline Function) --------`-------- : --------`-------- --------`-------- --------`-------- --------`-------- : xenvif!__Bug+0x24 (Inline Function @ fffff806`19f9084a) [E:\jenkins\workspace\nvif_private_martinhar_CA-355670\local\src\xenvif\assert.h @ 49] ffff9588`17ca1750 fffff806`19f9903a : ffffe084`cd6e19f0 00000000`c0000001 00000000`000000ca 00000000`00000000 : xenvif!TransmitterQueuePacket+0x14a [E:\jenkins\workspace\nvif_private_martinhar_CA-355670\local\src\xenvif\transmitter.c @ 5198] ffff9588`17ca17c0 fffff806`1ab2702b : ffffffff`fffffffe 00000000`0000000c 00000000`00000000 00000000`00000000 : xenvif!VifTransmitterQueuePacket+0x3ea [E:\jenkins\workspace\nvif_private_martinhar_CA-355670\local\src\xenvif\vif.c @ 439] (Inline Function) --------`-------- : --------`-------- --------`-------- --------`-------- --------`-------- : xennet!__TransmitterSendNetBufferList+0x1d8 (Inline Function @ fffff806`1ab2702b) [E:\jenkins\workspace\win-xennet_master\local\src\xennet\transmitter.c @ 329] ffff9588`17ca1830 fffff806`185a5e7f : ffffe084`d0132010 ffffe084`d2297644 00000000`00000000 00000000`00000018 : xennet!TransmitterSendNetBufferLists+0x27b [E:\jenkins\workspace\win-xennet_master\local\src\xennet\transmitter.c @ 384] ffff9588`17ca1900 fffff806`185a5d6e : 00000000`00000000 ffffe084`d22972c0 00000000`00000000 00000000`00000002 : NDIS!ndisMSendNBLToMiniportInternal+0xff ffff9588`17ca19d0 fffff806`185a8062 : fffff806`1745ee00 00000000`0012a00f 00000000`00001080 fffff806`16b3f01c : NDIS!ndisMSendNBLToMiniport+0xe ffff9588`17ca1a10 fffff806`185a2290 : 00000000`00000002 ffff9588`17ca1ae9 ffffe084`cd5ae800 fffffb80`00000000 : NDIS!ndisInvokeNextSendHandler+0x42 ffff9588`17ca1a60 fffff806`19106471 : ffffe084`d22972c0 ffffe084`d22972c0 00000014`00000000 ffff9588`00000001 : NDIS!NdisSendNetBufferLists+0x5f0 ffff9588`17ca1b50 fffff806`19105d4b : ffffe084`cd1620c0 00000000`00000014 ffffe084`cbf6d8a0 fffff806`00000006 : tcpip!IpNlpFastContinueSendDatagrams+0x551 ffff9588`17ca1ca0 fffff806`1910360a : ffffe084`d03a57e0 1a000001`00000014 00000000`00000000 ffffe084`cf886720 : tcpip!IpNlpFastSendDatagram+0x23b ffff9588`17ca1d60 fffff806`19101bcb : 00000000`3b7599ec 00000000`00989680 ffffe084`cd13e168 00000000`00000000 : tcpip!TcpTcbSend+0x141a ffff9588`17ca20b0 fffff806`19101865 : 00000000`00000000 00000000`00000000 00000000`00000000 ffff9588`17ca26d0 : tcpip!TcpEnqueueTcbSend+0x34b ffff9588`17ca21d0 fffff806`16b135d8 : 00000000`00000000 00000000`00000000 00000000`00000000 ffffe084`cbd6b8f0 : tcpip!TcpTlConnectionSendCalloutRoutine+0x25 ffff9588`17ca2210 fffff806`16b1354d : fffff806`19101840 ffff9588`17ca22f0 ffffe084`d069f878 00000000`00000000 : nt!KeExpandKernelStackAndCalloutInternal+0x78 ffff9588`17ca2280 fffff806`1915de0d : 00000000`00000000 00000000`00000000 00000000`0000018b 00000000`00000000 : nt!KeExpandKernelStackAndCalloutEx+0x1d ffff9588`17ca22c0 fffff806`19a2b266 : ffffe084`d069f878 ffffe084`d05645a0 00000000`00000018 ffffe084`d069f878 : tcpip!TcpTlConnectionSend+0x8d ffff9588`17ca2330 fffff806`19a2ae7b : ffffe084`d05645a0 ffff9588`17ca2ae0 ffffe084`d069f670 ffffe084`d06336e0 : afd!AfdTLStartBufferedVcSend+0xb2 ffff9588`17ca23f0 fffff806`19a27d87 : ffffe084`d069f878 ffffe084`d05645a0 ffff9588`00000018 ffff9588`17ca28c8 : afd!AfdFastConnectionSend+0x293 ffff9588`17ca24c0 fffff806`16f936db : 00000000`00000284 00000000`00000000 00000000`00000000 ffffe084`d04c9d90 : afd!AfdFastIoDeviceControl+0xf27 ffff9588`17ca2850 fffff806`16f93ce6 : 00000000`00000000 00000000`00000000 00000000`00000000 000000c0`00134368 : nt!IopXxxControlFile+0x6fb ffff9588`17ca2980 fffff806`16c28c35 : ffffe084`d0324f60 00000000`00000000 00000085`00000000 ffffe084`d04101c0 : nt!NtDeviceIoControlFile+0x56 ffff9588`17ca29f0 00007ff9`eb9cefd4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25 (TrapFrame @ ffff9588`17ca2a60) 00000085`601ffb68 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ff9`eb9cefd4 31: kd>
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |