Xen project Mailing List

Re: Lazily construct slab commit causes BSOD/freeze on xen 4.16

To: "win-pv-devel@xxxxxxxxxxxxxxxxxxxx" <win-pv-devel@xxxxxxxxxxxxxxxxxxxx>

From: Joel Upham <uphamj@xxxxxxxxxxxx>

Date: Sun, 26 Feb 2023 19:42:17 +0000

Accept-language: en-US

Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ainfosec.com; dmarc=pass action=none header.from=ainfosec.com; dkim=pass header.d=ainfosec.com; arc=none

Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector5401; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=NXZhAG0BQCoezTnUTvgpzILRBV5DDSbENTrqSolVDwo=; b=ZQxam597I+25K8uUcUhpl2ltN23WYPffM9oHQMJ/0w3qqan9ie8Wn2MBUOLWTfMAjQajLcJNhAkoFCsjwtH8yKWHjFS9+MwonibWP0zH+w5iOIJfNqV0IxrZJ42bfNY7+mVQBfylSky6eFcqghTLY+ms9uUqsDkDCnQ70G1YjI1NSUK+F6L7xcDDR3/XCcvt7JpXN/TxQa9gIPirk3oSLj6JhyF9uQxjPZGPYB8isGj6yjAJ4+iLBJtpe9jwUyuMF59dmiZx3fRCkEKxZen9jdL6l1W7vEoP6NqT9WzOHwhN5xMBT8ieFKgcRJ+Jpd0mri3/VgOoGdhMRI4Lssf1GQ==

Arc-seal: i=1; a=rsa-sha256; s=arcselector5401; d=microsoft.com; cv=none; b=LiRzWdGWihmJ7mEreaxwvJ0CIVeU5q7SypwZKVv1JEmfNLq6rlSMvxUzJCEs86mLswgXgL3GacSA4nRH8g6BQj7tb/SoeU89h5Lc8cyEq8H0dTICpXjLXGWqVAYa3l5WUXf74+raASYnY7UDPSdtT+H1wOWBcLxVJpSFUBZ/6hrHs5qfGBQ/q80KeWd6bNDv0ft6zut+AaIwusl+kGqAmub4D3Y7O1KVu4pG34/5MiekLEtUmNUUErs/cxveSWo0HZmA5uF3v+ENp+lkShtdPpraXD3KEZ9gGO35a6bZvq0ucKqiJyErGlsMNxQbpJDssdlV88rFkjEpfb9PI12msg==

Delivery-date: Mon, 27 Feb 2023 09:52:16 +0000

List-id: Developer list for the Windows PV Drivers subproject <win-pv-devel.lists.xenproject.org>

Msip_labels:

Thread-index: AQHZK20C375p4jUD802sUWH5S2b6dq6lg3MAgACEQ4CAOr2ekA==

Thread-topic: Lazily construct slab commit causes BSOD/freeze on xen 4.16

I was able to confirm that the freeze occurs in OpenXT, however interestingly given 48 hours of waiting, the guest eventually continues to work. I have some logs from this event. The Short Log is more concise: it shows directly before the messages of

Feb 1 04:05:02.952634 VM hypervisor: (d12) xenbus|RangeSetPop: fail2

Feb 1 04:05:02.953460 VM hypervisor: (d12) xenbus|RangeSetPop: fail1 (c000009a)

Feb 1 04:05:02.953835 VM hypervisor: (d12) GNTTAB: MAP XENMAPSPACE_grant_table[4] @ 00000001.22805000

Feb 1 04:05:02.954312 VM hypervisor: (d12) xenbus|GnttabExpand: added references [00000800 - 000009ff]

Feb 1 04:05:02.957648 VM hypervisor: (d12) xenbus|RangeSetPop: fail2

Feb 1 04:05:02.958151 VM hypervisor: (d12) xenbus|RangeSetPop: fail1 (c000009a)

Feb 1 04:05:02.960012 VM hypervisor: (d12) GNTTAB: MAP XENMAPSPACE_grant_table[5] @ 00000001.22806000

Feb 1 04:05:02.960651 VM hypervisor: (d12) xenbus|GnttabExpand: added references [00000a00 - 00000bff]

Feb 1 04:05:02.971874 VM hypervisor: (d12) xenbus|RangeSetPop: fail2

Feb 1 04:05:02.978007 VM hypervisor: (d12) xenbus|RangeSetPop: fail1 (c000009a)

Feb 1 04:05:02.979312 VM hypervisor: (d12) GNTTAB: MAP XENMAPSPACE_grant_table[6] @ 00000001.22807000

Feb 1 04:05:02.980254 VM hypervisor: (d12) xenbus|GnttabExpand: added references [00000c00 - 00000dff]

start in perpetuity. I haven't gotten a BSOD yet, but if I reproduce it I will send that as well. The BROKEN_MESSAGES attachment shows that the guest froze as the beginning and I waited a long time for it to BSOD, and restarted it eventually. Any insight as to what might be happening and why we are seeing this freeze. I got these results with the current xen tools obtained from https://xenproject.org/downloads/windows-pv-drivers/development-builds/windows-pv-master/ to ensure it was not any patches that we might add. Xen version 4.16.4 as the host.

-Joel

From: win-pv-devel <win-pv-devel-bounces@xxxxxxxxxxxxxxxxxxxx> on behalf of Paul Durrant <xadimgnik@xxxxxxxxx>
Sent: Thursday, January 19, 2023 12:55 PM
To: win-pv-devel@xxxxxxxxxxxxxxxxxxxx <win-pv-devel@xxxxxxxxxxxxxxxxxxxx>
Subject: Re: Lazily construct slab commit causes BSOD/freeze on xen 4.16

Notice: This message originated outside of ainfosec.com

On 19/01/2023 10:02, Owen Smith wrote:
> I've not seen BSODs in this area with XenServer drivers which are based
> on the same commit, though we are carrying a patch to this area.
> Do you have any details about the crash (crashdumps, bugcheck IDs, etc)
> that could help pinpoint the problem?
>
> Attached is the patch XenServer's tools are currently carrying.
>

Owen, why did/do you need this patch? Did you try running with
CacheAudit() turned on (as it is by default in a debug build)?

Paul

Attachment: BROKEN_MESSAGES
Description: BROKEN_MESSAGES

Attachment: short_log
Description: short_log