Re: [win-pv-devel] Porting libvchan to use the Windows PV Drivers

[RafaÅ WojdyÅa <omeg@xxxxxxxxxxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2015-03-13 10:17, Paul Durrant wrote:
>> -----Original Message----- From:
>> win-pv-devel-bounces@xxxxxxxxxxxxxxxxxxxx [mailto:win-pv-devel- 
>> bounces@xxxxxxxxxxxxxxxxxxxx] On Behalf Of Rafal Wojdyla Sent: 12
>> March 2015 20:05 To: Paul Durrant;
>> win-pv-devel@xxxxxxxxxxxxxxxxxxxx Cc: Marek Marczykowski-GÃrecki 
>> Subject: Re: [win-pv-devel] Porting libvchan to use the Windows
>> PV Drivers
>> 
> On 12.03.2015 18:09, Paul Durrant wrote:
>>>>> -----Original Message----- From: 
>>>>> win-pv-devel-bounces@xxxxxxxxxxxxxxxxxxxx
>>>>> [mailto:win-pv-devel- bounces@xxxxxxxxxxxxxxxxxxxx] On
>>>>> Behalf Of Rafal Wojdyla Sent: 12 March 2015 17:06 To: Paul
>>>>> Durrant; win-pv-devel@xxxxxxxxxxxxxxxxxxxx Cc: Marek
>>>>> Marczykowski-GÃrecki Subject: Re: [win-pv-devel] Porting
>>>>> libvchan to use the Windows PV Drivers
>>>>> 
>>>> On 2015-03-12 17:45, Paul Durrant wrote:
>>>>>>>> -----Original Message----- From: RafaÅ WojdyÅa 
>>>>>>>> [mailto:omeg@xxxxxxxxxxxxxxxxxxxxxx] Sent: 12 March
>>>>>>>> 2015 16:17 To: Paul Durrant;
>>>>>>>> win-pv-devel@xxxxxxxxxxxxxxxxxxxx Subject: Re:
>>>>>>>> [win-pv-devel] Porting libvchan to use the Windows PV
>>>>>>>> Drivers
>>>>>>>> 
>>>>>>> On 2015-03-11 18:46, Paul Durrant wrote:
>>>>>>>>>>> -----Original Message----- From: 
>>>>>>>>>>> win-pv-devel-bounces@xxxxxxxxxxxxxxxxxxxx 
>>>>>>>>>>> [mailto:win-pv-devel-
>>>>>>>>>>> bounces@xxxxxxxxxxxxxxxxxxxx] On Behalf Of
>>>>>>>>>>> Rafal Wojdyla Sent: 10 March 2015 20:16 To:
>>>>>>>>>>> win-pv-devel@xxxxxxxxxxxxxxxxxxxx Subject: Re: 
>>>>>>>>>>> [win-pv-devel] Porting libvchan to use the
>>>>>>>>>>> Windows PV Drivers
>>>>>>>>>>> 
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> I'm unable to properly reply to the thread since
>>>>>>>>>> I just subscribed to this list but I figured it's
>>>>>>>>>> worth chiming in (last message is here: 
>>>>>>>>>> http://lists.xenproject.org/archives/html/win-pv-devel/2015-
>
>>>>>>>>>> 
01/msg0
>>>> 
>>>>>>>>>> 
> 006
>>>>>>> 
>>>>>>>>>> 
>>>> 0.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> html)
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> Yes, I understand; I just saw your
>>>>>>>>>>> subscription message :-)
>>>>>>>>>> 
>>>>>>>>>> First, some background about me. I'm currently
>>>>>>>>>> the main and pretty much the only
>>>>>>>>>> developer/maintainer of guest tools for Windows
>>>>>>>>>> for Qubes OS (https://wiki.qubes-os.org/). Some
>>>>>>>>>> of you may have heard of Qubes -- in short, it's
>>>>>>>>>> an attempt at creating a secure OS based on
>>>>>>>>>> lightweight AppVMs, currently using Linux/Xen as
>>>>>>>>>> base. It supports Windows HVMs and our guest
>>>>>>>>>> tools provide integration with dom0/other domUs
>>>>>>>>>> (services like data transfer, remote execution, 
>>>>>>>>>> seamless GUI experience etc).
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> Cool.
>>>>>>>>>> 
>>>>>>>>>> We're in the process of finalizing the next
>>>>>>>>>> major release (r3) of Qubes, it will use Xen 4.4
>>>>>>>>>> instead of r2's Xen 4.1. As for our Windows
>>>>>>>>>> tools, they are (currently) using PV drivers
>>>>>>>>>> based on James Harper's cod e.
>>>>>>>>>> 
>>>>>>>>>> Our inter-VM communication protocol uses vchan
>>>>>>>>>> (in fact, vchan originates from our patch
>>>>>>>>>> accepted into Xen's source a few years ago). In
>>>>>>>>>> Qubes r2 we have a Windows libvchan
>>>>>>>>>> implementation, but as stated above, it uses old
>>>>>>>>>> PV drivers interfaces. You can find it here:
>>>>>>>>>> https://github.com/QubesOS/qubes-core-vchan-xen
>>>>>>>>>> 
>>>>>>>>>> That implementation has one big flaw: client
>>>>>>>>>> side vchan functions are not implemented. It
>>>>>>>>>> didn't matter for Qubes r2, where all vchan
>>>>>>>>>> communication is passing through dom0 anyway. In
>>>>>>>>>> Qubes r3 however, we need that working because of
>>>>>>>>>> redesigned inter-VM communication protocol that
>>>>>>>>>> allows direct VM-VM communication after dom0
>>>>>>>>>> arbitration.
>>>>>>>>>> 
>>>>>>>>>> Unfortunately Harper's drivers don't seem to
>>>>>>>>>> implement the needed kernel interfaces for that
>>>>>>>>>> as well.
>>>>>>>>>> 
>>>>>>>>>>> I assume you mean grant mapping? Or maybe just
>>>>>>>>>>> grant copy, since that would be safer?
>>>>>>>>>> 
>>>>>>>>>> I didn't need to look into PV drivers sources
>>>>>>>>>> before, but it seems I will need to do that now
>>>>>>>>>> :) I found the new PV drivers and this mailing
>>>>>>>>>> list, found the thread about vchan
>>>>>>>>>> implementation... and that's pretty much it for
>>>>>>>>>> now.
>>>>>>>>>> 
>>>>>>>>>> As I said, I don't have much experience in Xen
>>>>>>>>>> APIs (didn't need to tinker with them directly
>>>>>>>>>> before). I do, however, have extensive WinAPI
>>>>>>>>>> knowledge and moderate amount of Windows driver
>>>>>>>>>> development experience (part of our guest tools
>>>>>>>>>> is a custom display driver that allows no-copy
>>>>>>>>>> video memory sharing with dom0). I managed to
>>>>>>>>>> build the new drivers and will test them on our
>>>>>>>>>> dev Qubes build soon.
>>>>>>>>>> 
>>>>>>>>>> So, to summarize, I'm very interested in
>>>>>>>>>> developing a Windows vchan implementation on top
>>>>>>>>>> of the new PV drivers. I'll be reading through
>>>>>>>>>> the driver sources for a bit still to familiarize
>>>>>>>>>> myself with the environment. If anyone managed to
>>>>>>>>>> get something working, or just has ideas, let me
>>>>>>>>>> know.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> If you want to look at adding the necessary
>>>>>>>>>>> code to the XENBUS_GNTTAB interface to do grant
>>>>>>>>>>> map/copy then I don't imagine it will be too
>>>>>>>>>>> hard. Adding support for copy would be easiest
>>>>>>>>>>> but it would also be possible to grant map
>>>>>>>>>>> pages into the platform PCI device's BAR (which
>>>>>>>>>>> is where the shared info page and the grant
>>>>>>>>>>> table itself live).
>>>>>>>>>> 
>>>>>>>>>>> Let me know if have any specific questions or
>>>>>>>>>>> need some help getting the drivers going in
>>>>>>>>>>> your environment.
>>>>>>>>>> 
>>>>>>> I've tested the drivers on a Win7 pro x64 HVM on Qubes
>>>>>>> r2 (r3 is still a bit unstable). Xenbus and xeniface
>>>>>>> both install fine. Xenvbd installs OK but the OS BSODs
>>>>>>> on reboot with code 7B (inaccessible boot device). I'll
>>>>>>> try to pinpoint the exact failure spot once I setup the
>>>>>>> pvdrivers sources inside my development VM.
>>>>>>> 
>>>>>>> 
>>>>>>>> 0x7B can occur in many circumstances. The drivers do
>>>>>>>> log quite a bit of info, particularly in checked
>>>>>>>> builds, so there'll probably be something there to
>>>>>>>> indicate the exact nature of the failure. The main
>>>>>>>> informational logging (which is the same for free or
>>>>>>>> checked builds) is written to the qemu logging port
>>>>>>>> (0x12) and debug logging (checked build only) goes to
>>>>>>>> the Xen port (0xE9). If you watch wherever you have
>>>>>>>> those redirected then you may be able to spot the
>>>>>>>> problem. If you can't then post them to the list and
>>>>>>>> I'll take a look.
>>>>>>> 
>>>>>>> Do the drivers have specific requirements for backend 
>>>>>>> (Xen/Qemu version)? We're not really using Qemu in
>>>>>>> dom0, only in minimal stubdoms for HVMs, so that may be
>>>>>>> a problem.
>>>>>>> 
>>>>>>> 
>>>>>>>> That's not usually a problem. Do you have PV backends
>>>>>>>> for disk and net set up though? The fact that you got
>>>>>>>> a 0x7B after installing xenvbd may simply mean that
>>>>>>>> your toolstack has just not set up a PV backend.
>>>> We do have backends set up (xen-blkback for vbd). I'll check
>>>> in dom0 whether it crashes after the device gets attached or
>>>> before.
>>>> 
>>>> 
>>>>> Ok. I must admit that I tend to use qdisk as a backend in
>>>>> most of my testing, but blkback should be fine. I'll sanity
>>>>> check it myself when I get time though.
>>>> 
> Sometimes the VM BSODs with 0x7E. I managed to connect WinDbg to
> it and grab some logs (in the attachment). At a glance it seems
> like a lot of event channel failures...
> 
> XENVBD|PdoReset:ASSERTION FAILED: (((NTSTATUS)(Status)) >= 0) 
> Assertion 
> f:\qubes-builder\qubes-src\xen-pv\xenvbd\src\xenvbd\pdo.c(2297): 
> (((NTSTATUS)(Status)) >= 0)
> 
> 
>> Yes, that certainly sounds like it could be the cause. What
>> version of Xen are you running on? The latest XENBUS supports
>> FIFO event channels and per-cpu upcalls (which is patch that went
>> into Xen post-4.5) but should fall back to 2-level and standard
>> callback via if those features are not in the hypervisor. There
>> was a bug in 2-level event handling in XENBUS which I fixed
>> with:
> 
>> commit f321e204a081f9c4dcc732e71283a401751a241b Author: Paul
>> Durrant <paul.durrant@xxxxxxxxxx> Date:   Fri Feb 27 13:48:46
>> 2015 +0000
> 
>> Fix event channel unmasking for two-level ABI
> 
>> The two-level ABI requires that an event is masked for the
>> unmask hypercall to raise the event, so the test-and-clear
>> operation in the guest basically means that pending events get
>> stuck. The simple fix is to re-mask pending events before making
>> the hypercall. This is unnecessary when the FIFO ABI is used, but
>> it's safe. Hence this patch unconditionally re-masks pending
>> events, regardless of ABI, before making the unmask hypercall.
> 
>> Signed-off-by: Paul Durrant <paul.durrant@xxxxxxxxxx>
> 
>> If you are on a Xen that's old enough not to have the FIFO ABI
>> you'll definitely need that fix. Paul
We're using Xen 4.1 in Qubes r2, 4.4 in the upcoming r3. I'll check on
r3 once it runs properly.

As for the xenbus fix, I do have it -- I've cloned the sources and
built all the drivers myself a few days ago.
> 
>>>> Also adding Marek to the conversation, he's one of our core 
>>>> architects and knows more about backend stuff than I do :)
>>>> 
>>>> 
>>>>> Cool.
>>>> 
>>>>> Paul
>>>> 
>>>>>>> 
>>>>>>>> Paul
> 

- -- 
RafaÅ WojdyÅa
Qubes Tools for Windows developer
-----BEGIN PGP SIGNATURE-----

iQEcBAEBAgAGBQJVAuXPAAoJEIWi9rB2GrW7LdsIAJLil/ZeLFbeztDxjoO+8hj5
loxqfxxnRo6ZklS0m/rlRiUb8aetsJdhtwrJHcuyeIkV8ZypYcIrQ4vsdiNCBs8i
apXQrNmCrvAh+flHcuXbZrHE72l81u4gpiAoJHYP1F/WP/NUKqubXoWA+6z5jaeE
8CJErkHfPnflr7v4V99FmjnKoY16HSfwwMKyk4lOel5dKkl4Oi2YIGLV+ZnjWio8
kO4xx1xGF3RQM9HHKEc5uGaQ18yYfiiy3sTEv5V2SXUrVRQ3AqY5BijhRb8iInOb
Qf/jgh6T6tNwTKdNiAMg254etN5FpIE90r3FcGgFCma3LOfTttJ+u7U/Ca92UK0=
=F5Jk
-----END PGP SIGNATURE-----

_______________________________________________
win-pv-devel mailing list
win-pv-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/win-pv-devel