Re: [win-pv-devel] Porting libvchan to use the Windows PV Drivers

[RafaÅ WojdyÅa <omeg@xxxxxxxxxxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2015-04-08 02:44, RafaÅ WojdyÅa wrote:
> On 2015-03-31 14:59, Paul Durrant wrote:
>>> -----Original Message----- From: RafaÅ WojdyÅa 
>>> [mailto:omeg@xxxxxxxxxxxxxxxxxxxxxx] Sent: 31 March 2015 13:22
>>>  To: Paul Durrant; win-pv-devel@xxxxxxxxxxxxxxxxxxxx Subject:
>>> Re: [win-pv-devel] Porting libvchan to use the Windows PV
>>> Drivers
>>> 
>> On 2015-03-31 11:54, Paul Durrant wrote:
>>>>>> -----Original Message----- From: RafaÅ WojdyÅa 
>>>>>> [mailto:omeg@xxxxxxxxxxxxxxxxxxxxxx] Sent: 31 March 2015 
>>>>>> 06:51 To: Paul Durrant;
>>>>>> win-pv-devel@xxxxxxxxxxxxxxxxxxxx Subject: Re:
>>>>>> [win-pv-devel] Porting libvchan to use the Windows PV
>>>>>> Drivers
>>>>>> 
>>>>> After some thought I've decided to implement the missing 
>>>>> pieces in the GPLPV drivers first, mostly to not do many 
>>>>> things at once (like porting our Windows tools to use the
>>>>> new drivers, changing our build process etc). I'll
>>>>> definitely test the new drivers once Qubes r3 becomes more
>>>>> stable.
>>>>> 
>>>>> 
>>>>>> Ok, fair enough.
>>>>> 
>>>>> I have some questions though, mostly on the topic of Xen's 
>>>>> grant mapping and memory hypercalls. As I wrote before, I'm
>>>>> a newcomer to Xen API on this level and to me the existing 
>>>>> documentation seems pretty impenetrable. The overall
>>>>> picture is mostly clear, but many important details seem
>>>>> missing.
>>>>> 
>>>>> Let's take the GNTTABOP_map_grant_ref call. In the header
>>>>> we see:
>>>>> 
>>>>> If GNTMAP_host_map is specified then a mapping will be
>>>>> added at * either a host virtual address in the current
>>>>> address space, or at * a PTE at the specified machine
>>>>> address.  The type of mapping to * perform is selected
>>>>> through the GNTMAP_contains_pte flag, and the * address is
>>>>> specified in <host_addr>.
>>>>> 
>>>>> I'm passing GNTMAP_host_map flag, but what type of address 
>>>>> should be passed for the mapped page? The section above
>>>>> says "mapping will be added at either a host virtual
>>>>> address..." but I doubt a VA is wanted, because I'd need to
>>>>> allocate some kernel memory to pass a VA to this call, and
>>>>> allocation means mapping in itself. Should this be a
>>>>> physical address? A PFN? Something else entirely?
>>>>> 
>>>>> 
>>>>>> Hypercalls on HVM guests usually deal in GFNs. There's 
>>>>>> basically no way for an HVM guest to get hold of MFN 
>>>>>> values. PV guests OTOH deal in MFNs.
>>>>> 
>>>>> After reading some Linux code that uses this call I saw
>>>>> that it "allocated" the address for the mapped page via
>>>>> the balloon driver. On Windows this means getting a page
>>>>> from Xen via XENMEM_increase_reservation as far as I know.
>>>>> 
>>>>>> Unless your toolstack is allocating extra memory that's 
>>>>>> not already in the guest P2M then that's going to fail.
>>>>>> I think what you probably want to do is either find a
>>>>>> 'hole' (i.e. GFNs not populated with RAM) such as the Xen
>>>>>> Platform PCI device's BAR space, or create a hole using 
>>>>>> decrease_reservation. Either way, you'll have a GFN in
>>>>>> your hand.
> 
>> I see that currently the function that maps the grant table
>> itself creates such "holes" with XENMEM_decrease_reservation.
> 
>>> Yeah, I used to do that too but these days I just put a simple 
>>> page allocator on top of the PCI BAR space. Good that you have 
>>> some code to copy'n'paste though :-)
> 
>>>>> 
>>>>> But then again: XENMEM_increase_reservation returns a MFN
>>>>> of the allocated page. What good is this for the above
>>>>> purpose? If I understand correctly, a HVM doesn't know real
>>>>> MFNs. Should I translate such MFN into a GMFN/GPFN and then
>>>>> pass it to GNTTABOP_map_grant_ref as the address? If so,
>>>>> how to perform such translation?
>>>>> 
>>>>> I experimented with some XENMEM hypercalls. 
>>>>> XENMEM_machphys_mapping "returns the location in virtual 
>>>>> address space of the machine_to_phys mapping table", but
>>>>> the structure and meaning of such table is not explained.
>>>>> I assume it translates real MFNs into GMFNs in some way.
>>>>> The call succeedes with following output values:
>>>>> 
>>>>> v_start  0xffff8000`00000000 v_end    0xffff8040`00000000 
>>>>> max_mfn 0x00000007`ffffffff
>>>>> 
>>>>> ...but that VA range appears to be not accessible/not 
>>>>> mapped.
>>>>> 
>>>>> XENMEM_machphys_mfn_list call also worked and returned a
>>>>> list of 9 values on my test HVM (Win7 x64 with 1GB of RAM).
>>>>> I'm not really sure what to make of them, the description
>>>>> isn't very helpful for someone who doesn't already know how
>>>>> this works: "Returns a list of MFN bases of 2MB extents
>>>>> comprising the machine_to_phys mapping table".
>>>>> 
>>>>> Maybe I'm overcomplicating things and overlooked something
>>>>>  obvious. I'd welcome any corrections and/or hints :)
>>>>> 
>>>>>> Yeah, I think you're overcomplicating things :-) See 
>>>>>> above.
>>>>> 
>>>>>> Paul
> 
>> Thank you, I'll tinker some more and report what comes out of
>> it.
> 
> 
>>> Thanks,
> 
>>> Paul
> 
> Alright, after a lot of tinkering I finally got it to work. It
> turns out that the GNTTABOP_map_grant_ref hypercall actually takes
> a physical guest address, not a PFN.
> 
> I was confused at first because the call succeeded when I passed a
> PFN in. I of course didn't see the mapped content at my address,
> because the PFN was treated as a physical address. This caused
> seemingly random memory corruption BSODs that were tricky to
> diagnose.
> 
> Finally I put some unique string into the shared page and searched
> for it in the mapping guest's memory. WinDbg's !search found the
> pattern at a PFN that was my hypercall's argument shifted right 12
> bits. Sure enough, that's a physical address-to-PFN conversion.
> 
> Now that it works I need to add proper ioctl interface for
> (un)mapping pages. Since I'm allocating kernel memory (or PCI
> address range) I'll need to ensure it's freed in all cases (the
> dreaded DuplicateHandle problem). The existing code for granting
> pages deals with it by pending the ioctl forever and using another
> ioctl to actually return data to the caller. This is pretty awkward
> but I don't know a better way to ensure everything is OK in case
> the original user mode caller dies while there's a duplicated
> handle open somewhere (apparently killing a process that contains
> locked pages causes a BSOD). Any thought s?

For now I went with PsSetCreateProcessNotifyRoutine for cleanup
purposes and it seems to work well. The callback runs at PASSIVE_LEVEL
and in the context of the process being destroyed so that allows for
easy cleanup of locked memory allocations and the like.

The grant mapping itself finally works well after ironing out some
more issues. Now I need to implement the libxc functions used by our
libxenvchan (like xc_gnttab_map_grant_ref_notify). After that's
working it should be relatively easy to port the code to the new
pvdrivers as well :)
- -- 
RafaÅ WojdyÅa
Qubes Tools for Windows developer
-----BEGIN PGP SIGNATURE-----

iQEcBAEBAgAGBQJVJzy1AAoJEIWi9rB2GrW7O/cH/04Nw2OltcDsUAjOEf08Bz5C
J3W3fKq8MQQK8htfPSui6Tt276LTHpqIvy50wx/2Pc84VdfME4cnq3f8vNWxUA00
De/QHjRHD5Y0Zuu9tHHMhSDMInkA92nR9J4wx/AfY/eJcu+9WREQlIzB90YIqN27
5ZSaOxUAmF919u0whWT/lfTXwWnq6EIl+Mz4YEXOhdYDYAkCQueLN6z2qh9I4Xjh
jZFrdZqP5r1GPr94GgI54IXlYi/hb8kkwwKUtTGbGa1HhZ5+QXruS2hqvtOrUB9a
qDBBeXyLqtP92HvD/1khOuKYP4rOD9I0ovnVP7RtsBStDHU9ELcRzYT7ErJzUfw=
=/XE+
-----END PGP SIGNATURE-----

_______________________________________________
win-pv-devel mailing list
win-pv-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/win-pv-devel