Re: [win-pv-devel] Porting libvchan to use the Windows PV Drivers

[RafaÅ WojdyÅa <omeg@xxxxxxxxxxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2015-03-31 14:59, Paul Durrant wrote:
>> -----Original Message----- From: RafaÅ WojdyÅa
>> [mailto:omeg@xxxxxxxxxxxxxxxxxxxxxx] Sent: 31 March 2015 13:22 
>> To: Paul Durrant; win-pv-devel@xxxxxxxxxxxxxxxxxxxx Subject: Re:
>> [win-pv-devel] Porting libvchan to use the Windows PV Drivers
>> 
> On 2015-03-31 11:54, Paul Durrant wrote:
>>>>> -----Original Message----- From: RafaÅ WojdyÅa 
>>>>> [mailto:omeg@xxxxxxxxxxxxxxxxxxxxxx] Sent: 31 March 2015
>>>>> 06:51 To: Paul Durrant; win-pv-devel@xxxxxxxxxxxxxxxxxxxx
>>>>> Subject: Re: [win-pv-devel] Porting libvchan to use the
>>>>> Windows PV Drivers
>>>>> 
>>>> After some thought I've decided to implement the missing
>>>> pieces in the GPLPV drivers first, mostly to not do many
>>>> things at once (like porting our Windows tools to use the new
>>>> drivers, changing our build process etc). I'll definitely
>>>> test the new drivers once Qubes r3 becomes more stable.
>>>> 
>>>> 
>>>>> Ok, fair enough.
>>>> 
>>>> I have some questions though, mostly on the topic of Xen's
>>>> grant mapping and memory hypercalls. As I wrote before, I'm a
>>>> newcomer to Xen API on this level and to me the existing
>>>> documentation seems pretty impenetrable. The overall picture
>>>> is mostly clear, but many important details seem missing.
>>>> 
>>>> Let's take the GNTTABOP_map_grant_ref call. In the header we
>>>> see:
>>>> 
>>>> If GNTMAP_host_map is specified then a mapping will be added
>>>> at * either a host virtual address in the current address
>>>> space, or at * a PTE at the specified machine address.  The
>>>> type of mapping to * perform is selected through the
>>>> GNTMAP_contains_pte flag, and the * address is specified in
>>>> <host_addr>.
>>>> 
>>>> I'm passing GNTMAP_host_map flag, but what type of address
>>>> should be passed for the mapped page? The section above says
>>>> "mapping will be added at either a host virtual address..."
>>>> but I doubt a VA is wanted, because I'd need to allocate some
>>>> kernel memory to pass a VA to this call, and allocation means
>>>> mapping in itself. Should this be a physical address? A PFN?
>>>> Something else entirely?
>>>> 
>>>> 
>>>>> Hypercalls on HVM guests usually deal in GFNs. There's
>>>>> basically no way for an HVM guest to get hold of MFN
>>>>> values. PV guests OTOH deal in MFNs.
>>>> 
>>>> After reading some Linux code that uses this call I saw that
>>>> it "allocated" the address for the mapped page via the
>>>> balloon driver. On Windows this means getting a page from Xen
>>>> via XENMEM_increase_reservation as far as I know.
>>>> 
>>>>> Unless your toolstack is allocating extra memory that's
>>>>> not already in the guest P2M then that's going to fail. I
>>>>> think what you probably want to do is either find a 'hole'
>>>>> (i.e. GFNs not populated with RAM) such as the Xen Platform
>>>>> PCI device's BAR space, or create a hole using
>>>>> decrease_reservation. Either way, you'll have a GFN in your
>>>>> hand.
> 
> I see that currently the function that maps the grant table itself 
> creates such "holes" with XENMEM_decrease_reservation.
> 
>> Yeah, I used to do that too but these days I just put a simple
>> page allocator on top of the PCI BAR space. Good that you have
>> some code to copy'n'paste though :-)
> 
>>>> 
>>>> But then again: XENMEM_increase_reservation returns a MFN of
>>>> the allocated page. What good is this for the above purpose?
>>>> If I understand correctly, a HVM doesn't know real MFNs.
>>>> Should I translate such MFN into a GMFN/GPFN and then pass it
>>>> to GNTTABOP_map_grant_ref as the address? If so, how to
>>>> perform such translation?
>>>> 
>>>> I experimented with some XENMEM hypercalls. 
>>>> XENMEM_machphys_mapping "returns the location in virtual
>>>> address space of the machine_to_phys mapping table", but the
>>>> structure and meaning of such table is not explained. I
>>>> assume it translates real MFNs into GMFNs in some way. The
>>>> call succeedes with following output values:
>>>> 
>>>> v_start  0xffff8000`00000000 v_end    0xffff8040`00000000
>>>> max_mfn 0x00000007`ffffffff
>>>> 
>>>> ...but that VA range appears to be not accessible/not
>>>> mapped.
>>>> 
>>>> XENMEM_machphys_mfn_list call also worked and returned a list
>>>> of 9 values on my test HVM (Win7 x64 with 1GB of RAM). I'm
>>>> not really sure what to make of them, the description isn't
>>>> very helpful for someone who doesn't already know how this
>>>> works: "Returns a list of MFN bases of 2MB extents comprising
>>>> the machine_to_phys mapping table".
>>>> 
>>>> Maybe I'm overcomplicating things and overlooked something 
>>>> obvious. I'd welcome any corrections and/or hints :)
>>>> 
>>>>> Yeah, I think you're overcomplicating things :-) See
>>>>> above.
>>>> 
>>>>> Paul
> 
> Thank you, I'll tinker some more and report what comes out of it.
> 
> 
>> Thanks,
> 
>> Paul

Alright, after a lot of tinkering I finally got it to work. It turns
out that the GNTTABOP_map_grant_ref hypercall actually takes a
physical guest address, not a PFN.

I was confused at first because the call succeeded when I passed a PFN
in. I of course didn't see the mapped content at my address, because
the PFN was treated as a physical address. This caused seemingly
random memory corruption BSODs that were tricky to diagnose.

Finally I put some unique string into the shared page and searched for
it in the mapping guest's memory. WinDbg's !search found the pattern
at a PFN that was my hypercall's argument shifted right 12 bits. Sure
enough, that's a physical address-to-PFN conversion.

Now that it works I need to add proper ioctl interface for (un)mapping
pages. Since I'm allocating kernel memory (or PCI address range) I'll
need to ensure it's freed in all cases (the dreaded DuplicateHandle
problem). The existing code for granting pages deals with it by
pending the ioctl forever and using another ioctl to actually return
data to the caller. This is pretty awkward but I don't know a better
way to ensure everything is OK in case the original user mode caller
dies while there's a duplicated handle open somewhere (apparently
killing a process that contains locked pages causes a BSOD). Any thought
s?

- -- 
RafaÅ WojdyÅa
Qubes Tools for Windows developer
-----BEGIN PGP SIGNATURE-----

iQEcBAEBAgAGBQJVJHnTAAoJEIWi9rB2GrW7tVgH/2PNyK0By4LZ2QSUSrDJ++v+
aHTMgf4oIGrr3Rr7roiFuxESawcPgTaHIrUedUJShPyhL1jbYgPkE3b7Eb0lB84d
IJEUk1ZRk+VBIT2b1pubfCMjXyPsydRF4Bz0jQ1VVxDIKVcQdOinOBAeh61ZUP0M
5yjbunBgEkSQwjKaMFqT8XRMP9UM9OV4VX8jF/C6rBXinrGuy6v4HQXKWsqOsuHF
Doo69hpkE6BCW1EfXTvZNwqS9r/qPse1NQPxE31ZvDes4RG2/21cF6UrJjqWkSBA
POdlOEsnc9KsWby46u1J7qyj0x3eBlI8moMp0du0ajlhHTnrkSIdzKBQQIMidWc=
=/5ll
-----END PGP SIGNATURE-----

_______________________________________________
win-pv-devel mailing list
win-pv-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/win-pv-devel