Re: [Xen-users] Error accessing memory mapped by xenforeignmemory_map()

[Brett Stahlman <brettstahlman@xxxxxxxxx>]

Julien,

On Mon, Oct 30, 2017 at 1:37 PM, Julien Grall <julien.grall@xxxxxxx> wrote:
> On 30/10/17 16:26, Brett Stahlman wrote:
>>
>> Hello Julien,
>
>
> Hello Brett,
>
>> On Sun, Oct 29, 2017 at 3:37 PM, Julien Grall <julien.grall@xxxxxxx>
>> wrote:
>>>
>>> Hello Brett,
>>>
>>> On 27/10/2017 22:58, Brett Stahlman wrote:
>>>>
>>>>
>>>> On Fri, Oct 27, 2017 at 3:22 PM, Stefano Stabellini
>>>> <sstabellini@xxxxxxxxxx> wrote:
>>>>>
>>>>>
>>>>> CC'ing the tools Maintainers and Paul
>>>>>
>>>>> On Fri, 27 Oct 2017, Brett Stahlman wrote:
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 27, 2017 at 9:31 AM, Roger Pau Monné
>>>>>> <roger.pau@xxxxxxxxxx>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Adding the ARM maintainers.
>>>>>>>
>>>>>>> On Wed, Oct 25, 2017 at 11:54:59AM -0500, Brett Stahlman wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm trying to use the "xenforeignmemory" library to read arbitrary
>>>>>>>> memory ranges from a Xen domain. The code performing the reads is
>>>>>>>> designed to run in dom0 on a Zynq ultrascale MPSoC (ARM64), though
>>>>>>>> I'm
>>>>>>>> currently testing in QEMU. I constructed a simple test program,
>>>>>>>> which
>>>>>>>> reads an arbitrary domid/address pair from the command line,
>>>>>>>> converts
>>>>>>>> the address (assumed to be physical) to a page frame number, and
>>>>>>>> uses
>>>>>>>> xenforeignmemory_map() to map the page into the test app's virtual
>>>>>>>> memory space. Although xenforeignmemory_map() returns a non-NULL
>>>>>>>> pointer, my attempt to dereference it fails with the following
>>>>>>>> error:
>>>>>>>>
>>>>>>>> (XEN) traps.c:2508:d0v1 HSR=0x93810007 pc=0x400a20 gva=0x7f965f7000
>>>>>>>> gpa=0x00000030555000
>>>>>>>>
>>>>>>>> [   74.361735] Unhandled fault: ttbr address size fault (0x92000000)
>>>>>>>> at 0x0000007f965f7000
>>>>>>>> Bus error
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I'm not sure what a Bus error means on ARM, have you tried to look
>>>>>>> at traps.c:2508 to see if there's some comment explaining why this
>>>>>>> fault is triggered?
>>>>>>
>>>>>>
>>>>>>
>>>>>> I believe the fault is occurring because mmap() failed to map the
>>>>>> page.
>>>>>> Although xenforeignmemory_map() is indeed returning a non-NULL
>>>>>> pointer,
>>>>>> code comments indicate that this does not imply success: page-level
>>>>>> errors might still be returned in the provided "err" array. In my
>>>>>> case,
>>>>>> it appears that an EINVAL is produced by mmap(): specifically, I
>>>>>> believe
>>>>>> it's coming from privcmd_ioctl_mmap_batch() (drivers/xen/privcmd.c),
>>>>>> but
>>>>>> there are a number of conditions that can produce this error code, and
>>>>>> I
>>>>>> haven't yet determined which is to blame...
>>>>>>
>>>>>> So although I'm not sure why I would get an "address size" fault, it
>>>>>> makes sense that the pointer dereference would generate some sort of
>>>>>> paging-related fault, given that the page mapping was unsuccessful.
>>>>>> Hopefully, ARM developers will be able to explain why it was
>>>>>> unsuccessful, or at least give me an idea of what sorts of things
>>>>>> could
>>>>>> cause a mapping attempt to fail... At this point, I'm not particular
>>>>>> about what address I map. I just want to be able to read known data at
>>>>>> a
>>>>>> fixed (non-paged) address (e.g., kernel code/data), so I can prove to
>>>>>> myself that the page is actually mapped.
>>>>>
>>>>>
>>>>>
>>>>> The fault means "Data Abort from a lower Exception level". It could be
>>>>> an MMU fault or an alignment fault, according to the ARM ARM.
>>>>>
>>>>> I guess that the address range is not good. What DomU addresses are you
>>>>> trying to map?
>>>>
>>>>
>>>>
>>>> The intent was to map fixed "guest physical" addresses corresponding to
>>>> (e.g) the "zero page" of a guest's running kernel. Up until today, I'd
>>>
>>>
>>>
>>> What do you mean by "zero page"? Is it the guest physical address 0? If
>>> so,
>>> the current guest memory layout does not have anything mapped at the
>>> address.
>>
>>
>> No. I didn't mean guest physical address 0, but rather the start of the
>> linux kernel itself: specifically, the code in head.S. IIUC, the kernel
>> bootstrap code responsible for decompressing the kernel typically loads
>> this code at a fixed address, which on x86 architectures, happens to be
>> 0x100000. Thus, my assumption has been that if an unmodified Linux OS
>> were run in an x86 Xen guest, Xen would need to map guest physical
>> address 0x100000 to the machine physical address where the guest Linux
>> kernel is actually loaded. I'd also been assuming that if code running
>> in dom0 wished to use the foreignmemory interface to read the first page
>> of such a guest's kernel, it would need to provide the "guest physical"
>> address 0x100000 to xenforeignmemory_map(). I'm still thinking this may
>> be true for an *unmodified* guest (i.e., HVM), but having read more
>> about Xen's paravirtualized memory over the weekend, I'm thinking it
>> would not hold true for a paravirtualized (PV) guest, which doesn't have
>> the same concept of "guest physical" addresses.
>
>
> I am not x86 expert and will let x86 folks answered to that.
>
> To give the Arm64 view, the image headers allow to specify whether the
> kernel needs to be close to the beginning of the DRAM. But it is still not a
> fixed address.
>
> In practice, the toolstack will always load the Image at the ram base + text
> offset (specified in the kernel). But that's just for convenience and may
> change in the future.

Ok. So I'm thinking the relevant address would be 0xffffffc000080000,
which is where System.map places the kernel's "_text" segment. IIUC,
aarch64 Linux uses 39-bit addresses in a range controlled by TTBR1,
beginning at 0xffffff8000000000. I'm assuming the kernel's address
space uses a direct, 1-to-1 mapping (i.e., guest virtual == guest
physical). Correct? Also, should a signed or unsigned right shift be
used to convert a kernel address to a pfn? I.e., would the pfn
corresponding to 0xffffffc000080000 be 0xfffffffffc000080 or
0x000ffffffc000080?

Thanks,
Brett S.

>
>>
>>>
>>>> assumed that a PV guest's kernel would be loaded at a known "guest
>>>> physical" address (like 0x100000 on i386), and that such addresses
>>>> corresponded to the gfn's expected by xenforeignmemory_map(). But now I
>>>> suspect this was an incorrect assumption, at least for the PV case. I've
>>>> had trouble finding relevant documentation on the Xen site, but I did
>>>> find a presentation earlier today suggesting that for PV's, gfn == mfn,
>>>> which IIUC, would effectively preclude the use of fixed addresses in a
>>>> PV guest. IOW, unlike an HVM's kernel, a PV's kernel cannot be loaded at
>>>> a "known" address (e.g., 0x100000 on i386).
>>>>
>>>> Perhaps my use case (reading a guest kernel's code/data from dom0) makes
>>>> sense for an HVM, but not a PV? Is it not possible for dom0 to use the
>>>> foreignmemory interface to map PV guest pages read-only, without knowing
>>>> in advance what, if anything, those pages represent in the guest? Or is
>>>> the problem that the very concept of "guest physical" doesn't exist in a
>>>> PV? I guess it would help if I had a better understanding of what sort
>>>> of frame numbers are expected by xenforeignmemory_map() when the target
>>>> VM is a PV. Is the Xen code the only documentation for this sort of
>>>> thing, or is there some place I could get a high-level overview?
>>>
>>>
>>>
>>> I am a bit confused with the rest of this e-mail. There are no concept of
>>> HVM or PV on Arm. This is x86 specific. For Arm, there is a single type
>>> of
>>> guest that borrow the goods of both HVM and PV.
>>>
>>> For instance, like HVM, the hardware is used to provide a separate
>>> address
>>> space for each virtual machine. Arm calls that stage-2 translation. So
>>> gfn
>>> != mfn.
>>
>>
>> I was not aware that the HVM/PV concept didn't apply directly to ARM.
>> Is there a document that summarizes the way Xen's address translation
>> works on ARM? The document I've been looking at is...
>>
>> https://wiki.xen.org/wiki/X86_Paravirtualised_Memory_Management
>
>
> The closest I would find explaining Xen's address translation scheme would
> be my talk at Xen Developer Summit:
>
> https://www.slideshare.net/xen_com_mgr/keeping-coherency-on-arm-julien-grall-arm
>
> The address translation is very simple and only follow the scheme introduced
> by the Arm Arm.
>
>>
>> ...but I haven't found anything analogous for ARM. At any rate, if the
>> ARM hardware is providing a separate address space for each VM, then I
>> suppose the concept of "guest physical" addresses is still valid. Does a
>> guest physical address correspond to the output of stage-1 translation?
>
>
> Yes.
>
>> And are guest kernels on ARM generally loaded at fixed addresses (like
>> 0x100000 in the x86 case), or is the kernel load address determined
>> dynamically?
>
>
> See my answer above.
>
> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
https://lists.xen.org/xen-users