Re: [Xen-users] Error accessing memory mapped by xenforeignmemory_map()

[Brett Stahlman <brettstahlman@xxxxxxxxx>]

On Mon, Oct 30, 2017 at 6:03 PM, Stefano Stabellini
<sstabellini@xxxxxxxxxx> wrote:
> On Mon, 30 Oct 2017, Brett Stahlman wrote:
>> Julien,
>>
>> On Mon, Oct 30, 2017 at 1:37 PM, Julien Grall <julien.grall@xxxxxxx> wrote:
>> > On 30/10/17 16:26, Brett Stahlman wrote:
>> >>
>> >> Hello Julien,
>> >
>> >
>> > Hello Brett,
>> >
>> >> On Sun, Oct 29, 2017 at 3:37 PM, Julien Grall <julien.grall@xxxxxxx>
>> >> wrote:
>> >>>
>> >>> Hello Brett,
>> >>>
>> >>> On 27/10/2017 22:58, Brett Stahlman wrote:
>> >>>>
>> >>>>
>> >>>> On Fri, Oct 27, 2017 at 3:22 PM, Stefano Stabellini
>> >>>> <sstabellini@xxxxxxxxxx> wrote:
>> >>>>>
>> >>>>>
>> >>>>> CC'ing the tools Maintainers and Paul
>> >>>>>
>> >>>>> On Fri, 27 Oct 2017, Brett Stahlman wrote:
>> >>>>>>
>> >>>>>>
>> >>>>>> On Fri, Oct 27, 2017 at 9:31 AM, Roger Pau Monné
>> >>>>>> <roger.pau@xxxxxxxxxx>
>> >>>>>> wrote:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Adding the ARM maintainers.
>> >>>>>>>
>> >>>>>>> On Wed, Oct 25, 2017 at 11:54:59AM -0500, Brett Stahlman wrote:
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> I'm trying to use the "xenforeignmemory" library to read arbitrary
>> >>>>>>>> memory ranges from a Xen domain. The code performing the reads is
>> >>>>>>>> designed to run in dom0 on a Zynq ultrascale MPSoC (ARM64), though
>> >>>>>>>> I'm
>> >>>>>>>> currently testing in QEMU. I constructed a simple test program,
>> >>>>>>>> which
>> >>>>>>>> reads an arbitrary domid/address pair from the command line,
>> >>>>>>>> converts
>> >>>>>>>> the address (assumed to be physical) to a page frame number, and
>> >>>>>>>> uses
>> >>>>>>>> xenforeignmemory_map() to map the page into the test app's virtual
>> >>>>>>>> memory space. Although xenforeignmemory_map() returns a non-NULL
>> >>>>>>>> pointer, my attempt to dereference it fails with the following
>> >>>>>>>> error:
>> >>>>>>>>
>> >>>>>>>> (XEN) traps.c:2508:d0v1 HSR=0x93810007 pc=0x400a20 gva=0x7f965f7000
>> >>>>>>>> gpa=0x00000030555000
>> >>>>>>>>
>> >>>>>>>> [   74.361735] Unhandled fault: ttbr address size fault (0x92000000)
>> >>>>>>>> at 0x0000007f965f7000
>> >>>>>>>> Bus error
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> I'm not sure what a Bus error means on ARM, have you tried to look
>> >>>>>>> at traps.c:2508 to see if there's some comment explaining why this
>> >>>>>>> fault is triggered?
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> I believe the fault is occurring because mmap() failed to map the
>> >>>>>> page.
>> >>>>>> Although xenforeignmemory_map() is indeed returning a non-NULL
>> >>>>>> pointer,
>> >>>>>> code comments indicate that this does not imply success: page-level
>> >>>>>> errors might still be returned in the provided "err" array. In my
>> >>>>>> case,
>> >>>>>> it appears that an EINVAL is produced by mmap(): specifically, I
>> >>>>>> believe
>> >>>>>> it's coming from privcmd_ioctl_mmap_batch() (drivers/xen/privcmd.c),
>> >>>>>> but
>> >>>>>> there are a number of conditions that can produce this error code, and
>> >>>>>> I
>> >>>>>> haven't yet determined which is to blame...
>> >>>>>>
>> >>>>>> So although I'm not sure why I would get an "address size" fault, it
>> >>>>>> makes sense that the pointer dereference would generate some sort of
>> >>>>>> paging-related fault, given that the page mapping was unsuccessful.
>> >>>>>> Hopefully, ARM developers will be able to explain why it was
>> >>>>>> unsuccessful, or at least give me an idea of what sorts of things
>> >>>>>> could
>> >>>>>> cause a mapping attempt to fail... At this point, I'm not particular
>> >>>>>> about what address I map. I just want to be able to read known data at
>> >>>>>> a
>> >>>>>> fixed (non-paged) address (e.g., kernel code/data), so I can prove to
>> >>>>>> myself that the page is actually mapped.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> The fault means "Data Abort from a lower Exception level". It could be
>> >>>>> an MMU fault or an alignment fault, according to the ARM ARM.
>> >>>>>
>> >>>>> I guess that the address range is not good. What DomU addresses are you
>> >>>>> trying to map?
>> >>>>
>> >>>>
>> >>>>
>> >>>> The intent was to map fixed "guest physical" addresses corresponding to
>> >>>> (e.g) the "zero page" of a guest's running kernel. Up until today, I'd
>> >>>
>> >>>
>> >>>
>> >>> What do you mean by "zero page"? Is it the guest physical address 0? If
>> >>> so,
>> >>> the current guest memory layout does not have anything mapped at the
>> >>> address.
>> >>
>> >>
>> >> No. I didn't mean guest physical address 0, but rather the start of the
>> >> linux kernel itself: specifically, the code in head.S. IIUC, the kernel
>> >> bootstrap code responsible for decompressing the kernel typically loads
>> >> this code at a fixed address, which on x86 architectures, happens to be
>> >> 0x100000. Thus, my assumption has been that if an unmodified Linux OS
>> >> were run in an x86 Xen guest, Xen would need to map guest physical
>> >> address 0x100000 to the machine physical address where the guest Linux
>> >> kernel is actually loaded. I'd also been assuming that if code running
>> >> in dom0 wished to use the foreignmemory interface to read the first page
>> >> of such a guest's kernel, it would need to provide the "guest physical"
>> >> address 0x100000 to xenforeignmemory_map(). I'm still thinking this may
>> >> be true for an *unmodified* guest (i.e., HVM), but having read more
>> >> about Xen's paravirtualized memory over the weekend, I'm thinking it
>> >> would not hold true for a paravirtualized (PV) guest, which doesn't have
>> >> the same concept of "guest physical" addresses.
>> >
>> >
>> > I am not x86 expert and will let x86 folks answered to that.
>> >
>> > To give the Arm64 view, the image headers allow to specify whether the
>> > kernel needs to be close to the beginning of the DRAM. But it is still not 
>> > a
>> > fixed address.
>> >
>> > In practice, the toolstack will always load the Image at the ram base + 
>> > text
>> > offset (specified in the kernel). But that's just for convenience and may
>> > change in the future.
>>
>> Ok. Perhaps it would help if I examined this code...
>>
>> >
>> >>
>> >>>
>> >>>> assumed that a PV guest's kernel would be loaded at a known "guest
>> >>>> physical" address (like 0x100000 on i386), and that such addresses
>> >>>> corresponded to the gfn's expected by xenforeignmemory_map(). But now I
>> >>>> suspect this was an incorrect assumption, at least for the PV case. I've
>> >>>> had trouble finding relevant documentation on the Xen site, but I did
>> >>>> find a presentation earlier today suggesting that for PV's, gfn == mfn,
>> >>>> which IIUC, would effectively preclude the use of fixed addresses in a
>> >>>> PV guest. IOW, unlike an HVM's kernel, a PV's kernel cannot be loaded at
>> >>>> a "known" address (e.g., 0x100000 on i386).
>> >>>>
>> >>>> Perhaps my use case (reading a guest kernel's code/data from dom0) makes
>> >>>> sense for an HVM, but not a PV? Is it not possible for dom0 to use the
>> >>>> foreignmemory interface to map PV guest pages read-only, without knowing
>> >>>> in advance what, if anything, those pages represent in the guest? Or is
>> >>>> the problem that the very concept of "guest physical" doesn't exist in a
>> >>>> PV? I guess it would help if I had a better understanding of what sort
>> >>>> of frame numbers are expected by xenforeignmemory_map() when the target
>> >>>> VM is a PV. Is the Xen code the only documentation for this sort of
>> >>>> thing, or is there some place I could get a high-level overview?
>> >>>
>> >>>
>> >>>
>> >>> I am a bit confused with the rest of this e-mail. There are no concept of
>> >>> HVM or PV on Arm. This is x86 specific. For Arm, there is a single type
>> >>> of
>> >>> guest that borrow the goods of both HVM and PV.
>> >>>
>> >>> For instance, like HVM, the hardware is used to provide a separate
>> >>> address
>> >>> space for each virtual machine. Arm calls that stage-2 translation. So
>> >>> gfn
>> >>> != mfn.
>> >>
>> >>
>> >> I was not aware that the HVM/PV concept didn't apply directly to ARM.
>> >> Is there a document that summarizes the way Xen's address translation
>> >> works on ARM? The document I've been looking at is...
>> >>
>> >> https://wiki.xen.org/wiki/X86_Paravirtualised_Memory_Management
>> >
>> >
>> > The closest I would find explaining Xen's address translation scheme would
>> > be my talk at Xen Developer Summit:
>> >
>> > https://www.slideshare.net/xen_com_mgr/keeping-coherency-on-arm-julien-grall-arm
>>
>> After watching your talk and reading through the "Xen ARM with
>> Virtualization Extensions" whitepaper, I think I have a slightly better
>> understanding of how Xen ARM handles address translations. So which sort
>> of address does xenforeignmemory_map() expect?
>>
>> 1. stage 1 input (VA)
>> 2. stage 1 output / stage 2 input (IPA)
>> 3. stage 2 output (PA)
>>
>> (I'm assuming #1 or #2...)
>
> #2, guest physical addresses (also called psudo-physical addresses in
> the Arm manuals)

Ok. So if I wanted to map the first page of a guest's kernel, I could add its
"text offset" to ram base (from Julien's formula, "ram base + text offset") to
obtain the pfn to pass to xenforeignmemory_map()?

Thanks,
Brett S.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
https://lists.xen.org/xen-users