Re: [Xen-devel] PAE guest address maps?

[Randy Thelen <rthelen@xxxxxxxxxx>]

Steven Hand wrote:

> Take a look at xen/include/asm-x86/config.h ; this defines the
> overall memory layout for x86_32, PAE & x86_64 systems, as well
> as including some comments that should help you.
Thanks, Steven.  I hadn't seen that file before.

However, I'm still confused (or frustrated, I'm not entirely sure  
which!).  Here's the memory layout table from that file:
/*
* Memory layout (high to low):                          SIZE   PAE-SIZE
*                                                       ------ ------
*  I/O remapping area                                   ( 4MB)
*  Direct-map (1:1) area [Xen code/data/heap]           (12MB)
*  Per-domain mappings (inc. 4MB map_domain_page cache) ( 8MB)
*  Shadow linear pagetable                              ( 4MB) ( 8MB)
*  Guest linear pagetable                               ( 4MB) ( 8MB)
*  Machine-to-physical translation table [writable]     ( 4MB) (16MB)
*  Frame-info table                                     (24MB) (96MB)
*   * Start of guest inaccessible area
*  Machine-to-physical translation table [read-only]    ( 4MB) (16MB)
*   * Start of guest unmodifiable area
*/

When my guest domain starts (FreeBSD in PAE mode), I begin looking at  
the page tables.  I note that I've modified Xend (the libxc code) to  
build 4 pages of Page Directories because FreeBSD uses the model of  
assuming that there are four Page Directories linearly mapped.  This  
way it can just forget about the 4 PTD pointers in the L3 page.  It  
means that 16KB are consumed for every address space, but it greatly  
simplifies the code.
(gdb) x/4x xen_start_info->pt_base
0xc03ae000:     0x00000000fb152001      0x00000000fb151001
0xc03ae010:     0x00000000fb150001      0x00000000fb14f001

And, my kernel starts at virtual address 0xC000.0000, so only page 3  
(in 0..3) of the 4 pages needs to have anything.  And sure enough,  
that's what I find:
# ./xenctrl 4 -l3 0xfb153 dump64 0xc03af000 0x4000  # I'm dumping 64  
bit values
0xc03af000:      -- page of zeros --
0xc03b0000:      -- page of zeros --
0xc03b1000:      -- page of zeros --
0xc03b2000:      0x00fb14e067    0x00fb14d067    0x00fb14c067     
0x00fb14b067
        ...
0xc03b2d60:      0x00fbc000e1    0x00fba000e1    0x0000000000     
0x0000000000
        ...
0xc03b2da0:      0x00018001e3    0x0001a001e3    0x0001c001e3     
0x0001e001e3
0xc03b2dc0:      0x00020001e3    0x00022001e3    0x00024001e3     
0x00026001e3
0xc03b2de0:      0x00028001e3    0x0002a001e3    0x0002c001e3     
0x0002e001e3
        ...
0xc03b2f20:      0x00fbc001e3    0x00fba001e3    0x0000000000     
0x0000000000
        ...
0xc03b2f60:      0x00fb152063    0x00fb151063    0x00fb150063     
0x00fb14f063
        ...
0xc03b2fa0:      0x0000224063    0x0000225063    0x0000226063     
0x0000227063
0xc03b2fc0:      0x0000211063    0x0000212063    0x0000213063     
0x0000214063
0xc03b2fe0:      0x0000215063    0x0000216063    0x0000217063     
0x0000bfe063
#

The four page table entries at 0xc03b2000 map my kernel (base address  
0xC000.0000 means the first entries in page 3 of 0..3).  And, my  
kernel is 8MB, so all is well.
The following entries are broken-out as follows:

0xC03B2D60 has two entries which map to base address 0xF580.0000 for  
4MB.  This is 168 MB below 4GB.  So, this should be the machine-to- 
physical translation table.  However, I can't map it from a tool in  
Dom0, nor can I see it in GDB:
(gdb) x/32w 0xf5800000
0xf5800000:     Cannot access memory at address 0xf5800000

Nor can I map the physical pages (of the L1 page table!):

# ./xenctrl 4 -l3 0xfb153 dump64-mfn 0xfbc00000 0x1000
Error: Couldn't map mfn fbc00000

This tells me I can't determine the L0 pages used.  That's too bad.

So, I'm not sure how I'm suppose to use this.  My guest domain -can-  
see the F58 address region.  And, so I can bcopy() the data into a  
visible space for gdb.  In the following snippet, I am using two  
tools: GDB and a homebrew tool to view domain 0 pages, xenctrl:
(gdb) call bzero(0xc0000000, 0x1000)               # Just to certify  
that I can affect 0xC0000000
$22 = 0

# ./xenctrl 4 -l3 0xfb153 dump 0xc0000000 0x1000   # Verify that I  
can view it
0xc0000000:      -- page of zeros --

(gdb) call bcopy(0xf5800000, 0xc0000000, 0x1000)   # See what's in  
the machine-to-phys table
$23 = -897581056

# ./xenctrl 4 -l3 0xfb153 dump 0xc0000000 0x100
0xc0000000:      0xffffffff    0xffffffff    0xffffffff     
0xffffffff    0xffffffff    0xffffffff    0xffffffff    0xffffffff
0xc0000020:      0xffffffff    0xffffffff    0xffffffff     
0xffffffff    0xffffffff    0xffffffff    0xffffffff    0xffffffff
0xc0000040:      0xffffffff    0xffffffff    0xffffffff     
0xffffffff    0xffffffff    0xffffffff    0xffffffff    0xffffffff
0xc0000060:      0xffffffff    0xffffffff    0xffffffff     
0xffffffff    0xffffffff    0xffffffff    0xffffffff    0xffffffff
0xc0000080:      0xffffffff    0xffffffff    0xffffffff     
0xffffffff    0xffffffff    0xffffffff    0xffffffff    0xffffffff
0xc00000a0:      0xffffffff    0xffffffff    0xffffffff     
0xffffffff    0xffffffff    0xffffffff    0xffffffff    0xffffffff
0xc00000c0:      0xffffffff    0xffffffff    0xffffffff     
0xffffffff    0xffffffff    0xffffffff    0xffffffff    0xffffffff
0xc00000e0:      0xffffffff    0xffffffff    0xffffffff     
0xffffffff    0xffffffff    0xffffffff    0xffffffff    0xffffffff
I'm not sure what I'm suppose to make of those numbers.  If I view  
the whole page (all 4K) I get long sequences of 0xffff's and 0x5555's.
Is there any documentation on the machine-to-physical region of  
memory?  Any sample code on its usage?  I tried looking in Linux, but  
I wasn't able to identify the machine-to-physical conversion macros  
or functions for x86-32 on PAE.

Moving on to the next address space, I find Page Directory Entries  
for the address space 0xFD800000.  I have learned that this address  
space is the recursive page table address space.  I.e., in theory my  
guest domain could perform virtual to machine page translations here  
with the algorithm I stated last Wednesday: ((virtual-address &  
0xFFFF.F000) >> 9) +0xFD80.0000 would generate the address of the PTE  
for the current virtual page (assuming the virtual address is valid  
in the first place!).  But, the comment in the config.h file says  
this is beyond the "Start of guest inaccessible area."  Which is a  
damn shame:
(gdb) call bcopy(0xfde00000, 0xc0000000, 8)
<hang>

Because I can get access to the data from Dom0 (I actually executed  
the following steps -before- the steps above, which caused a hang):
(gdb) p/x ((0xC0000000 & 0xFFFFF000) >> 9) +0xFD800000
$26 = 0xfde00000
(gdb) x/1g $26
0xfde00000:     0x00000000d1620063
(gdb)

It would be -very- useful to my program if I could use the existing  
recursive page table.
This all comes to pass because I'm trying to build the recursive  
address table for FreeBSD and I need information I can't find; i.e.,  
the machine to physical table.
-- Randy



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel