Re: [Xen-users] issues with pvh mode for dom0 and domU on xen 4.5.0 / kernel 3.18.9

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

Hello,

El 15/06/15 a les 15.05, Atom2 ha escrit:
> Am 15.06.15 um 09:33 schrieb Roger Pau Monné:
>> Hello,
> Roger, thanks for your reply.
>>
>> El 15/06/15 a les 0.46, Atom2 ha escrit:
>>> Hi guys,
>>> I recently switched from xen 4.4.2 to 4.5.0 after it became stable on
>>> gentoo (kernel version is 3.18.9) and thought I'd give pvh for both dom0
>>> and domUs a spin - unfortunately with not too much success:
>>>
>>>
>>> DOM0:
>>> For dom0 I simply added dom0pvh=1 to the xen command line. The system
>>> booted up and I was also able to confirm that dom0 is running in the
>>> correct mode by checking with xen-detect.
>>>
>>> What I however found out is that xen creates a bunch of error/warning
>>> messages some of which make it to xl's dmesg file (the majority seems to
>>> get dropped due to rate limits). None of these messages are there when
>>> started without dom0pvh=1. Please see the two attached files for a
>>> comparision between the xl dmesg for PVH ("dmesg.xl.pvh") and the xl
>>> dmesg in non-PVH mode ("dmesg.xl").
>>>
>>> There are a few (most likely irrelevant) differences at line 109 to 111
>>> relating to messages about the "Start info", the "Page tables", and the
>>> "Boot stack". The main difference is in the additional lines in the file
>>> "dmesg.xl.pvh" on line numbers 121-122 and 124-156 including 8 lines
>>> about suppressed messages totalling in excess of 236.000 (ignored)
>>> messages.  It's probably worth noteing that no further messages make it
>>> to xl's dmesg and also /var/log/messages does not have anything strange
>>> once the dom0 is up and running.
>> Those are errors from the IOMMU, your BIOS is probably missing some RMRR
>> regions. Do you know which devices are at 0000:00:1a.0 and 0000:00:02.0?
> Sure:
> 0000:00:1a.0 is one of the two USB controllers as part of the chipset;
> lspci output for this device is as follows:
> 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
> Family USB Enhanced Host Controller #2 (rev 05)
> 
> 0000:00:02.0 is the integrated graphics chip (IGD) in the XEON processor;
> lspci output for this device is as follows:
> 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200
> Processor Family Integrated Graphics Controller (rev 09)

I've seen problems with other integrated Intel gfx cards, I still have
to figure out what's wrong.

> For reference: The system is a C206 chipset motherboard with a XEON
> E3-1260L processor
> The system has 32GB ECC memory installed
> 
>>
>> (XEN) [VT-D]DMAR:[DMA Read] Request device [0000:00:1a.0] fault addr
>> dc086000, iommu reg = ffff82c000203000
>> (XEN) [VT-D]DMAR: reason 06 - PTE Read access is not set
>>
>> The above error looks reasonable, this memory is between 3 and 4 GiB
>> which matches the MMIO hole.
>>
>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:02.0] fault addr
>> 72c481000, iommu reg = ffff82c000201000
>> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
>>
>> This however is very high address, and is within a region marked as
>> usable in the memory map:
>>
>> (XEN)  0000000100000000 - 000000081e600000 (usable)
>>
>> There are some patches on the list to add additional RMRR regions on the
>> command line, however they are not committed yet, so there's not much
>> you can try right now (apart from trying it on a different box).
> That probably means I'll have to wait and see for further developments,
> most likely in 4.6
> I guess there's also nothing for you guys to be gained from this report ...

Yes, PVH is still experimental, and it will take some time to nail it down.

>>
>>> It, however, appears that the pvh dom0 compared to the standard dom0
>>> consumes _significantly_ more CPU time as shown by "xl info" from within
>>> dom0 - which to me seems counter-intuitive given my (limited)
>>> understanding of what pvh tries to achieve.
> Any idea about that ...

No, we are not yet at the point of doing performance measurements, not
until the ABI is finished, then we can start speaking about performance.
FWIW I have some old PVH/PV/PVHVM performance comparison here:

http://xenbits.xen.org/people/royger/talks/fosdem2014.pdf

>>>
>>>
>>> DOMU:
>>> For a test domU I just added pvh=1 to it's (otherwise unchanged)
>>> configuration file and tried to start the domU by issuing
>>> xl -v -v -v /path/to/config/file -c
>>>
>>> The domU did not come up at all (but works flawlessly when commenting
>>> out the pvh=1 configuration line); details of the xl command output for
>>> the failed attempt can be found in the attached file xl.domU. I honestly
>>> can't make much sense out of the error message which in essence seems to
>>> complain about an unsupported feature and a missing file or directory
>>> before giving up.
>> AFAICT from the log provided you seem to be trying to launch a MiniOS
>> based guest with pvh=1, which is not supported. MiniOS doesn't support
>> the PVH mode yet.
> Well, I am using pvgrub to fire up the domUs and provide the grub
> configuration via a ramdisk:
> 
> === relevant part of my config file ===
> kernel                  = '/usr/libexec/xen/boot/pv-grub-x86_64.gz'
> ramdisk                 = '/etc/xen/guests/grub.d/mysql.grub'
> === end relevant part of config file ===
> 
> I suppose pvgrub is then what you refer to with the term MiniOS.
> Because other than this it's just a standard linux kernel w/ an
> initramfs that get's booted within the domU.

Yes, it at least reports it's using the MiniOS kernel, which means it
doesn't have PVH support (because MiniOS itself doesn't have PVH support
yet).

> In other words: pvh currently does not work with pvgrub at all?
> Any plans to change this - pvgrub IMHO is the most flexible tool to
> start a domU.
> Once you get used to that, there's no way back ...

Not right now, we are planning to do ABI changes to the PVH interface,
which means that even if pvgrub was ported to PVH right now it would
have to be modified afterwards in order to cope with the ABI changes.

Roger.


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users