Re: [Xen-users] pvgrub2(-like?) booting methods for PVHv2 guests

[Juergen Gross <jgross@xxxxxxxx>]

On 07/02/18 15:02, Hans van Kranenburg wrote:
> On 02/07/2018 02:55 PM, Juergen Gross wrote:
>> On 07/02/18 14:14, Hans van Kranenburg wrote:
>>> On 02/07/2018 09:37 AM, Juergen Gross wrote:
>>>> On 06/02/18 19:34, Hans van Kranenburg wrote:
>>>>> On 02/06/2018 06:02 PM, Juergen Gross wrote:
>>>>>> On 06/02/18 17:53, Hans van Kranenburg wrote:
>>>>>>> On 02/06/2018 09:35 AM, Juergen Gross wrote:
>>>>>>>> On 05/02/18 20:49, Hans van Kranenburg wrote:
>>>>>>>>> On 01/25/2018 03:31 PM, Juergen Gross wrote:
>>>>>>>>>> On 25/01/18 15:12, Hans van Kranenburg wrote:
>>>>>>>>>>> On 01/25/2018 02:46 PM, Hans van Kranenburg wrote:
>>>>>>>>>>>> On 25/01/2018 13:29, Juergen Gross wrote:
>>>>>>>>>>>>> On 25/01/18 13:19, Andy Smith wrote:
>>>>>>>>>>>>>> Hi Hans,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 12:39:56PM +0100, Hans van Kranenburg 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> And now I get console output and things happen. Only it can't 
>>>>>>>>>>>>>>> find the disk.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I was trying similar thing (4.10 and PVH) and also ended up with 
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> guest with no block devices. I reported this on grub-devel:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>     
>>>>>>>>>>>>>> <http://lists.gnu.org/archive/html/grub-devel/2018-01/msg00018.html>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> as I was thinking this was not a Xen problem since same thing 
>>>>>>>>>>>>>> boots
>>>>>>>>>>>>>> okay outside grub with direct kernel boot.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Juergen did reply and said I needed this kernel patch in the 
>>>>>>>>>>>>>> guest:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>     
>>>>>>>>>>>>>> <https://lists.xen.org/archives/html/xen-devel/2017-11/msg01681.html>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But I think you have this don't you?
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, see my earlier mail with all the steps that I did, step 6.
>>>>>>>>>>>>
>>>>>>>>>>>>> As the ACPI tables are found, I'd say yes. :-)
>>>>>>>>>>>>
>>>>>>>>>>>> dmesg output is pretty different when I boot directly with the 
>>>>>>>>>>>> kernel
>>>>>>>>>>>> and initrd copied on the dom0.
>>>>>>>>>>>>
>>>>>>>>>>>> Remember, it's the same kernel/initrd, and without grub in between 
>>>>>>>>>>>> it
>>>>>>>>>>>> boots with all vcpus network and disk.
>>>>>>>>>>>>
>>>>>>>>>>>> With grub in between, this at least does look suspicious:
>>>>>>>>>>>>
>>>>>>>>>>>> [    0.032110] PCI: System does not support PCI
>>>>>>>>>>>
>>>>>>>>>>> Eh, that's in both, stay awake Hans.
>>>>>>>>>>>
>>>>>>>>>>>> And yes, there are also no successful netfront lines.
>>>>>>>>>>>>
>>>>>>>>>>>> Without grub: http://paste.debian.net/plainh/7120cef2
>>>>>>>>>>>> With grub: http://paste.debian.net/plainh/426bed60
>>>>>>>>>>>
>>>>>>>>>>> Or the diff between them, which shows what changes when inserting 
>>>>>>>>>>> grub
>>>>>>>>>>> in between:
>>>>>>>>>>>
>>>>>>>>>>> http://paste.debian.net/plainh/52b2d618
>>>>>>>>>>>
>>>>>>>>>>> I must admit I don't know too much (yet) about all those changed 
>>>>>>>>>>> lines,
>>>>>>>>>>> but this next is also a very interesting change?...
>>>>>>>>>>>
>>>>>>>>>>> -Booting paravirtualized kernel on Xen PVH
>>>>>>>>>>> +Booting paravirtualized kernel on Xen HVM
>>>>>>>>>>
>>>>>>>>>> Aah, yes, this should be the reason for the problems.
>>>>>>>>>>
>>>>>>>>>> I addressed the ACPI problem first. What is missing now is to set PVH
>>>>>>>>>> mode when booting via grub.
>>>>>>>>>>
>>>>>>>>>> So please stay tuned...
>>>>>>>>>
>>>>>>>>> For my info... Is this something like flipping a bit somewhere, or 
>>>>>>>>> will
>>>>>>>>> it involve a whole lot more complicated wizardry?
>>>>>>>>>
>>>>>>>>> Real question is of course if there's a PoC solution to apply that I 
>>>>>>>>> can
>>>>>>>>> use now? ;-] I'm throwing custom build kernels at my test environment,
>>>>>>>>> and if I could save the time (and mistakes) of manually editing the
>>>>>>>>> guest files and instead hit the pvgrub2+PVHv2 path many times to 
>>>>>>>>> test...
>>>>>>>>
>>>>>>>> So I looked into this briefly and discovered that my memory really 
>>>>>>>> needs
>>>>>>>> the backup I have on my hard disk: you want commit
>>>>>>>> 418492ba40b2c2bbdaf1a169aac5b1673bde8189 which was for 4.15.
>>>>>>>
>>>>>>> Aha. Well, I can as well better jump to 4.15 now, since picking that
>>>>>>> patch on 4.14.17 shows I need more other patches it depends on, related
>>>>>>> to struct x86_legacy_features and x86_hyper_init reorganization.
>>>>>>>
>>>>>>> So, I just built a 4.15.1 kernel (using gcc 7.2.0), which definitely has
>>>>>>> this one in it:
>>>>>>>   418492ba40b2 x86/virt/xen: Use guest_late_init to detect Xen PVH guest
>>>>>>>
>>>>>>> I can boot via pvgrub2 (the Xen that's running now is current
>>>>>>> stable-4.10 plus the RSDP for PVH guest near 4GB patch).
>>>>>>>
>>>>>>> But, it still tells me...
>>>>>>>   Booting paravirtualized kernel on Xen HVM
>>>>>>>
>>>>>>> ...and then hangs somewhere halfway.
>>>>>>>
>>>>>>> Full boot log here: http://paste.debian.net/plainh/ae3ea14b
>>>>>>
>>>>>> You didn't add the RSDP detection patch, right? In the log I see:
>>>>>>
>>>>>> ACPI BIOS Error (bug): A valid RSDP was not found (20170831/tbxfroot-244)
>>>>>
>>>>> Yes, I didn't have those. Argh...
>>>>>
>>>>> However, after putting the 'xen: re-enable booting as Xen PVH guest' v2
>>>>> patches on top, same happens.
>>>>
>>>> Sure. Those don't take RSDP from boot parameters set by grub2.
>>>>
>>>> You need to apply the 4 patch series from:
>>>>
>>>> git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/boot
>>>>
>>>> Commit-Ids are:
>>>> 2f74cbf947f45fa082dda8eac1a1f1299a372f49
>>>> 0c89cf36424f7c1177de8a5712514d7cc2eb369f
>>>> 88750a6c33f813b815516990f01fb5ee488c477e
>>>> 930ba49b2ce7b09a5eddc21385fd944ba6b4e829
>>>
>>> Ah, D'OH. The patch sets currently bite each other. And, for some reason
>>> I messed up and was wrongly thinking I already had those in 4.15, since
>>> which is entirely not the case. No idea why I thought that.
>>>
>>> So, since I'm testing pvgrub2, I threw the 'xen: re-enable booting as
>>> Xen PVH guest' out again, and applied those 4 again on 4.15.1.
>>>
>>> Now I at least have this...
>>> [    0.000000] Booting paravirtualized kernel on Xen PVH
>>> ...and no more 'A valid RSDP was not found'...
>>>
>>> ...but quite some complaining going on about ACPI, like 'BIOS bug: APIC
>>> version mismatch' and 'ACPI BIOS Warning (bug): Incorrect checksum in
>>> table' however, and per cpu there's a 10 second boot delay (didn't see
>>> that one before yet), and I have only 1 vcpu in the end etc...
>>>
>>> Full output:
>>> http://paste.debian.net/plainh/262cfbcf
>>
>> e820: BIOS-provided physical RAM map:
>> BIOS-e820: [mem 0x0000000000000000-0x00000000fbffffff] usable
>> BIOS-e820: [mem 0x00000000fc000000-0x00000000fc000fff] ACPI data
>> BIOS-e820: [mem 0x00000000fc001000-0x00000000fc008fff] usable
>> BIOS-e820: [mem 0x00000000fc009000-0x00000000fc009fff] ACPI data
>> BIOS-e820: [mem 0x00000000fc00a000-0x00000000fedfffff] usable
>>
>> and
>>
>> ACPI: Early table checksum verification disabled
>> ACPI: RSDP 0x00000000FC009000 000024 (v02 Xen   )
>> ACPI: XSDT 0x00000000FC007FB0 000034 (v01 Xen HVM 00000000 HVML 00000000)
>> ACPI: FACP 0x00000000FC007D70 00010C (v05 Xen HVM 00000000 HVML 00000000)
>> ACPI: DSDT 0x00000000FC001050 000003 (v164 �mn FE942AD6 ?    FC001510)
>> ACPI:      0x00000000FC001010 000001 (v08 <-   6DB08FA4      62757267)
>> ACPI:      0x00000000FC001010 000001 (v08 <-   6DB08FA4      62757267)
>> ACPI: APIC 0x00000000FC007E80 00007C (v02 Xen HVM 00000000 HVML 00000000)
>>
>> Seems as if some ACPI data areas are missing in the E820 map.
>>
>> Can you try configuring the guest with e.g. 3GB memory instead of 4GB?
> 
> Instead of 100 GiB you mean? :]

Hmm, this wasn't reflected in the E820 map of the guest.

Would be interesting to see the output of xl -vvv create ... with the
100GB guest. I guess there is some 32 bit truncation somewhere in the
Xen tools...

> type="pvh"
> kernel = "/yolo/grub-i386-xenpvh-fire-ze-missile"
> memory = 102400
> vcpus = 10
> 
> When I change that to...
> 
> memory = 3072
> vcpus = 4
> 
> ...I get this:
> 
> http://paste.debian.net/plainh/f8676322

So this seems to work now.


Juergen


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users