Re: [Xen-devel] Criteria / validation proposal: drop Xen

[Lars Kurth <lars.kurth.xen@xxxxxxxxx>]

Hi all,

I am going to step in here with my hat as Xen Project community
manager. We had a discussion about this issue as part of last week's
community call. I CC'ed a number of stake-holders, which probably
should have been on the thread such as ITL (which builds QubesOS
on top of Fedora) and Michael A Young (the Xen package manager).

First of all apologies that this issue has lingered so long. Key
members of the community were not aware of the issues raised in
this thread, otherwise we would have acted earlier. With this in
mind, please in future raise issues with me, on xen-devel@,
committers@ or the Xen-Fedora package manager. The Xen Community
would like to see Fedora running as guest: in fact it would be
somewhat odd if there was a Xen-Dom0 package and guest support
didn't work. And there are some downstreams such as QubesOS,
which depend on this support.

> On 6 Jul 2017, at 13:45, Adam Williamson <adamwill@xxxxxxxxxxxxxxxxx> wrote:
> 
> On Thu, 2017-07-06 at 15:13 -0400, Konrad Rzeszutek Wilk wrote:
>> On Thu, Jul 06, 2017 at 11:59:01AM -0700, Adam Williamson wrote:
>>> Hi, folks! A while ago, Xen virtualization functionality was added to
>>> the criteria and the validation test case set, on the understanding
>>> that Oracle would provide testing for it (and help fix bugs as they
>>> arose).
>>> 
>>> For the last couple of releases we really have not had any such testing
>> 
>> We had been doing the testing, it just that we (or rather me and
>> Dariof) seem to get a wind of this at the last minute. Not sure exactly
>> how to fix that thought.
> 
> Well, I mean, every few *days* a compose gets nominated for validation
> testing, and a mail is sent to test-announce. Just check your test-
> announce archives for mails with "nominated for testing" in their
> subject lines, and you'll see dozens. Is this not sufficient
> notification?

We discussed this at the community call and came to the conclusion that
we can run regular tests of Fedora RC's as part of our OSSTEST
infrastructure. Ian Jackson volunteered to implement this, but there
are some questions on
a) The installer (which we can handle ourselves)
b) When we would trigger a test - aka is there some trigger other than the
c) How would results best be reported back to Fedora

Apologies, I am not very familiar with how the Fedora Test group works.
Is there some documentation which would help integrate what you to with
the test system of another open source project?

>>> from Oracle. On that basis, I'm proposing we remove this Final
>>> criterion:
>> 
>> s/Oracle/Xen Project/ I believe?
> 
> Perhaps, it's just that it always seemed to be you doing the testing,
> so they got a bit conflated :)

Can we come to some arrangement, by which such issues get communicated
to the Xen Project earlier? Aka me, xen-devel@ or committers@

>>> "The release must boot successfully as Xen DomU with releases providing
>>> a functional, supported Xen Dom0 and widely used cloud providers
>>> utilizing Xen."
>>> 
>>> and change the 'milestone' for the test case -
>>> https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_Xen_Para_Virt -
>>> from Final to Optional.
>>> 
>>> Thoughts? Comments? Thanks!
>> 
>> I would prefer for it to remain as it is.
> 
> This is only practical if it's going to be tested, and tested regularly
> - not *only* on the final release candidate, right before we sign off
> on the release. It needs to be tested regularly throughout the release
> cycle, on the composes that are "nominated for testing".

Would the proposal above work for you? I think it satisfies what you are
looking for. We would also have someone who monitors these test results
pro-actively.

Then, there are the specific grub issues that need resolving
[A1] https://bugzilla.redhat.com/show_bug.cgi?id=1486002
     (and a recently filed duplicate @
      https://bugzilla.redhat.com/show_bug.cgi?id=1691559) caused by
      [A2])
[A2] https://bugzilla.redhat.com/show_bug.cgi?id=1703700
[B1] https://bugzilla.redhat.com/show_bug.cgi?id=1264103

The first makes it harder to boot Fedora _dom0_ (but workarounds exist).
While it is unpleasant, it doesn't break the release criterion, but
probably has deterred people from testing. The second one [B1] is about
Fedora _domU_, which breaks the release criterion.

Marek and Michael had a discussion about these and there was also
a summary from Daniel.

== On [A1]/[A2] ==
Lack of GRUB2 multiboot2/module2 commands in Fedora/RH which does not
allow you load Xen as dom0 on EFI platforms with or without secure
boot. Here are some relevant snippets from the discussions:

"In general both modules were dropped due to CVE-2015-5281 (grub2:
modules built in on EFI builds that allow loading arbitrary code,
circumventing secure boot) [A3][A4]. Of course this makes sense
because we do not want to break UEFI secure boot. But this means
that you cannot boot Xen dom0 on UEFI platforms. The Multiboot2
protocol support is required to do that. Potentially you can
use xen.efi directly but AFAICT many people prefer to use GRUB2.
The CVE issue does not exist in GRUB2 upstream. It was fixed at
the end of 2019."

Is there any chance these can get upstreamed into Fedora/RH?

"However, this is only one piece of the puzzle. Another is a
requirement for additional set of patches for Xen which allow
you to load xen.efi instead of xen.gz using Mulitboot2. I
started work on it last year but it is currently stalled."

I have taken an action to get this resolved
(aka find someone to do the work).

[A3] https://access.redhat.com/security/cve/cve-2015-5281
[A4] http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-5281
[A5] https://lists.xenproject.org/archives/html/xen-devel/2018-06/msg01292.html

== On [B1] / grub2-switch-to-blscfg  ==
This issue is about Fedora _domU_ and breaks the release
criterion. And looks like, it wasn't tested at all.

"blscfg is okay in _dom0_ - it looks like the xen setup still
gets put in non-blscfg format, and doesn't seem to matter in
HVM _domU_."

"The big issue is _domU_ in PV which would need a fair amount
of work in pygrub to fix properly, including reading variables
from grubenv and extracting details from the loader files. This
is really something to be fixed on the Xen side ... I do keep
intending to have a look at it myself though I may not get around
to it."

Instead of fixing pygrub, it would be better, more future proof
and easier to "use pvgrub2 instead. To be honest, its very unclear
to me why would anyone want to use pygrub, when pvgrub2 exists.
pygrub is much more fragile (as it needs to re-implement a
parser for 3rd-party configuration format, without stable
specification) and less secure - it does that in dom0, including
mounting domU controlled disk.

That said, the pvgrub2 option also requires some work, because:
- Fedora grub2 packages do not include the "xen" target platform
- Non-Fedora grub2 package don't have blscfg support
- If we'd talk about PVH (which isn't the case here), it requires grub
  2.04, which is at RC1 and isn't packaged for Fedora yet"

That would be much simpler, if blscfg was upstreamed into grub2 by
Fedora community members. Do you know whether the Fedora has plans
to do this?

In any case, I have taken an action to get this resolved
(aka find someone to do the work).

@xen-devel: this should probably be discussed separately, such that
we don't flood test@fedoraproject with unnecessary traffic

== In Summary ==
I think we can find a way forward on the testing side. Would
the proposal work for you?

Resolving the current blockers, this seems to have been caused by a
lack of communication or not understanding the impact of the
grub2-switch-to-blscfg in Fedora. In any case, we are where we are.

Best Regards
Lars

Attachment: signature.asc
Description: Message signed with OpenPGP

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel