Re: [Xen-devel] [qemu-upstream-4.11-testing test] 136184: regressions - FAIL

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Tue, 4 Jun 2019, Ian Jackson wrote:
> Stefano Stabellini writes ("Re: [Xen-devel] [qemu-upstream-4.11-testing test] 
> 136184: regressions - FAIL"):
> > I agree with you it would be desirable to test both LIVEPATCH and
> > non-LIVEPATCH, and I understand about limitation of resources and test
> > matrix explosion.
> > 
> > Given the chance, I think it would be better if we had an explicit test
> > about LIVEPATCH rather than a "hidden" enablement of it within another
> > different test. Or maybe just call it out explicitly, renaming the test
> > run to qemu-upstream-livepatch or something like that. In any case, I'll
> > leave it to you.
> 
> I think maybe you have misunderstood ?
>
> The thing that triggers this bug, here, is *compiling* Xen with
> CONFIG_LIVEPATCH *disabled*.

I followed, but I mistyped inverting the condition.


> So, in fact, if it is a hidden anything, it is a hidden *dis*ablement
> of a feature which is deliberately only compiled in, and only tested
> on, tests of the xen-* branches.
> 
> That *disabling* this feature would cause a regression is surprising,
> and I think this is only the case because Xen only works by accident
> on these boxes ?  (Considering the discussion of ARM ARM violations.)

Yes, that is the current thinking.


> To make it an "explicit" test as you suggest would involve compiling
> Xen an additional time.  I guess that would actually be changing some
> tests on xen-* branches to a version of Xen compiled *without*
> livepatch.  Right now we build
> 
> most other branches
>    Xen amd64  with XSM   no livepatch
>    Xen armhf  no   XSM   no livepatch
>    Xen arm64  with XSM   no livepatch
> 
> xen-* branches
>    Xen amd64  with XSM   with livepatch
>    Xen armhf  no   XSM   with livepatch
>    Xen arm64  with XSM   with livepatch
> 
> What without-livepatch build should be added to the xen-* branches ?
> And in which tests should it replace the existing with-livepatch
> builds ?  Should I just pick one or two apparently at random ?
> 
> NB that I doubt the livepatch maintainers have much of an opinion
> here.  We would normally expect that compiling in livepatching might
> break something but that compiling it out would be fine.  So the
> current situation is good from that point of view and we might even
> worry that changing some of the existing tests to not have
> livepatching compiled in might miss some actual livepatch-related
> bugs.  My normal practice is to try to enable as much as is relevant
> and might break things.

I think it is a good practice in general, especially if we only have the
resources for one type of tests.

My point is that differences in the kconfig (except maybe for drivers
such as UARTs) can have an important impact either directly or
indirectly, like in this case. The problem will only get worse as more
kconfig options will be introduced. We cannot test all possible
combinations. However, I think different kconfigs deserve to be called
out explicitly in the tests. This is what I was trying to say. Maybe we
can pick 2 or 3 "interesting" Xen kconfigs and run tests for them. But
of course this is predicated on hardware and resource availability that
we might not have.

Specifically in your matrix above, maybe:

 xen-* branches
    Xen amd64  kconfig_1
    Xen amd64  kconfig_2
    Xen armhf  kconfig_1
    Xen arm64  kconfig_1
    Xen arm64  kconfig_2

where kconfig_1 has few options as possible enabled (no XSM, no
LIVEPATCH) and kconfig_2 has as many options as possible enabled (both
XSA and LIVEPATCH). Note that I only added kconfig_1 to the armhf line
because it doesn't look like a good idea to run both on arm32. One day
it would be great to add a kconfig_3 with a hand-picked set of options,
and maybe more (kconfig_4, maybe a random kconfig, etc.).

The other branches ideally would follow the same patten. If we don't
have enough resources, they could run with kconfig_1 or kconfig_2 only.

Funnily enough, we discussed something very similar just this morning in
the FuSa Call because we'll need a special kconfig for safety
certifications to be tested. It might end up looking very much like
kconfig_1 (CC'ing Lars here to connect the dots.)


> But what we have here is *not* a livepatch-related bug.  It has
> nothing to do with livepatch.  It is just that by luck, compiling Xen
> *with* livepatching somehow masks the random failure, presumably by
> changing exact orderings and timings of memory accesses etc.
> 
> Does that make sense ?

Yes, I got it.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel