Re: [xen-4.12-testing test] 169199: regressions - FAIL

[Jan Beulich <jbeulich@xxxxxxxx>]

On 08.04.2022 10:09, Roger Pau Monné wrote:
> On Fri, Apr 08, 2022 at 09:01:11AM +0200, Jan Beulich wrote:
>> On 07.04.2022 10:45, osstest service owner wrote:
>>> flight 169199 xen-4.12-testing real [real]
>>> http://logs.test-lab.xenproject.org/osstest/logs/169199/
>>>
>>> Regressions :-(
>>>
>>> Tests which did not succeed and are blocking,
>>> including tests which could not be run:
>>>  test-amd64-amd64-xl-qemut-debianhvm-i386-xsm 12 debian-hvm-install fail 
>>> REGR. vs. 168480
>>
>> While the subsequent flight passed, I thought I'd still look into
>> the logs here since the earlier flight had failed too. The state of
>> the machine when the debug keys were issued is somewhat odd (and
>> similar to the earlier failure's): 11 of the 56 CPUs try to
>> acquire (apparently) Dom0's event lock, from evtchn_move_pirqs().
>> All other CPUs are idle. The test failed because the sole guest
>> didn't reboot in time. Whether the failure is actually connected to
>> this apparent lock contention is unclear, though.
>>
>> One can further see that really all about 70 ECS_PIRQ ports are
>> bound to vCPU 0 (which makes me wonder about lack of balancing
>> inside Dom0 itself, but that's unrelated). This means that all
>> other vCPU-s have nothing at all to do in evtchn_move_pirqs().
>> Since this moving of pIRQ-s is an optimization (the value of which
>> has been put under question in the past, iirc), I wonder whether we
>> shouldn't add a check to the function for the list being empty
>> prior to actually acquiring the lock. I guess I'll make a patch and
>> post it as RFC.
> 
> Seems good to me.
> 
> I think a better model would be to migrate the PIRQs when fired, or
> even better when EOI is performed?  So that Xen doesn't pointlessly
> migrate PIRQs for vCPUs that aren't running.

Well, what the function does is mark the IRQ for migration only
(IRQ_MOVE_PENDING on x86). IRQs will only ever be migrated in the
process of finishing the handling of an actual instance of the
IRQ, as otherwise it's not safe / race-free.

>> And of course in a mostly idle system the other aspect here (again)
>> is: Why are vCPU-s moved across pCPU-s in the first place? I've
>> observed (and reported) such seemingly over-aggressive vCPU
>> migration before, most recently in the context of putting together
>> 'x86: make "dom0_nodes=" work with credit2'. Is there anything that
>> can be done about this in credit2?
>>
>> A final, osstest-related question is: Does it make sense to run Dom0
>> on 56 vCPU-s, one each per pCPU? The bigger a system, the less
>> useful it looks to me to actually also have a Dom0 as big, when the
>> purpose of the system is to run guests, not meaningful other
>> workloads in Dom0. While this is Xen's default (i.e. in the absence
>> of command line options restricting Dom0), I don't think it's
>> representing typical use of Xen in the field.
> 
> I could add a suitable dom0_max_vcpus parameter to osstest.  XenServer
> uses 16 for example.

I'm afraid a fixed number won't do, the more that iirc there are
systems with just a few cores in the pool (and you don't want to
over-commit by default). While for extreme cases it may not suffice,
I would like to suggest to consider using ceil(sqrt(nr_cpus)). But
of course this requires that osstest has a priori knowledge of how
many (usable) CPUs each system (pair) has, to be able to form such
a system-dependent command line option.

> Albeit not having such parameter has likely led you into figuring out
> this issue, so it might not be so bad.  I agree however it's likely
> better to test scenarios closer to real world usage.

True. One might conclude that we need both then. But of course that
would make each flight yet more resource hungry.

Jan