[Xen-devel] xen/arm: Domain not fully destroyed when using credit2

[Julien Grall <julien.grall@xxxxxxx>]

Hi all,

Before someone dig into the scheduler, I don't think this is an issue in 
credit2 but the use of it highlight a bug in another component (I think 
RCU).
Whilst testing other patches today, I have noticed that some part of the 
resources allocated to a guest were not released during the destruction.
The configuration of the test is:
        - ARM platform with 6 cores
        - staging Xen with credit2 enabled by default
        - DOM0 using 2 pinned vCPUs

The test is creating a guest vCPUs and then destroyed. After the test, 
some resourced are not released (or could be released a long time
after).

Looking at the code, domain resources are released in 2 phases:
	- domain_destroy: called when there is no more reference on the domain 
(see put_domain)
        - complete_domain_destroy: called when the RCU is quiescent

The function domain_destroy will setup the RCU callback 
(complete_domain_destroy) by calling call_rcu. call_rcu will add the 
callback into the RCU list and then will may send an IPI (see 
force_quiescent_state) if the threshold reached. This IPI is here to 
make sure all CPUs are quiescent before calling the callbacks (e.g 
complete_domain_destroy). In my case, the threshold has not reached and 
therefore an IPI is not sent.
On ARM, the idle will run when the pCPU has no work to do. This loop 
will wait to receive an interrupt (see wfi) and check if there is some 
work to do when the CPU has waken-up (i.e an interrupt was received).
The problem I encountered is the idle CPU will never receive interrupts 
(no timer, nor IPI...) and therefore never check whether the RCU has 
some work to do.
From my understanding, this is a bug in how RCU is handled (see comment 
above rcu_start_batch), it expects each CPU (no broadcast) to check 
whether there is RCU work. But this is relying on someone else (timer?) 
to fire an interrupt.
Any incoming interrupts will make a pCPU checking the RCU state. On ARM, 
the biggest source of IPI was credit1 or timer if a guest vCPU was 
scheduled on that pCPU. But it looks like the IPI traffic with credit2 
was reduced to none (which is a really good thing :)), and no guest 
timer was scheduled because no vCPU ever run on this pCPU.
I think the bug has always been here (both ARM and x86), but never 
detected because any incoming interrupts will make the pCPU to check the 
RCU state.
However, I am not sure how to resolve this issue. Any thoughts?

Cheers,

--
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel