Re: [Xen-devel] [PATCH 0/2] xen: credit2: fix vcpu starvation due to too few credits

[Jürgen Groß <jgross@xxxxxxxx>]

On 12.03.20 14:44, Dario Faggioli wrote:
> Hello everyone,
>
> There have been reports of a Credit2 issue due to which vCPUs where
> being starved, to the point that guest kernel would complain or even
> crash.
>
> See the following xen-users and xen-devel threads:
> https://lists.xenproject.org/archives/html/xen-users/2020-02/msg00018.html
> https://lists.xenproject.org/archives/html/xen-users/2020-02/msg00015.html
> https://lists.xenproject.org/archives/html/xen-devel/2020-02/msg01158.html
>
> I did some investigations, and figured out that the vCPUs in question
> are not scheduled for long time intervals because they somehow manage to
> be given an amount of credits which is less than the credit the idle
> vCPU has.
>
> An example of this situation is shown here. In fact, we can see d0v1
> sitting in the runqueue while all the CPUs are idle, as it has
> -1254238270 credits, which is smaller than -2^30 = −1073741824:
>
>      (XEN) Runqueue 0:
>      (XEN)   ncpus              = 28
>      (XEN)   cpus               = 0-27
>      (XEN)   max_weight         = 256
>      (XEN)   pick_bias          = 22
>      (XEN)   instload           = 1
>      (XEN)   aveload            = 293391 (~111%)
>      (XEN)   idlers: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
>      (XEN)   tickled: 00,00000000,00000000,00000000,00000000,00000000,00000000
>      (XEN)   fully idle cores: 
> 00,00000000,00000000,00000000,00000000,00000000,0fffffff
>      [...]
>      (XEN) Runqueue 0:
>      (XEN) CPU[00] runq=0, sibling=00,..., core=00,...
>      (XEN) CPU[01] runq=0, sibling=00,..., core=00,...
>      [...]
>      (XEN) CPU[26] runq=0, sibling=00,..., core=00,...
>      (XEN) CPU[27] runq=0, sibling=00,..., core=00,...
>      (XEN) RUNQ:
>      (XEN)     0: [0.1] flags=0 cpu=5 credit=-1254238270 [w=256] load=262144 
> (~100%)
>
> This happens bacause --although very rarely-- vCPUs are allowed to
> execute for much more than the scheduler would want them to.
>
> For example, I have a trace showing that csched2_schedule() is invoked at
> t=57970746155ns. At t=57970747658ns (+1503ns) the s_timer is set to
> fire at t=57979485083ns, i.e., 8738928ns in future. That's because credit
> of snext is exactly that 8738928ns. Then, what I see is that the next
> call to burn_credits(), coming from csched2_schedule() for the same vCPU
> happens at t=60083283617ns. That is *a lot* (2103798534ns) later than
> when we expected and asked. Of course, that also means that delta is
> 2112537462ns, and therefore credits will sink to -2103798534!
Current ideas are:

- Could it be the vcpu is busy for very long time in the hypervisor?
  So either fighting with another vcpu for a lock, doing a long
  running hypercall, ...

- The timer used is not reliable.

- The time base is not reliable (tsc or whatever is used for getting
  the time has jumped 2 seconds into the future).

- System management mode has kicked in.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel