Re: XEN 4.11 PV questions

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 11/04/2020 15:10, Asaf Fisher wrote:
> Hello,
> In general I have a intel family 6 model 94 and xen does not support
> so I want to add support to it.

Model 94 (== 0x5e) is Skylake, which has been around for quite a while
now.  (Alternatively, if you mean 0x94, I'm not sure that is even a
production CPU.)

Which CPU do you have, and what is actually going wrong?

> For the question:
>
> I’m trying to understand exactly how and when dom0’s vCPU gets a
> runtime and where in the code is like the “vmenter”(I know there is no
> such a thing in pv..)

Mechanically, PV guests running under Xen is just like regular
userspace.  You get there via IRET/SYSRET.

>
> So:
> 1. I got 2 pCPUs and I see that after the secondary cpu gets setup it
> goes into and idle loop and wait for a task. 
>
> 2. When primary cpu finishes init xen, it unpauses dom0 and therefore
> schedule it’s vCPU and call the wake function on the credit scheduler.
> I’m getting a hard time understanding what the wake do... does it put
> a tasklet in the percpu section and then the pCPU see it in its
> tasklet on the idle loop? If not what really happens? If yes, how what
> is the code flow that causes the dom0 code to be executed? Is it a
> context switch? If so where? Or is it just a function call?(I think
> it’s highly unlikely)

During Xen's boot, all APs start up and starts running the idle vCPU
(there is actually one idle vcpu for each CPU in the system).  This is
idle loop.

The very end of Xen's boot path unpauses dom0 (marks the scheduler
softirq pending), and runs the idle vCPU.  At this point, d0v0 is the
only non-idle and runnable vcpu in the system.

As a softirq is pending, the idle loop processes that first before going
to sleep.  This runs the schedule() function which finds d0v0 ready to
run, and context switches to it.

On x86, we have per-CPU stacks, not per-vCPU stacks, so context switch
involves playing with state at the base of the current stack, rather
than changing to a different stack.  After all of this is done, the end
of context_switch() invokes  d->arch.ctxt_switch->tail() which, for PV
guests, which resets the stack pointer to the base, and executes
ret_from_intr().  This is now in assembly code, and eventually IRET's to
dom0's entrypoint.

~Andrew