Re: Event delivery and "domain blocking" on PVHv2

[Martin Lucina <martin@xxxxxxxxxx>]

On 2020-06-22 15:58, Roger Pau Monné wrote:
> On Mon, Jun 22, 2020 at 12:58:37PM +0200, Martin Lucina wrote:
> > Aha! Thank you for pointing this out. I think you may be right, but 
> > this
> > should be possible without doing the demuxing in interrupt context.
> If you don't do the demuxing in the interrupt context (ie: making the
> interrupt handler a noop), then you don't likely need such interrupt
> anyway?
I need the/an interrupt to wake the VCPU from HLT state if we went to 
sleep.
> > How about this arrangement, which appears to work for me; no hangs I 
> > can see
> > so far and domU survives ping -f fine with no packet loss:
> >
> > CAMLprim value
> > mirage_xen_evtchn_block_domain(value v_deadline)
> > {
> >     struct vcpu_info *vi = VCPU0_INFO();
> >     solo5_time_t deadline = Int64_val(v_deadline);
> >
> >     if (solo5_clock_monotonic() < deadline) {
> >         __asm__ __volatile__ ("cli" : : : "memory");
> >         if (vi->evtchn_upcall_pending) {
> >             __asm__ __volatile__ ("sti");
> >         }
> >         else {
> >             hypercall_set_timer_op(deadline);
> What if you set a deadline so close that evtchn_upcall_pending gets
> set by Xen here and the interrupt is injected? You would execute the
> noop handler and just hlt, and could likely end up in the same blocked
> situation as before?
Why would an interrupt be injected here? Doesn't the immediately 
preceding
"cli" disable that?

Or perhaps I need to do a PV/HVM hybrid and set vi->evtchn_upcall_mask 
just
before the cli, and clear it after the sti?

> > i.e. Always go to sleep with interrupts disabled, but before doing so
> > re-check that no events have become pending since the last time
> > evtchn_demux_pending() was called. This holds, since the only thing 
> > that
> > sets vi->evtchn_upcall_pending is Xen, and the only thing that clears 
> > it is
> > evtchn_demux_pending().
> >
> > Right?
> TBH this is a hard model to get right, I think your best bet at
> attempting something along this lines is to forget about using the
> event channel interrupt and instead use SCHEDOP_poll. You could do
> something like (written in pure C as I have no idea of the ocaml
> bindings):
> [SCHEDOP_poll code snipped]
Thanks for the suggestion. This brings us full-circle -- I found [1] and
[2] way back from 2013 when Mirage/Xen was initially using SCHEDOP_poll
and then switched to the current interrupts + SCHEDOP_block approach.

Part of the motivation for the change at the time was to allow waiting
on/servicing more than 128 ports (the limit for SCHEDOP_poll). I doubt
anyone wants to do that these days, but it still makes me a bit 
reluctant
to change back to SCHEDOP_poll.

> > In an attempt to understand why the original PV code worked I re-read 
> > the PV
> > Mini-OS block_domain code again and realised that I had entirely 
> > missed one
> > part of its behaviour, which is that it intends[*] to run with
> > interrupts/upcalls disabled *all* the time and relies on SCHEDOP_block
> > atomically re-enabling them and triggering an upcall before returning 
> > (PV)
> > or "briefly enabling interrupts to allow handlers to run" (HVM). We're 
> > doing
> > the inverse, but our behaviour matches my mental model of how things 
> > should
> > work.
> Not really IMO, as SCHEDOP_block is a single 'instruction' from a
> guest PoV that does the enabling of interrupts and returns if there
> are pending ones.
Indeed and this is exactly the operation I need in the HVM world with 
the
current model.

> Also SCHEDOP_block is not exactly the same on HVM, as it just checks
> for pending vectors to inject, but not for pending event channels.
... well, I don't want to use SCHEDOP_block anyway since that is not 
possible
on ARM, and I would like to keep the code at least "portable with not 
too
much effort". So there should be a non-racy "HVM way" to do this?

> HVM you cannot call hlt with interrupts disabled, or the vCPU will be
> taken down.
>
> There are quite a lot of subtle differences between PV and HVM in this
> regard, and I think the best approach would be to use SCHEDOP_poll in
> order to implement the kind of model you describe.
I can see that now. The interactions between the "virtual hardware"
(interrupt delivery, VCPU IF) and "PV" parts (event delivery, masking) 
are
hard to understand for me, yet the two are intertwined in the way HVM
works :-(

Coming back to your earlier suggestion of moving the event demuxing (but 
not
the handling) into the upcall interrupt handler itself, perhaps that 
approach
is still worth investigating in combination with re-working the OCaml 
side array
view of pending events, or at least ensuring that atomic operations are 
used
since it would now be updated asynchronously.

Martin

[1] 
https://lists.cam.ac.uk/pipermail/cl-mirage/2013-September/msg00053.html
[2] https://github.com/mirage/mirage-platform/pull/58