[Xen-devel] [RFC] Pass-through Interdomain Interrupts Sharing (HVM/Dom0)

["Guy Zana" <guy@xxxxxxxxxxxx>]

We propose the following method in order to support interdomain
interrupt sharing, where one of the domains is an HVM assigned with a
pass-through device. This method is limited in a way that we can support
sharing between just two domains: dom0 and an HVM. This method is based
on changing polarity.

Terminology
=========
Change polarity algorithm (CPA) - Algorithm when polarity inversion is
used for the EOI recognition. For details see
http://lists.xensource.com/archives/html/xen-devel/2007-05/msg01148.html
.
PLINE - Physical Line. This is the reflection of the physical line. By
changing polarity we know what is the physical line's status.
VLINE - Virtual Line. This is the HVM virtual line.
PT Device - A pass-through PCI device assigned to the HVM.
Dom0 Device - A PCI device assigned to dom0 (by default).
Interrupt Sharing - Determined by two or more PCI devices, which their's
intx lines are connected to the same IOAPIC's pin (OR wired), and
assigned to different domains.
Re-occurring interrupts - The pline is held asserted while the IOAPIC
fire interrupts continuously.
Spurious interrupts - Whitin a domain context, an interrupt that passed
the ISR chain without handling.

NOTE: A single PCI device can not be assigned to more than one domain
simultaneously.

When a single device is assigned to an HVM, using CPA, we update the
HVM's VLINE according to the PLINE state (both hold the same value) thus
providing complete reflection. It is trivial to see how more than one
device that shares the same line could be assigned to the HVM (using the
same CPA).

In general, we should consider the situation were N devices from Dom0
shares the same line with M devices from HVM. There are 3 cases
possible:

1. N=0, i.e. this line belongs to HVM devices. This case is already
solved with CPA.
2. M=0, i.e. this line belongs to Dom0 devices. This is basic dom0
functionality.
3. N != 0, M != 0. This is the situation that we want to handle now,
from now on we'll refer to this situation as interdomain shared
interrupt.

Although, our method could be extended to contain handling for all of
the above cases.

Problems related to Interdomain Interrupt Sharing
====================================
* Spurious interrupts.
* Interrupt starvation.
* When we use CPA, we are not getting re-occurring interrupts, this
should be taken into account.
* Even if a shared interrupt was handled by a domain specific ISR, it is
not guaranteed that the pline will be deasserted.
* Interrupt storming - _Physical_ storming is solved transparently by
CPA.

Goals
=====
* Letting both the HVM and DOM0 a chance to handle the interrupt
* Update the HVM's VLINE correctly when sharing an interrupt
* Avoid spurious interrupts or at least minimize the number of such
interrupts injected into HVM.
* Stay with a reasonable interrupt latency.

Proffered Method
=============
1. We gain shared line assertion state by using CPA, at an
assert/deassert event we save the line's state.
2. We perform most of the logic in a periodic timer module.

Modules
======
1. Timer module. Periodic callback that does all the logic processing.
2. XEN interrupt handler. Handler is replaced by CPA that updates PLINE.
3. Dom0 ISR chain. At the end of the chain, we know whether the
interrupt was handled or not, and update the status in Xen using a
hypercall.

States
=====
1. Idle. The PLINE is deasserted. This is "relax state". We're awaiting
the interrupt to come.
2. In Dom0. The interrupt is currently handled by Dom0. The event was
sent into Dom0 and Dom0 ISR is processing it.
3. Process Interrupt. The interrupt was handled by Dom0. Dom0 got back
to us with the results of the handling. Now we need to decide what to do
next. This state can be reached only from state [2]. 

State machine
===========
The timer callback implement the state machine, it freezes when we are
in the idle state.
The "events" described below are polled by the timer. We also perform
changes in dom0's ISR chain in order to generate these "events".

The following events are handled:

A. PLINE is deasserted. This event will move state machine to _Idle_
state from any state.
This can happen in one of 2 cases: 
1. Initialization. 
2. As a result of PLINE deassertion. If PLINE went down, it means that
we're done. 

B. Idle state and PLINE is asserted. In this case the interrupt is
injected into DOM0. The state machine moves to "In Dom0". We always
firstly let domain0 try to handle the interrupt, thus logically creating
an interdomain ISR chain beginning with dom0.

C. "In Dom0" and PLINE status is asserted (We read the status from a
timer). Do nothing. We don't know what to do with this interrupt yet.

D. "Process Interrupt" and PLINE is asserted.
Few cases are possible:
1. If Dom0 successfully handled the last interrupt and the interrupt
wasn't injected into the HVM, inject the interrupt into Dom0 and move to
state "In Dom0". This is the Dom0 interrupt, keep injecting into Dom0.
2. If Dom0 successfully handled the last interrupt and the interrupt was
injected into the HVM, deassert the HVM vline, and re-inject the
interrupt into Dom0. Move to state "In Dom0".
(This is done in order to solve a case where the HVM was handling the
interrupt, but the line didn't get deasserted because a Dom0 device
asserted it before the a PT device deasserted it (as result of the HVM
handling). In this case we assume that the HVM is done with it and now
it's Dom0's turn.) 
3. If Dom0 didn't successfully handle the last interrupt and the
interrupt was not injected into the HVM, inject the interrupt into the
HVM and stay in the same state. This is an HVM's interrupt. Dom0
rejected it.
4. If Dom0 didn't successfully handle the last interrupt and the
interrupt was injected into HVM, inject interrupt into Dom0 and move to
state "in Dom0". HVM is not done yet with current interrupt.

E."Process Interrupt" and the PLINE is deasserted,- deassert the HVM
interrupt(if neccesary) and move to idle. We handled the interrupt.
Prepare ourselves for the new one.

The main idea here is to inject the interrupt into Dom0 when we don't
know what to do with it. If Dom0 takes the ownership, then let it handle
the interrupt. If not, we inject it into the HVM. We recognize that all
of the PT devices are not asserting the line by PLINE deassertion or by
Dom0 taking the ownership back to it. 

Any ideas and comments are welcome.
 
Best regards,
Alex Novik,
Neocleus.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel