[Xen-devel] Xen/arm: Virtual ITS command queue handling

[Vijay Kilari <vijay.kilari@xxxxxxxxx>]

Hi,

   As discussed, here is the design doc/txt.

ARM GICv3 provides ITS (Interrupt Translation Service) feature to handle
MSIx interrupts.  Below are various mechanisms to handle
ITS commands.

ITS command completion detection mechanism:
----------------------------------------------------------------------
1) Append INT command to receive interrupt from ITS hardware after completion
of ITS command
2) Poll ITS command Queue by reading CREADER register

ITS driver running the guest can follow either one or both of the approaches
to know command completion.

Assumptions:
--------------------
1) Each VM will have one Virtual ITS (vITS)
2) VM is trapped on CWRITER write.
3) ITS commands should be processed in order of occurance.
   Though we release vITS lock before we put in physical ITS queue,
   there will not be any other VCPU that can trap and post another
   ITS command because the current VCPU is trapped on CWRITER.
   so another VCPU of the same domain cannot trap on CWRITER update.
   If this assumption is not valid, then vITS lock should be held
   untill command is posted to physical ITS.

Below are the proposed methods to emulate ITS commands in Xen.

Proposal 1:
----------------
Here when guest writes command to vITS queue and updates CWRITER registers,
it is trapped in XEN and below steps are followed to process ITS command

1) Trap of CWRITER write by guest
2) Take vITS lock
3) Read command written by guest, translate it.
   command queue.
4) Release vITS lock
5) Take physical ITS (pITS) lock
6) write CMD to physical ITS
7) Release pITS lock
8) Poll physical CREADER for completion of command.
9) Update vITS CREADER of the guest
10)If next command is available goto step (2)
   else
   return from trap

Cons:
   - VCPU loops in Xen untill all commands written by Guest are completed.
   - All the ITS commands written by guest is translated at processed before
     VCPU returns from trap.
   - If guest floods with ITS commands, VCPU keeps posting commands continously.

Pros:
   - Only one set of ITS commands sent by one VCPU per domain is
processed at a time

Handling Command queue state:
 - vITS Queue cannot be full as VCPU returns only on completion of ITS command.
 - Physical Queue cannot be full as it 64KB there by it can accomodate
1K ITS commands.
   If physical Queue is full, then VCPU will poll looking for empty physical.
   On timeout return error.

Behaviour of Polling and completion interrupt based guest driver:
 - If completion interrupt (INT) is used by guest driver, then guest driver will
   always see updated CREADER as commands are completed as it is
written to Queue.
 - If polling mode is used, trap on CREADER checks for completion of command.

Proposal 2:
----------------
Here when guest writes command to vITS queue and updates CWRITER registers,
it is trapped in XEN and below steps are followed to process ITS command

- Dom0 creates a ITS completion device with device id (00:00.1) and reserves
  n number (256 or so) irqs (LPIs) for this device.
- One irq/LPI (called completion_irq) of this completion device is
allocated per domain
- With this irq/LPI descriptor we can identify the domain/vITS.
- Info of all the ongoing ITS requests(put in pITS Queue) of this domain is
  stored in ITS command status array (called its_requests). This is
managed per vITS.

1) Trap of CWRITER write by guest
2) Take vITS lock
3) Read all the commands written by guest, translate it
    - If one of the guest command is INT command
       a) Append INT command with completion_irq and write this batch as
          seperate request and goto (3) to process next commands
    - If more than 'n' commands are sent by guest, start a timer to process
      remaining commands
4) Append INT command with completion_irq of current domain
5) Release vITS lock
6) Take physical ITS (pITS) lock
7) Write translated cmds to physical ITS
8) Add entry in its_requests
9) Release pITS lock
10) return from trap

One receiving completion interrupt:

1) Take the first pending request from its_requests.
2) Update vITS CREADER of the guest indicating completion of command to guest

Cons:
   - Has overhead of processing completion interrupt.
   - Need to reserve a fake device to generate completion interrupt and
     reserve one LPI per-domain

Pros:
   - VCPU does not poll in Xen for completion of commands.
   - Handles guest flooding command queue with commands. But needs timer

Handling Command queue state:
 - Physical Queue cannot be full as it 64KB there by it can accomodate
1K ITS commands.
   In case it is full, VCPU has to poll with timeout till physical
Queue is empty before it post
   next command
 - If vITS Queue condition should be managed by guest ITS driver.

Behaviour of Polling and completion interrupt based guest driver:
 - If completion interrupt (INT) is used by guest driver, then insert
Xen completion
   INT command so that CREADER is updated before guest's INT command is injected
 - If polling mode is used, trap on CREADER checks for completion of command


Proposal 3: (From Julien)
------------------------------------
Here when guest writes command to vITS queue and updates CWRITER registers,
it is trapped in XEN and below steps are followed to process ITS command

1) Trap of CWRITER write by guest
2) Check if vITS state is IDLE. If not return from trap
3) Take vITS lock
4) Read 'n' commands at a time written by guest, translate it
5) Set vITS state as IN_PROGRESS
5) Release vITS lock
6) Take physical ITS (pITS) lock
7) Write translated cmds to physical ITS
9) Release pITS lock
10) return from trap

On CREADER trap from guest (which polls for completion)
11) Check if posted command is completed.
   if completed,
     - update CREADER of vITS
     - Set vITS state as IDLE
     - Post next 'n' set of commands (Jump to 1)
   else
     return from trap.

Cons:
   - Guest ITS driver should always poll on CREADER to know completion of
     commands. If guest does not poll, vITS CREADER will not be updated.

Pros:
   - VCPU will not poll in Xen
   - Handles guest flooding with ITS commands scenario by processing 'n'
     commands at a time.

Handling Command queue state:
 - Physical Queue cannot be full as it 64KB there by it can accomodate
1K ITS commands.
   In case it is full, VCPU has to poll with timeout till physical
Queue is empty
   before it posts the commands
 - If vITS Queue condition should be managed by guest ITS driver.

Behaviour of Polling and completion interrupt based guest driver:
 - If completion interrupt (INT) is used by guest driver, then guest driver
   should read CREADER, so that it is trapped and gets updated CREADER.
   If CREADER is not polled, CREADER will not be updated.
 - If polling mode is used, trap on CREADER checks for completion of command


Regards
Vijay

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel