Re: [PATCH v1] xen: fusa: arch_specs: Definition of the Virtual Machines' roles

[Ayan Kumar Halder <ayankuma@xxxxxxx>]

On 23/06/2025 10:34, Yann Sionneau wrote:
> CAUTION: This message has originated from an External Source. Please use proper 
> judgment and caution when opening attachments, clicking links, or responding to 
> this email.
>
>
> Hi Ayan, Stefano,
Hi Yann,
> This doc looks nice overall!
>
> However I do think that maybe the role part should be separate from all
> the explanations about virtio.
>
> I would say virtio deserves its own page.
>
> Also, I'm curious what "FFI" is.
"FFI" is freedom from interference. This term is taken from ISO-26262-6 
Annex D.  As per understanding, FFI for Xen means whether Xen is able to 
prevent or detect and mitigate interference to one or more VM from other 
VMs. The VM/s which cen be protected from interference, can be 
considered for running safety critical tasks.
Now, there are various operations which are used for communication 
between VMs. For eg grants, event channels, shared memory. Also, there 
are operations which have a system wide impact. For eg suspend, power 
off, clock control. Each of them is a potential source of interference 
between VMs. What we need to determine is whether Xen can prevent or 
detect such interferences for any (or a set of) VM/s in a given 
scenario/s using VM roles and other properties. The scenario/s will be 
validated for safety certification (under testing for FFI).
You can find some relevant discussion on "hardware domain and control 
domain separation".
@Bertrand , do you want to add something or correct me ?

- Ayan

> Thanks!
>
> Regards,
>
> Yann
>
> On 3/4/25 19:31, Ayan Kumar Halder wrote:
> > From: Stefano Stabellini <stefano.stabellini@xxxxxxx>
> >
> > Define "Hardware Domain", "Control Domain", "DomUs", "SafeVMs" and
> > "UnsafeVMs".
> >
> > Explain the limitations that VirtIO introduces and the way they affect
> > the defined roles.
> >
> > Signed-off-by: Stefano Stabellini <stefano.stabellini@xxxxxxx>
> > Signed-off-by: Ayan Kumar Halder <ayan.kumar.halder@xxxxxxx>
> > ---
> >    docs/fusa/arch_specs/roles.rst | 200 +++++++++++++++++++++++++++++++++
> >    docs/fusa/index.rst            |   1 +
> >    2 files changed, 201 insertions(+)
> >    create mode 100644 docs/fusa/arch_specs/roles.rst
> >
> > diff --git a/docs/fusa/arch_specs/roles.rst b/docs/fusa/arch_specs/roles.rst
> > new file mode 100644
> > index 0000000000..0b3fe821c3
> > --- /dev/null
> > +++ b/docs/fusa/arch_specs/roles.rst
> > @@ -0,0 +1,200 @@
> > +===============================
> > +Xen Domain Roles and Privileges
> > +===============================
> > +
> > +Roles
> > +=====
> > +
> > +We define the following roles that Xen domains can have:
> > +- Hardware Domain
> > +- Control Domain
> > +- DomU
> > +
> > +In addition, we classify domains based on the criticality of the tasks
> > +they perform:
> > +- SafeVM
> > +- UnSafeVM
> > +
> > +DomU
> > +----
> > +
> > +A DomU is a regular domain without extra privileges to perform
> > +operations that affect other domains or the entire platform. A DomU is
> > +therefore unprivileged and can only affect itself.
> > +
> > +Hardware Domain
> > +---------------
> > +
> > +The Hardware Domain is a domain that has direct access to all hardware
> > +peripherals by default, except for those controlled by Xen (CPU, MMU,
> > +IOMMU, Interrupt Controller, Timers, UART). Hardware resources can be
> > +directly assigned to other domains ("passthrough"), but the default is
> > +that they get assigned to the Hardware Domain. As such, there can only
> > +be one Hardware Domain in the system. The Hardware Domain is not
> > +privileged: it cannot perform operations that affect other domains or
> > +the entire platform.
> > +
> > +In cases where VirtIO is used to share access to physical devices among
> > +multiple domains, the Hardware Domain requires some limited privileges
> > +to run the VirtIO backends.
> > +
> > +Control Domain
> > +--------------
> > +
> > +The Control Domain has full privileges over the system. As such, it can
> > +perform privileged operations, typically implemented as hypercalls, that
> > +affect other domains and even the entire platform, such as platform
> > +reboot. There can only be one Control Domain in the system. A
> > +non-exhaustive list of these operations follows:
> > +
> > +- Reboot other domains
> > +- Reboot and shutdown the platform
> > +- Suspend/resume other domains
> > +- Pause/unpause other domains
> > +
> > +The Control Domain is optional.
> > +
> > +SafeVMs and UnSafeVMs
> > +---------------------
> > +
> > +A SafeVM is a domain running a safety-critical task. An UnsafeVM is a
> > +domain running non-safety-critical software only. No safety-critical
> > +tasks are running in the UnsafeVM.
> > +
> > +The Control Domain must be a SafeVM because the Control Domain has
> > +enough privileges to affect the entire system.
> > +
> > +Without VirtIO (see below), both the Hardware Domain and DomUs can be
> > +SafeVMs or UnsafeVMs. There are no restrictions. For example, the
> > +following configuration is compatible with safety:
> > +
> > +- Hardware Domain: UnsafeVM
> > +- 2x DomU: SafeVMs
> > +- 2x DomU: UnsafeVMs
> > +
> > +Privileges
> > +==========
> > +
> > +Full Privileges
> > +---------------
> > +
> > +A domain with full privileges has unrestricted access to all services
> > +provided by Xen, including the ability to perform operations that impact
> > +other VMs and the entire platform. Only one domain holds these full
> > +privileges: the Control Domain.
> > +
> > +Unprivileged
> > +------------
> > +
> > +An Unprivileged domain lacks special access rights, allowing it to
> > +perform only operations that affect itself without impacting other
> > +domains. For example, an Unprivileged domain cannot map the memory of
> > +another domain.
> > +
> > +VirtIO Privileges
> > +-----------------
> > +
> > +Domains running VirtIO backends, such as QEMU, require specific
> > +privileges over domains with VirtIO frontends to enable proper
> > +functionality. For instance, they need permission to map the memory of
> > +the domain with the VirtIO frontend. These additional privileges are
> > +necessary for standard VirtIO operation but can be removed when using
> > +the VirtIO extensions described in the VirtIO chapter. This is the list
> > +of VirtIO privileges:
> > +- mapping memory of VirtIO frontend domains (foreign map hypercall)
> > +- device model hypercalls targeting VirtIO frontend domains
> > +
> > +VirtIO
> > +======
> > +
> > +VirtIO is a specification and set of drivers to share a physical device
> > +among multiple domains. For example, assuming the Hardware Domain has
> > +access to an Ethernet device, VirtIO provides a way for the Hardware
> > +Domain to allow one or more DomUs to access the Ethernet device.
> > +
> > +VirtIO is based on a frontend/backend architecture: the frontend is a
> > +driver similar to a regular Ethernet device driver (using Ethernet as an
> > +example), running in the DomU. The backend is a "proxy" driver running
> > +in the same domain as the device driver and have direct access to the
> > +device. That domain is typically the Hardware Domain because it is the
> > +default domain for hardware assignment, but it is also possible to run
> > +VirtIO backends in other domains, provided they have been configured with
> > +passthrough access to a physical device. These other domains are called
> > +"Driver Domains," and the limitations discussed in the context of VirtIO
> > +for the Hardware Domain also apply to Driver Domains.
> > +
> > +There are multiple issues in VirtIO that affect freedom from
> > +interference, so deploying VirtIO introduces limits on the configuration
> > +regarding what domains can run safety-critical workloads.
> > +
> > +It is important to note that VirtIO is not the only way for domains to
> > +communicate with each other. Specifically, there are other protocols
> > +such as Argo and plain shared memory that do not have the same
> > +interference concerns.
> > +
> > +FFI: Shared Memory and Interrupts
> > +---------------------------------
> > +
> > +All VirtIO protocols are based on ring buffers over shared memory,
> > +called "VirtQueues." Although there are well-known methods to set up
> > +ring buffers over shared memory that are free from interference, in
> > +practice, the existing VirtIO specification and current frontend/backend
> > +driver implementations do not follow these best practices. Thus, the
> > +ring buffers over shared memory can be a source of interference.
> > +
> > +Given the current state of the VirtIO specification and drivers, it is
> > +recommended that both VirtIO frontends and backends run in UnsafeVMs
> > +only. This way, SafeVMs will not be affected by interference caused by
> > +VirtIO protocols.
> > +
> > +As ring buffers exist over memory shared between domains, it is the
> > +domains' responsibility to ensure they are safe. This is outside of
> > +Xen's responsibility, and out of scope for this document.
> > +
> > +FFI: Memory Mappings
> > +--------------------
> > +
> > +VirtIO expects the backend to access ("map") arbitrary memory addresses
> > +passed by the frontend. In other words, the backend needs to map any
> > +memory of the frontend, which is a privileged operation in Xen.
> > +
> > +Xen implements a Xen-specific extension to VirtIO to address this issue:
> > +"VirtIO with Grants."
> > +
> > +Xen has a feature called the "grant table" that guests can use to
> > +voluntarily share individual memory pages with other domains. This way,
> > +the backend does not need the privilege to map any memory page of the
> > +domain running the frontend, as the frontend whitelists pages for the
> > +backend to access.
> > +
> > +Unless "VirtIO with Grants" is used, to enable VirtIO, a domain with
> > +VirtIO backends (the Hardware Domain or a Driver Domain) is given the
> > +extra privilege of being able to map any memory of domains with VirtIO
> > +frontends.  As this is a source of interference, only UnsafeVMs can run
> > +VirtIO frontends without "VirtIO with Grants".
> > +
> > +FFI: Synchronous Emulation
> > +--------------------------
> > +
> > +VirtIO expects certain operations to be synchronous: the frontend
> > +virtual CPU requesting the operation is fully blocked until the backend
> > +completes the operation. If the backend does nothing, the frontend vCPU
> > +remains blocked. This is a source of interference.
> > +
> > +When VirtIO is enabled, the domain with the backends (Hardware Domain or
> > +Driver Domain) implements these synchronous operations for domains
> > +running VirtIO frontends.
> > +
> > +Xen implements a Xen-specific extensions to VirtIO to address this
> > +issue: "non-blocking VirtIO." Two extra virtio-pci registers are
> > +provided, allowing the frontend to perform non-blocking operations. By
> > +using the two Xen-specific non-blocking registers, the frontend virtual
> > +CPU doesn't have to block while waiting for the backend to complete the
> > +operations.
> > +
> > +Unless "non-blocking VirtIO" is used, to enable VirtIO, a domain with
> > +VirtIO backends (the Hardware Domain or a Driver Domain) is given the
> > +extra privilege of being able to implement these synchronous operations
> > +for domains running VirtIO frontends. As this is a source of
> > +interference, only UnsafeVMs can run VirtIO frontends without
> > +"non-blocking VirtIO."
> > diff --git a/docs/fusa/index.rst b/docs/fusa/index.rst
> > index 5f1e8acfc4..9d989f5105 100644
> > --- a/docs/fusa/index.rst
> > +++ b/docs/fusa/index.rst
> > @@ -7,3 +7,4 @@ Functional Safety documentation
> >       :maxdepth: 2
> >
> >       reqs/index
> > +   arch_specs/roles
> Yann Sionneau | Vates XCP-ng Developer
>
> XCP-ng & Xen Orchestra - Vates solutions
>
> web: https://vates.tech