Re: [Xen-devel] [RFC PATCH v2 00/17] Add support for qemu-xen runnning in a Linux-based stubdomain.

[Ian Jackson <ian.jackson@xxxxxxxxxx>]

Marek Marczykowski-Górecki writes ("[RFC PATCH v2 00/17] Add support for 
qemu-xen runnning in a Linux-based stubdomain."):
> [stuff]

Thanks for this.  I'm very keen on this approach and keen on getting
it upstream.  Sorry for the delay reviewing it!  I have been away a
lot.

> General idea is to allow freely set device_model_version and
> device_model_stubdomain_override and choose the right options based on this 
> choice.
> Also, allow to specific path to stubdomain kernel/ramdisk, for greater 
> flexibility.
> Right now when qemu-xen in stubdomain is selected, it is assumed it's
> Linux-based and few decisions are based on it, specifically:

That sounds sensible.

>  - qemu args encoding (\x1b as separator, to allow spaces in arguments)

What if the arguments contain \x1b ?  I think you may need a quoting
scheme, or to use nul.

> It may be a good idea to document "stubdomain API" somewhere. Is there such
> document for MiniOS one?

No.

>  Here is an initial version for Linux one:
> 
>     Assumptions about Linux-based stubdomain for qemu-xen:
>     * qemu command line is stored by toolstack in
>       /vm/<target-uuid>/image/dmargs xenstore entry, arguments are separated
>       with \x1b character

Oh, xenstore.  From docs/misc/xenstore.txt:

| xenstore values should normally be 7-bit ASCII text strings
| containing bytes 0x20..0x7f only

I think you could have separate xenstore keys for the seperate
arguments, or you could encode it somehow.

>     * qemu can access saved state (if any) at its FD 3
>     * qemu can write its state (for save/migration) to its FD 4

That's a description of the protocol to *qemu*.  Is the toolstack then
responsible for the protocol across the domain boundary ?  I think it
would be nice to specify that here too.

>     * qemu expects backend for serial console at /dev/hvc3
>     * disks configured for the target are available as /dev/xvd*, in
>       configuration order
>     * qemu can call /etc/qemu-ifup and /etc/qemu-ifdown to connect/disconnect
>       network interface to appropriate network

Please can we put this in-tree and not in a 00/ cover letter :-).

> Initial version has no QMP support - in initial patches it is completely
> disabled, which means no suspend/restore. QMP normally would be used for PCI
> passthrough setup, but it is worked around with MiniOS-like control protocol
> over xenstore, which then is translated to QMP on stubdomain side.

How unpleasant.  But better than nothing.

> Some option is to use separate console for that, but that require
> xenstoled supporting multiple consoles per domain (the goal is to not have 
> qemu
> in dom0 at all). Also, it would be preferable to evaluate how libxl
> handle potentially malicious QMP responses.

We are working on that latter point anyway.

> Another option is to use xenstore - either translate everything needed to
> MiniOS-like thing, or write already json-formatted requests to xenstore and
> watch there for responses. When using separate xenstore dir for that, with
> per-command entries (command id as a key name?) that would solve concurrency
> problem.

That would be fine too.

> QMP support over separate console: patch "libxl: access QMP socket via console
> for qemu-in-stubdomain" implements some early PoC of this.
> Major limitation: only one connection at a time and no means of out of
> band reset (and re-negotiate). I've tried adding very simple `qmp_reset`
> command for in-band connection reset, but it is unreliable because of the
> first limitation - xl process running in background hold this connection open
> and every other xl instance interfere with it. On the other hand, for libvirt
> use case one connection could be enough (leaving alone libvirtd restart).

This is awkward, isn't it.  The Xen PV console protocol has no reset
mechanism.  Can we use libvchan here and provide qmp vchans ?

> Xenconsoled patches add support for multiple consoles in xenconsoled, to 
> avoid the need
> for qemu in dom0 for this to work. Multiple consoles for a stubdomain are 
> used for:
>  - logs (console 0)
>  - save + restore (console 1, 2)
>  - qmp (console 3)
>  - serial terminal to target domU (console 4)
> Xenconsoled patches are in fact a separate series, but I'm sending them here 
> to
> ease dependencies handling (latter libxl patches use that).

That sounds reasonable.

> What qmp-libxenstat socket is for?
> 
> PCI passthrough support require some more love. Right now, libxl tries to 
> setup
> pcifront for both target HVM and stubdomain and the former fails (race
> condition):
>     xen-pciback pci-259-0: 22 Couldn't locate PCI device (0000:00:1b.0)! 
> perhaps already in-use?
> Fortunately it isn't needed. There is also a PCI related problem on
> domain shutdown - it looks like first stubdomain is paused and then libxl 
> waits
> for pcifront there.
> Also, MSI doesn't work (qemu output):
> 
>     [00:05.0] xen_pt_msgctrl_reg_write: setup MSI (register: 81).
>     [00:05.0] msi_msix_setup: requested pirq 55 for MSI (vec: 0, entry: 0)
>     [00:05.0] msi_msix_setup: Error: Mapping of MSI (err: 1, vec: 0, entry 0)
>     [00:05.0] xen_pt_msgctrl_reg_write: Warning: Can not map MSI (register: 
> 80)!
>     [00:05.0] Write-back to unknown field 0x44 (partially) inhibited (0x00)
>     [00:05.0] If the device doesn't work, try enabling permissive mode
>     [00:05.0] (unsafe) and if it helps report the problem to xen-devel

I confess I don't understand PCI passthrough but it would be nice for
this to work.  I think a starting point might be to write down who is
supposed to do what...

> This patch series is third (fourth?) attempt to get rid of limitation
> "if you want to use stubdomain, you're stuck with qemu-traditional", done over
> years, by many people.
> I think it should be acceptable plan to gradually add features to
> qemu-xen+stubdomain configuration - not necessary waiting with committing any
> of those patches until full feature set of qemu-xen is supported. After all,
> right now "feature set supported by qemu-xen+stubdom" is empty, so wouldn't be
> worse.

Absolutely.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel