[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [PATCH v10 13/17] vpci: add initial support for virtual PCI bus topology
On Fri, 17 Nov 2023, Volodymyr Babchuk wrote: > > On Fri, 17 Nov 2023, Volodymyr Babchuk wrote: > >> Hi Julien, > >> > >> Julien Grall <julien@xxxxxxx> writes: > >> > >> > Hi Volodymyr, > >> > > >> > On 17/11/2023 14:09, Volodymyr Babchuk wrote: > >> >> Hi Stefano, > >> >> Stefano Stabellini <sstabellini@xxxxxxxxxx> writes: > >> >> > >> >>> On Fri, 17 Nov 2023, Volodymyr Babchuk wrote: > >> >>>>> I still think, no matter the BDF allocation scheme, that we should > >> >>>>> try > >> >>>>> to avoid as much as possible to have two different PCI Root Complex > >> >>>>> emulators. Ideally we would have only one PCI Root Complex emulated > >> >>>>> by > >> >>>>> Xen. Having 2 PCI Root Complexes both of them emulated by Xen would > >> >>>>> be > >> >>>>> tolerable but not ideal. > >> >>>> > >> >>>> But what is exactly wrong with this setup? > >> >>> > >> >>> [...] > >> >>> > >> >>>>> The worst case I would like to avoid is to have > >> >>>>> two PCI Root Complexes, one emulated by Xen and one emulated by QEMU. > >> >>>> > >> >>>> This is how our setup works right now. > >> >>> > >> >>> If we have: > >> >>> - a single PCI Root Complex emulated in Xen > >> >>> - Xen is safety certified > >> >>> - individual Virtio devices emulated by QEMU with grants for memory > >> >>> > >> >>> We can go very far in terms of being able to use Virtio in safety > >> >>> use-cases. We might even be able to use Virtio (frontends) in a SafeOS. > >> >>> > >> >>> On the other hand if we put an additional Root Complex in QEMU: > >> >>> - we pay a price in terms of complexity of the codebase > >> >>> - we pay a price in terms of resource utilization > >> >>> - we have one additional problem in terms of using this setup with a > >> >>> SafeOS (one more device emulated by a non-safe component) > >> >>> > >> >>> Having 2 PCI Root Complexes both emulated in Xen is a middle ground > >> >>> solution because: > >> >>> - we still pay a price in terms of resource utilization > >> >>> - the code complexity goes up a bit but hopefully not by much > >> >>> - there is no impact on safety compared to the ideal scenario > >> >>> > >> >>> This is why I wrote that it is tolerable. > >> >> Ah, I see now. Yes, I am agree with this. Also I want to add some > >> >> more > >> >> points: > >> >> - There is ongoing work on implementing virtio backends as a > >> >> separate > >> >> applications, written in Rust. Linaro are doing this part. Right now > >> >> they are implementing only virtio-mmio, but if they want to provide > >> >> virtio-pci as well, they will need a mechanism to plug only > >> >> virtio-pci, without Root Complex. This is argument for using single > >> >> Root > >> >> Complex emulated in Xen. > >> >> - As far as I know (actually, Oleksandr told this to me), QEMU has > >> >> no > >> >> mechanism for exposing virtio-pci backends without exposing PCI root > >> >> complex as well. Architecturally, there should be a PCI bus to which > >> >> virtio-pci devices are connected. Or we need to make some changes to > >> >> QEMU internals to be able to create virtio-pci backends that are not > >> >> connected to any bus. Also, added benefit that PCI Root Complex > >> >> emulator in QEMU handles legacy PCI interrupts for us. This is > >> >> argument for separate Root Complex for QEMU. > >> >> As right now we have only virtio-pci backends provided by QEMU and > >> >> this > >> >> setup is already working, I propose to stick to this > >> >> solution. Especially, taking into account that it does not require any > >> >> changes to hypervisor code. > >> > > >> > I am not against two hostbridge as a temporary solution as long as > >> > this is not a one way door decision. I am not concerned about the > >> > hypervisor itself, I am more concerned about the interface exposed by > >> > the toolstack and QEMU. > > > > I agree with this... > > > > > >> > To clarify, I don't particular want to have to maintain the two > >> > hostbridges solution once we can use a single hostbridge. So we need > >> > to be able to get rid of it without impacting the interface too much. > > > > ...and this > > > > > >> This depends on virtio-pci backends availability. AFAIK, now only one > >> option is to use QEMU and QEMU provides own host bridge. So if we want > >> get rid of the second host bridge we need either another virtio-pci > >> backend or we need to alter QEMU code so it can live without host > >> bridge. > >> > >> As for interfaces, it appears that QEMU case does not require any changes > >> into hypervisor itself, it just boils down to writing couple of xenstore > >> entries and spawning QEMU with correct command line arguments. > > > > One thing that Stewart wrote in his reply that is important: it doesn't > > matter if QEMU thinks it is emulating a PCI Root Complex because that's > > required from QEMU's point of view to emulate an individual PCI device. > > > > If we can arrange it so the QEMU PCI Root Complex is not registered > > against Xen as part of the ioreq interface, then QEMU's emulated PCI > > Root Complex is going to be left unused. I think that would be great > > because we still have a clean QEMU-Xen-tools interface and the only > > downside is some extra unused emulation in QEMU. It would be a > > fantastic starting point. > > I believe, that in this case we need to set manual ioreq handlers, like > what was done in patch "xen/arm: Intercept vPCI config accesses and > forward them to emulator", because we need to route ECAM accesses > either to a virtio-pci backend or to a real PCI device. Also we need > to tell QEMU to not install own ioreq handles for ECAM space. I was imagining that the interface would look like this: QEMU registers a PCI BDF and Xen automatically starts forwarding to QEMU ECAM reads/writes requests for the PCI config space of that BDF only. It would not be the entire ECAM space but only individual PCI conf reads/writes that the BDF only. > Another point is PCI legacy interrupts, which should be emulated on Xen > side. And unless I miss something, we will need some new mechanism to > signal those interrupts from QEMU/other backend. I am not sure if we can > use already existing IRQ signaling mechanism, because PCI interrupts are > ORed for all devices on a bridge and are level-sensitive. I hope we can reuse xc_hvm_set_pci_intx_level or another XEN_DMOP hypercall > Of course, we will need all of this anyways, if we want to support > standalone virtio-pci backends, but for me it sounds more like "finish > point" :)
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |