Re: PVH Dom0 related UART failure

[Jan Beulich <jbeulich@xxxxxxxx>]

On 19.05.2023 10:55, Roger Pau Monné wrote:
> On Fri, May 19, 2023 at 10:20:24AM +0200, Jan Beulich wrote:
>> On 19.05.2023 09:38, Roger Pau Monné wrote:
>>> On Fri, May 19, 2023 at 09:22:58AM +0200, Jan Beulich wrote:
>>>> On 18.05.2023 12:34, Roger Pau Monné wrote:
>>>>> On Wed, May 17, 2023 at 05:59:31PM -0700, Stefano Stabellini wrote:
>>>>>> I have run into another PVH Dom0 issue. I am trying to enable a PVH Dom0
>>>>>> test with the brand new gitlab-ci runner offered by Qubes. It is an AMD
>>>>>> Zen3 system and we already have a few successful tests with it, see
>>>>>> automation/gitlab-ci/test.yaml.
>>>>>>
>>>>>> We managed to narrow down the issue to a console problem. We are
>>>>>> currently using console=com1 com1=115200,8n1,pci,msi as Xen command line
>>>>>> options, it works with PV Dom0 and it is using a PCI UART card.
>>>>>>
>>>>>> In the case of Dom0 PVH:
>>>>>> - it works without console=com1
>>>>>> - it works with console=com1 and with the patch appended below
>>>>>> - it doesn't work otherwise and crashes with this error:
>>>>>> https://matrix-client.matrix.org/_matrix/media/r0/download/invisiblethingslab.com/uzcmldIqHptFZuxqsJtviLZK
>>>>>
>>>>> Jan also noticed this, and we have a ticket for it in gitlab:
>>>>>
>>>>> https://gitlab.com/xen-project/xen/-/issues/85
>>>>>
>>>>>> What is the right way to fix it?
>>>>>
>>>>> I think the right fix is to simply avoid hidden devices from being
>>>>> handled by vPCI, in any case such devices won't work propewrly with
>>>>> vPCI because they are in use by Xen, and so any cached information by
>>>>> vPCI is likely to become stable as Xen can modify the device without
>>>>> vPCI noticing.
>>>>>
>>>>> I think the chunk below should help.  It's not clear to me however how
>>>>> hidden devices should be handled, is the intention to completely hide
>>>>> such devices from dom0?
>>>>
>>>> No, Dom0 should still be able to see them in a (mostly) r/o fashion.
>>>> Hence my earlier RFC patch making vPCI actually deal with them.
>>>
>>> What's the difference between a hidden device and one that's marked RO
>>> then?
>>
>> pci_hide_device() simply makes the device unavailable for assignment
>> (by having it owned by DomXEN). pci_ro_device() additionally adds the
>> device to the segment's ro_map, thus protecting its config space from
>> Dom0 writes.
> 
> But above you mention that hidden devices should be visible to dom0
> "in a (mostly) r/o fashion.".

I'm sorry for the confusion. My reply was in the context of the UART
question here, which is a "r/o device" case. I didn't realize you
were asking a question not directly related to such UART devices.

> I understand that for RO devices the whole config space of the device
> is RO, in which case we should simply avoid using vPCI for them.  We
> however likely want to have the BARs of such devices permanently
> mapped into the dom0 physmap (if memory decoding is enabled).
> 
> But for hidden devices it's not clear to me what needs to be RO, do we
> also need to protect the config space from dom0 accesses?

No, then they would be identical to r/o devices. Dom0 should be allowed
to deal with hidden devices normally, I think, with the sole exception
of them not being possible to assign to another domain (not even DomIO,
if I'm not mistaken).

> It might be complicated for vPCI to deal with devices that have MSI-X
> interrupts in use by Xen for example.  So I would suggest that at
> leeat for the time being we don't handle hidden devices with vPCI.

If any interrupts are in use on a device, I think it needs to be made
"r/o", not just "hidden".

Jan

> We might want to do similar to RO devices and prevent write access to
> the config space for those also.
> 
>> And just to clarify - a PCI dev containing a UART isn't "hidden" in the
>> above sense, but "r/o" (by virtue of calling pci_ro_device() on it).
>> But the issue reported long ago (and now re-discovered by Stefano) is
>> common to "r/o" and "hidden" devices (it's the "hidden" aspect that
>> counts here, which is common for both).
> 
> Indeed, the issue is with any device assigned to dom_xen (or not
> assigned to the hardware domain, but accessible by the hardware domain
> from vPCI).