Re: Question about xen and Rasp 4B

[Jukka Kaartinen <jukka.kaartinen@xxxxxxxxxx>]

> > Glad to hear it works! IIRC, the swiotlb may become necessary when running
> > guest if the guest memory ends up to be used for DMA transaction.
> It is necessary if you are using PV network or PV disk: memory shared by
> another domU could end up being used in a DMA transaction. For that, you
> need swiotlb-xen.
Sounds bad :).


> > > Now that swiotlb is disabled what does it mean?
> > I can see two reasons:
> >    1) You have limited memory below the 30 bits mark. So Swiotlb and CMA may
> > try to fight for the low memory.
> >    2) We found a few conversion bugs in the swiotlb on RPI4 last year (IIRC the
> > DMA and physical address may be different). I looked at the Linux branch you
> > are using and they seem to all be there. So there might be another bug.
> >
> > I am not sure how to figure out where is the problem. Stefano, do you have a
> > suggestion where to start?
> Both 1) and 2) are possible. It is also possible that another driver,
> probably something related to CMA or DRM, has some special dma_ops
> handling that doesn't work well together with swiotlb-xen.
>
> Given that the original error seemed to be related to vc4_bo_create,
> which calls dma_alloc_wc, I would add a couple of printks to
> xen_swiotlb_alloc_coherent to help us figure it out:
>
>
> diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
> index 2b385c1b4a99..cac8b09af603 100644
> --- a/drivers/xen/swiotlb-xen.c
> +++ b/drivers/xen/swiotlb-xen.c
> @@ -295,6 +295,7 @@ xen_swiotlb_alloc_coherent(struct device *hwdev, size_t 
> size,
>         /* Convert the size to actually allocated. */
>         size = 1UL << (order + XEN_PAGE_SHIFT);
>   
> +	printk("DEBUG %s %d size=%lu flags=%lx attr=%lx\n",__func__,__LINE__,size,flags,attrs);
>         /* On ARM this function returns an ioremap'ped virtual address for
>          * which virt_to_phys doesn't return the corresponding physical
>          * address. In fact on ARM virt_to_phys only works for kernel direct
> @@ -315,16 +316,20 @@ xen_swiotlb_alloc_coherent(struct device *hwdev, size_t 
> size,
>         phys = dma_to_phys(hwdev, *dma_handle);
>         dev_addr = xen_phys_to_dma(hwdev, phys);
>         if (((dev_addr + size - 1 <= dma_mask)) &&
> -           !range_straddles_page_boundary(phys, size))
> +           !range_straddles_page_boundary(phys, size)) {
>                 *dma_handle = dev_addr;
> -       else {
> +               printk("DEBUG %s %d phys=%lx 
> dma=%lx\n",__func__,__LINE__,phys,dev_addr);
> +       } else {
>                 if (xen_create_contiguous_region(phys, order,
>                                                  fls64(dma_mask), dma_handle) 
> != 0) {
> +                       printk("DEBUG %s %d\n",__func__,__LINE__);
>                         xen_free_coherent_pages(hwdev, size, ret, 
> (dma_addr_t)phys, attrs);
>                         return NULL;
>                 }
>                 *dma_handle = phys_to_dma(hwdev, *dma_handle);
>                 SetPageXenRemapped(virt_to_page(ret));
> +               printk("DEBUG %s %d dma_mask=%d page_boundary=%d phys=%lx 
> dma=%lx\n",__func__,__LINE__,
> +                       ((dev_addr + size - 1 <= 
> dma_mask)),range_straddles_page_boundary(phys, size),phys,*dma_handle);
>         }
>         memset(ret, 0, size);
>         return ret;
Thanks I will try this.

> > > And also can I pass the GPU to domU? Raspberry Pi 4 is limited HW and
> > > doesn't have IOMMU. I'm trying to create similar OS like QubesOS where GPU,
> > > Network, keyboard/mouse, ... are isolated to their own VMs.
> > Without an IOMMU or any other HW mechamisns (e.g. MPU), it would not be safe
> > to assign a DMA-capable device to a non-trusted VM.
> >
> > If you trust the VM where you assigned a device, then a possible approach
> > would be to have the VM direct mapped (e.g. guest physical address == host
> > physical address). Although, I can foreese some issues if you have multiple
> > VMs requires memory below 30 bits (there seem to be limited amount)>
> >
> > If you don't trust the VM where you assigned a device, then your best option
> > will be to expose a PV interface of the device and have your backend
> > sanitizing the request and issuing it on behalf of the guest.
> FYI you could do that with the existing PVFB drivers that only support 2D
> graphics
I'll keep that in my mind, thanks.