Re: Disable swiotlb for Dom0

[Roman Skakun <Roman_Skakun@xxxxxxxx>]

Hi Julien,

> So 0xb6000000 is most likely the GFN used to mapped the grant from the domU.
>
> swiotlb-xen on Arm will convert it to the MFN because it is not aware
> whether the device is behind an IOMMU.

If I'm understand right, it seems like that swiotlb-xen is not ready to work properly in case 

when we retrieved MFN instead of proper GFN mapped to Dom0 memory.
Maybe you know some ideas to overcome this condition?

>  As the address is too high to be handled by the device, swiotlb will try
>  to bounce it. I think it is correct to bounce the page but I am not sure
>  why it can't. What the size of the DMA transaction?

The DMA map size is 3686400 bytes.

I've added several logs to swiotlb map_single() and see:
[  151.298455] <SWIOTLB> swiotlb_tbl_map_single() origin_addr: 64af97000, needed: 708, 

avail: 7fc0, stride: 2, index: 4160

It's expected because:
....
[  259.468006] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31742]=2 slots < 708 nslots. Continue...

[  259.477070] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31744]=80 slots < 708 nslots. Continue...

[  259.486229] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31746]=7e slots < 708 nslots. Continue...

[  259.495387] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31748]=7c slots < 708 nslots. Continue...

[  259.504546] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31750]=7a slots < 708 nslots. Continue...

[  259.513704] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31752]=78 slots < 708 nslots. Continue...

[  259.522863] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31754]=76 slots < 708 nslots. Continue...

[  259.532021] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31756]=74 slots < 708 nslots. Continue...

[  259.541179] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31758]=72 slots < 708 nslots. Continue...

[  259.550338] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31760]=70 slots < 708 nslots. Continue...

[  259.559496] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31762]=6e slots < 708 nslots. Continue...

[  259.568660] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31764]=6c slots < 708 nslots. Continue...

[  259.577813] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31766]=6a slots < 708 nslots. Continue...

[  259.586972] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31768]=68 slots < 708 nslots. Continue...

[  259.596130] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31770]=66 slots < 708 nslots. Continue...

[  259.605289] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31772]=64 slots < 708 nslots. Continue...

[  259.614447] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31774]=62 slots < 708 nslots. Continue...

[  259.623606] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31776]=60 slots < 708 nslots. Continue...

[  259.632764] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31778]=5e slots < 708 nslots. Continue...

[  259.641922] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31780]=5c slots < 708 nslots. Continue...

[  259.651081] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31782]=5a slots < 708 nslots. Continue...

[  259.660239] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31784]=58 slots < 708 nslots. Continue...

[  259.669398] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31786]=56 slots < 708 nslots. Continue...

[  259.678563] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31788]=54 slots < 708 nslots. Continue...

[  259.687714] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31790]=52 slots < 708 nslots. Continue...

[  259.696873] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31792]=50 slots < 708 nslots. Continue...

[  259.706032] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31794]=4e slots < 708 nslots. Continue...

[  259.715190] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31796]=4c slots < 708 nslots. Continue...

[  259.724348] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31798]=4a slots < 708 nslots. Continue...

[  259.733507] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31800]=48 slots < 708 nslots. Continue...

[  259.742665] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31802]=46 slots < 708 nslots. Continue...

[  259.751824] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31804]=44 slots < 708 nslots. Continue...

[  259.760982] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31806]=42 slots < 708 nslots. Continue...

[  259.770141] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31808]=40 slots < 708 nslots. Continue...

[  259.779299] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31810]=3e slots < 708 nslots. Continue...

[  259.788466] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31812]=3c slots < 708 nslots. Continue...

[  259.797615] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31814]=3a slots < 708 nslots. Continue...

[  259.806774] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31816]=38 slots < 708 nslots. Continue...

[  259.815933] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31818]=36 slots < 708 nslots. Continue...

[  259.825091] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31820]=34 slots < 708 nslots. Continue...

[  259.834249] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31822]=32 slots < 708 nslots. Continue...

[  259.843408] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31824]=30 slots < 708 nslots. Continue...

[  259.852567] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31826]=2e slots < 708 nslots. Continue...

[  259.861725] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31828]=2c slots < 708 nslots. Continue...

[  259.870883] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31830]=2a slots < 708 nslots. Continue...

[  259.880042] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31832]=28 slots < 708 nslots. Continue...

[  259.889200] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31834]=26 slots < 708 nslots. Continue...

[  259.898365] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31836]=24 slots < 708 nslots. Continue...

[  259.907516] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31838]=22 slots < 708 nslots. Continue...

[  259.916676] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31840]=20 slots < 708 nslots. Continue...

[  259.925834] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31842]=1e slots < 708 nslots. Continue...

[  259.934992] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31844]=1c slots < 708 nslots. Continue...

[  259.944151] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31846]=1a slots < 708 nslots. Continue...

[  259.953309] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31848]=18 slots < 708 nslots. Continue...

[  259.962468] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31850]=16 slots < 708 nslots. Continue...

[  259.971626] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31852]=14 slots < 708 nslots. Continue...

[  259.980784] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31854]=12 slots < 708 nslots. Continue...

[  259.989943] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31856]=10 slots < 708 nslots. Continue...

[  259.999102] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31858]=e slots < 708 nslots. Continue...

[  260.008181] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31860]=c slots < 708 nslots. Continue...

[  260.017245] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31862]=a slots < 708 nslots. Continue...

[  260.026318] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31864]=8 slots < 708 nslots. Continue...

[  260.035389] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31866]=6 slots < 708 nslots. Continue...

[  260.044461] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31868]=4 slots < 708 nslots. Continue...

[  260.053533] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31870]=2 slots < 708 nslots. Continue...

[  260.062606] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31872]=80 slots < 708 nslots. Continue...
....

Swiotlb did not fit requested slots because the maximum slot size equals IO_TLB_SEGSIZE=128 by default.

But I think, we cannot use 64af97000 address in the swiotlb_bounce() directly.

---

From: Julien Grall <julien@xxxxxxx>
Sent: Wednesday, August 11, 2021 6:20 PM
To: Roman Skakun <Roman_Skakun@xxxxxxxx>; sstabellini@xxxxxxxxxx <sstabellini@xxxxxxxxxx>
Cc: Bertrand Marquis <bertrand.marquis@xxxxxxx>; Andrii Anisov <Andrii_Anisov@xxxxxxxx>; Volodymyr Babchuk <Volodymyr_Babchuk@xxxxxxxx>; Oleksandr Tyshchenko <Oleksandr_Tyshchenko@xxxxxxxx>; Oleksandr Andrushchenko <Oleksandr_Andrushchenko@xxxxxxxx>; xen-devel@xxxxxxxxxxxxxxxxxxxx <xen-devel@xxxxxxxxxxxxxxxxxxxx>; Roman Skakun <rm.skakun@xxxxxxxxx>; Jan Beulich <jbeulich@xxxxxxxx>
Subject: Re: Disable swiotlb for Dom0

 

Hi,

On 11/08/2021 15:13, Roman Skakun wrote:
>> > Also, I added the log in xen_swiotlb_detect() and can see that swiotlb
>> > still used (other devices within dom0 used too), when dom0 is direct
> mapped:
>> >
>> > [    1.870363] xen_swiotlb_detect() dev: rcar-fcp,
>> > XENFEAT_direct_mapped, use swiotlb
>> > [    1.878352] xen_swiotlb_detect() dev: rcar-fcp,
>> > XENFEAT_direct_mapped, use swiotlb
>> > [    1.886309] xen_swiotlb_detect() dev: rcar-fcp,
>> > XENFEAT_direct_mapped, use swiotlb
>> >
>>> This means, that all  devices are using swiotlb-xen DMA fops.
>> > By the way, before applying this patches, dom0 always used swiotlb-xen
>> > fops for initial domain by design.
>
>> >This is expected because your domain is direct mapped.
>
> May be, I don't understand right, Stefano reported the same issue when
> dom0 is not direct mapped,
> but I have direct mapped dom0 and problem still exists.

I am not entirely sure why you think this is the same problem as
Stefano. He asked to bypass the swiotlb, but AFAIK, this is not because
the buffer get bounced.

Instead, it is because swiotlb-xen on Arm has been relying on its RAM to
be direct-mapped (GFN == MFN). With cache coloring, the memory will not
be direct-mapped, hence it will be broken.

>
>>Ok. Would you be able to provide more information on where the dom0
>>memory is allocated  and the list of host RAM?
>
> Host memory:
> DRAM:  7.9 GiB
> Bank #0: 0x048000000 - 0x0bfffffff, 1.9 GiB
> Bank #1: 0x500000000 - 0x57fffffff, 2 GiB
> Bank #2: 0x600000000 - 0x67fffffff, 2 GiB
> Bank #3: 0x700000000 - 0x77fffffff, 2 GiB
>
> dom0 memory map:
> (XEN) Allocating 1:1 mappings totalling 2048MB for dom0:
> (XEN) BANK[0] 0x00000048000000-0x00000050000000 (128MB)
> (XEN) BANK[1] 0x00000058000000-0x000000c0000000 (1664MB)
> (XEN) BANK[2] 0x00000510000000-0x00000520000000 (256MB)

Thanks! So you have some memory assigned above 4GB to dom0 as well.

>>> We retrieved dev_addr(64b1d0000)  + size > 32bit mask, but fcp driver
>>> wants to use only  32 bit boundary address, but that's consequence.
>>>
>> Ok. So your device is only capable to do a 32-bit DMA. Is that correct?
>
> Yes.
>
>> > I think, the main reason of using bounce buffer is MFN address, not DMA
>> > phys address.
>> >
>>I don't understand this sentence. Can you clarify it?
>
> This address looks like theMFN because I'm using mapped grant tables
> from domU.
>
> I've added the log and see the following:
> with swiotlb:
> [   78.620386] dma_map_sg_attrs() dev: rcar-du swiotlb, sg_page:
> fffffe0001b80000, page_to_phy: b6000000, xen_phys_to_dma: 64b1d0000
>
> without swiotlb (worked fine):
> [   74.456426] dma_map_sg_attrs() dev: rcar-du direct map, sg_page:
> fffffe0001b80000, page_to_phy: b6000000, xen_phys_to_dma:b6000000
>
> I guess, need to figure out why we got a normal dom0 DMA address
> (b6000000) and why 64b1d0000 when using swiotlb.

So 0xb6000000 is most likely the GFN used to mapped the grant from the domU.

swiotlb-xen on Arm will convert it to the MFN because it is not aware
whether the device is behind an IOMMU.

As the address is too high to be handled by the device, swiotlb will try
to bounce it. I think it is correct to bounce the page but I am not sure
why it can't. What the size of the DMA transaction?

However, even if you disable xen-swiotlb, you are likely going to face
the same issue sooner or later because the grant can be mapped anywhere
in the memory of dom0 (the balloon code doesn't look to restrict where
the memory can be allocated). So it is possible for the grant to be
mapped in the dom0 memory above 4GB.

Oleksandr is also looking to provide a safe range which would be outside
of the existing RAM. So, I believe, you will have to bounce the DMA
buffer unless we always force the grant/foreign mapping to be mapped
below 4GB.

Cheers,

--
Julien Grall