[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: HYPERVISOR_console_io + CONSOLEIO_write needs COPY_flush_dcache


  • To: Julien Grall <julien@xxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • From: Oleksandr Andrushchenko <Oleksandr_Andrushchenko@xxxxxxxx>
  • Date: Thu, 11 Jun 2020 12:18:18 +0000
  • Accept-language: en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xxjhrZ/jHmB4WOxIajiLtMregBOwhoysRuaJ/WsOvHo=; b=HEBGw3upNb9k0YcR3KOKLFnzUh9xvcp8yHPtzvfF687UYqnPCghdIBKcPqoU4z+0M1ZaJG6h5e5cpiQi7bkVhgceGkrVPAU2DKEjgjW0qGscK6BZNmyk2CZrmS61tZSFrkCzpfEL+QRIDCLyPDse5y178vVRhF8AEyo7CUaiDo0YeePx/bSV/vkFauCsGxorj70kw94YTaiBpXWJrW0g7bCPp0DVUxoNUjadJ4hDsn86uy8vgGaN3GwEWE/WEJSG6oN861PD/gFUcYAafGyOalTtsAutCXS5kDAT5ZtCDtOsDPL1jJawrTwCQ8xEtEF3Pmjr3LOCAFHaO4SxvqEgAA==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HuV4oV0s5UHwF3XyaDbe5Pnaqv+vZxYkRgYxP653ZK47YUgHSObMBIkB4EbbHP9nyECIZUu4zazOfRDVvNx1f2NNlMT8/CPCOU7dwejVHUXbJXQxODSVR+kU3zQ/lNTOg0iTFDtpCZvSYleyKgPTgmezgRbywmFeSshZTdtQEW8JQdE+EvnjbtYlDmXaDxQ0ffzUKxrh/Pc3AUCIOg4XXrtceVPA+ynsXgJvJ9O/GMKQr69N+XcmXbcZfMKTvAslPQx7SiLcP1xveHf6H6VWj/W2LLJQdecCWARFEwiebYK42UqtgypFtAYnk6nREcAnKlFx96OJ6eGpV9V4jTr4Lw==
  • Authentication-results: xen.org; dkim=none (message not signed) header.d=none;xen.org; dmarc=none action=none header.from=epam.com;
  • Cc: Stefano Stabellini <sstabellini@xxxxxxxxxx>, Bertrand Marquis <Bertrand.Marquis@xxxxxxx>
  • Delivery-date: Thu, 11 Jun 2020 12:18:30 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Thread-index: AQHWPv8SOx/IaSMb+EOuO5msEU03SajRzVWAgAGJcoA=
  • Thread-topic: HYPERVISOR_console_io + CONSOLEIO_write needs COPY_flush_dcache

On 6/10/20 3:50 PM, Julien Grall wrote:
> On 10/06/2020 09:13, Oleksandr Andrushchenko wrote:
>> Hello,
>
> Hi,
>
>> While adding support for para-virtualized block device in u-boot
>>
>> I faced an issue with HYPERVISOR_console_io + CONSOLEIO_write.
>>
>> I tried to use the hypercall during MMU setup stage while enabling data cache
>>
>> on arm64 platform (e.g. data cache is not yet enabled) and saw in guest's 
>> console
>>
>> either old data or some sort of garbage. Printing constant strings, just 
>> like mini-os
>>
>> does on boot, works as expected. 
>
> Per the hypercall ABI (see include/public/arch-arm.h) all the buffers must 
> reside in memory which is mapped as Normal Inner Write-Back Inner-Shareable.
>
> You are passing non-cacheable memory, so the behavior you see is expected.
This is what we come up with as well
>
>> So, looking at the Xen code [1] it seems
>>
>> that we should copy guest's buffer with COPY_flush_dcache set in this case
>
> COPY_flush_dcache is only meant to be used when copy to guest memory to make 
> sure the data reached the point of coherency in your system. It is not meant 
> to be used when copying from guest.
Understood and agree
>
>> as (usually?) this hypercall is used in guest's code which doesn't have any
>> other means to print yet, e.g. early printk case.
>
> I have been using it after the MMU is setup 
The thing is that in u-boot a lot of things happen before the MMU setup...
> but before the console is properly setup by the guest (there is a lot that 
> can happen in Linux between the two). In my case, the flush is not necessary 
> and would be wrong.
... and without the flush you lose the ability to debug that code.
>
> In general, the cache is never off on Arm64 because you may have system cache 
> or, in the cache of virtualization, cacheable mapping in the hypervisor (all 
> the memory is mapped).
>
> When updating data with MMU/D-cache off the normal approach is:
>    1) Remove any dirty lines from the cache, otherwise they may get evicted 
> while updating the memory and override any value written with MMU off.
>    2) Update the memory
>    3) Remove any lines that may have been speculated so when you turn onthe 
> MMU and D-cache, U-boot can obverse the correct data.
>
> Step 1 is only necessary if you are going to write outside of the loaded 
> Image (BSS is not part of it).
>
> You most likely already have similar steps in U-boot for other part of the 
> memory access with MMU/D-Cache off. So I think U-boot is the best place to 
> invalidate the cache before issuing the hypercall.

Yes, I have invalidated the buffers and everything works like a charm now, 
thanks

Probably my use-case should be somewhere documented, so another time someone 
steps into similar issues it is explained.

>
> Cheers,
>
Thank you for helping,

Oleksandr

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.