Re: Limitations for Running Xen on KVM Arm64

[Julien Grall <julien@xxxxxxx>]

(+ the other Arm maintainers)

On 31/10/2025 13:01, haseeb.ashraf@xxxxxxxxxxx wrote:
> Hello,
Hi,

Before answering to the rest, would you be able to configure your e-mail 
client to quote with '>' and avoid top-posting? Otherwise, it will 
become quite difficult to follow the conversation after a few round.
> I have seen no such performance issue with nested KVM. For Xen, if this 
> can be relaxed from |vmalls12e1| to |vmalle1|, this would still be a 
> huge performance improvement. I used Ftrace to get execution time of 
> each of these handler functions:
> handle_vmalls12e1is() min-max = 1464441 - 9495486 us
To clarify, Xen is using the local TLB version. So it should be 
vmalls12e1. But it looks like KVM will treat it the same way and I 
wonder whether this could be optimized? (I don't know much about the KVM 
implementation though).
> So, to summarize using HCR_EL2.FB (which Xen already enables?) and then 
> using vmalle1 instead of vmalls12e1 should resolve the issue-2 for vCPUs 
> switching on pCPUs.
I don't think HCR_EL2.FB would matter here.

> Coming back to issue-1, what do you think about creating a batch version 
> of hypercall XENMEM_remove_from_physmap (other batch versions exist such 
> as for XENMEM_add_to_physmap) and doing the TLB invalidation only once 
> per this hypercall?
Before going into batching, do you have any data showing how often 
XENMEM_remove_from_physmap is called in your setup? Similar, I would be 
interested to know the number of TLBs flush within one hypercalls and 
whether the regions unmapped were contiguous.
In your previous e-mail you wrote:

> During the creation of domu, first the domu memory is mapped onto 
dom0 domain, images are copied into it, and it is then unmapped. During 
unmapping, the TLB translations are invalidated one by one for each page 
being unmapped in XENMEM_remove_from_physmap hypercall. Here is the code 
snippet where the decision to flush TLBs is being made during removal of 
mapping.
Don't we map only the memory that is needed to copy the binaries? If 
not, then I would suggest to look at that first.
I am asking because even with batching, we may still send a few TLBs 
because:
   * We need to avoid long-running operations, so the hypercall may 
restart. So we will have to flush at mininum before every restart
   * The current way we handle batching is we will process one item at 
the time. As this may free memory (either leaf or intermediate 
page-tables), we will need to flush the TLBs first to prevent the domain 
accessing the wrong memory. This could be solved by keeping track of the 
list of memory to free. But this is going to require some work and I am 
not entirely sure this is worth it at the moment.
> I just realized that ripas2e1 is a range TLBI 
> instruction which is only supported after Armv8.4 indicated 
> by ID_AA64ISAR0_EL1.TLB == 2. So, on older architectures, full stage-2 
> invalidation would be required. For an architecture independent 
> solution, creating a batch version seems to be a better way.
I don't think we necessarily need a full stage-2 invalidation for 
processor not supporting range TLBI. We could use a series of TLBI 
IPAS2E1IS which I think is what TBLI range is meant to replace (so long 
the addresses are contiguous in the given space).
On the KVM side, it would be worth looking at whether the implementation 
can be optimized. Is this really walking block by block? Can it skip 
over large hole (e.g. if we know a level 1 entry doesn't exist, then we 
can increment by 1GB).
Cheers,

--
Julien Grall