Re: [PATCH v2 01/17] xen/riscv: implement sbi_remote_hfence_gvma()

[Jan Beulich <jbeulich@xxxxxxxx>]

On 24.06.2025 12:33, Oleksii Kurochko wrote:
> On 6/18/25 5:15 PM, Jan Beulich wrote:
>> On 10.06.2025 15:05, Oleksii Kurochko wrote:
>>> Instruct the remote harts to execute one or more HFENCE.GVMA instructions,
>>> covering the range of guest physical addresses between start_addr and
>>> start_addr + size for all the guests.
>> Here and in the code comment: Why "for all the guests"? Under what conditions
>> would you require such a broad (guest) TLB flush?
> 
> Hmm, it seems like KVM always do such a broad (guest) TLB flush during 
> detection
> of VMIDLEN:
>       void __init kvm_riscv_gstage_vmid_detect(void)
>       {
>               unsigned long old;
>       
>               /* Figure-out number of VMID bits in HW */
>               old = csr_read(CSR_HGATP);
>               csr_write(CSR_HGATP, old | HGATP_VMID);
>               vmid_bits = csr_read(CSR_HGATP);
>               vmid_bits = (vmid_bits & HGATP_VMID) >> HGATP_VMID_SHIFT;
>               vmid_bits = fls_long(vmid_bits);
>               csr_write(CSR_HGATP, old);
>       
>               /* We polluted local TLB so flush all guest TLB */
>               kvm_riscv_local_hfence_gvma_all();
>       
>               /* We don't use VMID bits if they are not sufficient */
>               if ((1UL << vmid_bits) < num_possible_cpus())
>                       vmid_bits = 0;
>       }
> 
> It is not clear actually why so broad and why not hfence_gvma_vmid(vmid_bits).
> 
> And I am not really 100% sure that any hfence_gvma() is needed here as I 
> don't see
> what could pollutes local guest TLB between csr_write() calls.
> 
> RISC-V spec. says that:
>       Note that writing hgatp does not imply any ordering constraints between 
> page-table updates and
>       subsequent G-stage address translations. If the new virtual machine’s 
> guest physical page tables have
>       been modified, or if a VMID is reused, it may be necessary to execute 
> an HFENCE.GVMA instruction
>       (see Section 18.3.2) before or after writing hgatp.
> 
> But we don't modify VM's guest physical page table. We could potentially 
> reuse VMID between csr_write()
> calls, but it is returning back and we don't switch to a guest with this 
> "new" VMID, so it isn't really used.

That would be my expectation, too. Yet I don't know if RISC-V has any
peculiarities there.

Jan