Re: [Xen-devel] Xen optimization

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>]

On Fri, 12 Oct 2018, Milan Boberic wrote:
> Hi Stefano, glad to have you back :D,
> this is my setup:
>         - dom0 is PetaLinux, has 1 vCPU and it's pinned for pCPU0
>         - there is only one domU and this is my bare-metal app that also have 
> one vCPU and it's pinned for pCPU1
> so yeah, there is only dom0 and bare-metal app on the board.
> 
> Jitter is the same with and without Dario's patch.
> 
> I'm still not sure about timer's passthrough because there is no mention of 
> triple timer counter is device tree so I added:
> 
> &ttc0 {
>    xen,passthrough = <0x1>;
> };
> 
> at the end of the xen-overlay.dtsi file which I included in attachment.

This is definitely wrong. Can you please also post the full host device
tree with your modifications that you are using for Xen and Dom0?  You
should have something like:


        timer@ff110000 {
            compatible = "cdns,ttc";
            interrupt-parent = <0x2>;
            interrupts = <0x0 0x24 0x4 0x0 0x25 0x4 0x0 0x26 0x4>;
            reg = <0x0 0xff110000 0x0 0x1000>;
            timer-width = <0x20>;
            power-domains = <0x3b>;
            xen,passthrough;
        };

For each of the nodes of the devices you are assigning to the DomU.


> About patch you sent, I can't find this funcion void vgic_inject_irq in 
> /xen/arch/arm/vgic.c file, this is link of git repository
> from where I build my xen so you can take a look if that printk can be put 
> somewhere else.
> 
> https://github.com/Xilinx/xen/

It's here: 
https://github.com/Xilinx/xen/blob/xilinx/stable-4.9/xen/arch/arm/vgic.c#L462

BTW you are using a pretty old branch, I suggest you moving to:

https://github.com/Xilinx/xen/tree/xilinx/versal/xen/arch/arm

It will work on your board too and it is based on the much newer Xen
4.11.


> I ran some more testing and realized that results are the same with or 
> without vwfi=native, which I think again points out that
> passthrough that I need to provide in device tree isn't valid.

In reality, the results are the same with and without vwfi=native only
if the baremetal app never issues any wfi instructions.


>  And of course, higher the frequency of interrupts results in higher jitter. 
> I'm still battling with Xilinx SDK and triple timer
> counter that's why I can't figure out what is the exact frequency set (I'm 
> just rising it and lowering it), I'll give my best to
> solve that ASAP because we need to know exact value of frequency set. 

Yep, that's important :-)


> 
> Thanks in advance!
> 
> Milan
>  
> 
> 
> On Fri, Oct 12, 2018 at 12:29 AM Stefano Stabellini 
> <stefano.stabellini@xxxxxxxxxx> wrote:
>       On Thu, 11 Oct 2018, Milan Boberic wrote:
>       > On Wed, Oct 10, 2018 at 6:41 PM Meng Xu <xumengpanda@xxxxxxxxx> wrote:
>       > >
>       > > The jitter may come from Xen or the OS in dom0.
>       > > It will be useful to know what is the jitter if you run the test on 
> PetaLinux.
>       > > (It's understandable the jitter is gone without OS. It is also 
> common
>       > > that OS introduces various interferences.)
>       >
>       > Hi Meng,
>       > well... I'm using bare-metal application and I need it exclusively to
>       > be ran on one CPU as domU (guest) without OS (and I'm not sure how
>       > would I make the same app to be ran on PetaLinux dom0 :D haha).
>       > Is there a chance that PetaLinux as dom0 is creating this jitter and
>       > how? Is there a way of decreasing it?
>       >
>       > Yes, there are no prints.
>       >
>       > I'm not sure about this timer interrupt passthrough because I didn't
>       > find any example of it, in attachment I included xen-overlay.dtsi file
>       > which I edited to add passthrough, in earlier replies there are
>       > bare-metal configuration file. It would be helpful to know if those
>       > setting are correct. If they are not correct it would explain the
>       > jitter.
>       >
>       > Thanks in advance, Milan Boberic!
> 
>       Hi Milan,
> 
>       Sorry for taking so long to go back to this thread. But I am here now :)
> 
>       First, let me ask a couple of questions to understand the scenario
>       better: is there any interference from other virtual machines while you
>       measure the jitter? Or is the baremetal app the only thing actively
>       running on the board?
> 
>       Second, it would be worth double-checking that Dario's patch to fix
>       sched=null is not having unexpected side effects. I don't think so, it
>       would be worth testing with it and without it to be sure.
> 
>       I gave a look at your VM configuration. The configuration looks correct.
>       There is no dtdev settings, but given that none of the devices you are
>       assigning to the guest does any DMA, it should be OK. You want to make
>       sure that Dom0 is not trying to use those same devices -- make sure to
>       add "xen,passthrough;" to each corresponding node on the host device
>       tree.
> 
>       The error messages "No valid vCPU found" are due to the baremetal
>       applications trying to configure as target cpu for the interrupt cpu1
>       (the second cpu in the system), while actually only 1 vcpu is assigned
>       to the VM. Hence, only cpu0 is allowed. I don't think it should cause
>       any jitter issues, because the request is simply ignored. Just to be
>       safe, you might want to double check that the physical interrupt is
>       delivered to the right physical cpu, which would be cpu1 in your
>       configuration, the one running the only vcpu of the baremetal app. You
>       can do that by adding a printk to xen/arch/arm/vgic.c:vgic_inject_irq,
>       for example:
> 
>       diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>       index 5a4f082..208fde7 100644
>       --- a/xen/arch/arm/vgic.c
>       +++ b/xen/arch/arm/vgic.c
>       @@ -591,6 +591,7 @@ void vgic_inject_irq(struct domain *d, struct vcpu 
> *v, unsigned int virq,
>        out:
>            spin_unlock_irqrestore(&v->arch.vgic.lock, flags);
> 
>       +    if (v != current) printk("DEBUG irq slow path!\n");
>            /* we have a new higher priority irq, inject it into the guest */
>            vcpu_kick(v);
> 
>       You don't want "DEBUG irq slow path!" to get printed.
> 
>       Finally, I would try to set the timer to generate events less frequently
>       than every 1us and see what happens, maybe every 5-10us. In my tests,
>       the IRQ latency overhead caused by Xen is around 1us, so injecting 1
>       interrupt every 1us, plus 1us of latency caused by Xen, cannot lead to
>       good results.
> 
>       I hope this helps, please keep us updated with your results, they are
>       very interesting!
> 
> 
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel