Re: [Xen-users] Xen boot time optimization

[Julien Grall <julien@xxxxxxx>]

Hi,

I can't find the previous e-mail in my inbox. So I am answering to 
Bertrand's e-mail.
On 06/03/2020 08:21, Bertrand Marquis wrote:
> > On 6 Mar 2020, at 05:50, Subhasree P V 
> > <subhasree.pv@xxxxxxxxxxxxxxxxxxx 
> > <mailto:subhasree.pv@xxxxxxxxxxxxxxxxxxx>> wrote:
> > Thank you so much for your reply Bertrand.I have done it and was able 
> > to reduce about 520ms time..
> Nice.
>
> > But still the delay before kernel load is the same.I think it is the 
> > time taken for the processing the interrupts.about 4s delay is there 
> > before kernel load.You can see it in the below log.
Do you have an early console setup? If not, the "delay" is mostly 
because Linux has not yet initialized the console. This does not mean it 
was because of interrupt processing... Linux is probably just 
initializing itself.
> > [2020-03-06 10:23:32.817] (XEN) Freed 264kB init memory.
> > [2020-03-06 10:23:36.719] [    0.000000] [c0] Booting Linux on 
> > physical CPU 0x0
> > [2020-03-06 10:23:36.719] [    0.000000] [c0] Initializing cgroup 
> > subsys cpu
> > [2020-03-06 10:23:36.724] [    0.000000] [c0] Initializing cgroup 
> > subsys cpuacct
> > I have tried by setting *GICH_V2_LR_MAINTENANCE_IRQ* only for 
> > non-hardware irqs and avoiding for hardware irqs.1s reduction in the 
> > delay has been there when i have done so.But dom0 kernel was not 
> > loading completely while doing so.It was hanging at some point.I 
> > thought the delay might be because of  too much inflight irqs.Could 
> > you please help me to solve it.
The only reason we would request a maintenance IRQs is for virtual 
interrupts and if the LRs are full.
So what did you try to avoid? Could you provide the exact version of Xen 
your are using and the modification you tried?
> >
> You are probably missing IRQs in Linux when you did that as the 
> maintenance IRQ is needed by the vGIC.
> Regarding the delay you see between “Freed” and first linux message it 
> might be interesting to check if this time is actually spent in Xen or 
> in Linux because the Freed is the last message printed by Xen here.
> I cannot setup a system now to do some investigation on this but boot 
> time with xen is something I will be definitely look at in the next months.
> Regards
> Bertrand
>
> > Thanks & Regards,
> > Subhasree
> >
> >
> >
> >
> >
> >
> > On Thu, Mar 5, 2020 at 10:47 PM Bertrand Marquis 
> > <Bertrand.Marquis@xxxxxxx <mailto:Bertrand.Marquis@xxxxxxx>> wrote:
> >     Hi,
> >
> > >     On 4 Mar 2020, at 10:40, Subhasree P V
> > >     <subhasree.pv@xxxxxxxxxxxxxxxxxxx
> > >     <mailto:subhasree.pv@xxxxxxxxxxxxxxxxxxx>> wrote:
> > >
> > >     Hi,
> > >     I have been working with using XEN on ODROID-XU4 board.I would
> > >     like to optimize the boot time of XEN to 3s. Currently it is
> > >     taking about 5s to boot.I have disabled the prints.There is about
> > >     4s delay before kernel load.
> > >     Below is the log before disabling prints.
> > >
> > >     [2020-02-13 12:53:18.791] (XEN) Brought up 8 CPUs
> > >     [2020-02-13 12:53:18.791] (XEN) P2M: 40-bit IPA
> > >     [2020-02-13 12:53:18.791] (XEN) P2M: 3 levels with order-1 root,
> > >     VTCR 0x80003558
> > >     [2020-02-13 12:53:18.797] (XEN) I/O virtualisation disabled
> > >     [2020-02-13 12:53:18.802] (XEN) *** LOADING DOMAIN 0 ***
> > >     [2020-02-13 12:53:18.802] (XEN) Loading kernel from boot module @
> > >     0000000060000000
> > >     [2020-02-13 12:53:18.808] (XEN) Allocating 1:1 mappings totalling
> > >     1024MB for dom0:
> > >     [2020-02-13 12:53:19.578] (XEN) BANK[0]
> > >     0x00000070000000-0x000000b0000000 (1024MB)
> This step takes 800ms
Which is not surprising given a 1GB of memory is allocated. The 
allocator will go through each page to mark them as inuse and also 
assign to the domain.
A couple of years ago, Samsung gave a good talk about reducing boot time 
on Xen (see [1]). This might be give you some clue where are the biggest 
time spending.
Most of Xen boot time is spent on memory initialization (i.e setup_mm()) 
and dom0 memory allocation (i.e allocate_memory()).
> > This message contains confidential information and is intended only 
> > for the individual(s) named.If you are not the intended recipient, you 
> > are notified that disclosing, copying, distributing or taking any 
> > action in reliance on the contents of this mail and attached file/s is 
> > strictly prohibited. Please notify the sender immediately and delete 
> > this e-mail from your system. E-mail transmission cannot be guaranteed 
> > to be secured or error-free as information could be intercepted, 
> > corrupted, lost, destroyed, arrive late or incomplete, or contain 
> > viruses. The sender therefore does not accept liability for any errors 
> > or omissions in the contents of this message, which arise as a result 
> > of e-mail transmission.
This is public mailing list, please configure your e-mail client to 
remove the disclaimer.
Cheers,

[1] 
https://static.sched.com/hosted_files/xensummit18/48/Xen_Summit2018_Samsung%20Automotive%20Virtualization.pdf
--
Julien Grall

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users