Re: [PATCH v2 3/5] automation: Add the expect script with test case for FVP

[Henry Wang <Henry.Wang@xxxxxxx>]

Hi Julien,

> On Dec 8, 2023, at 20:05, Julien Grall <julien@xxxxxxx> wrote:
> 
> Hi,
> 
> On 08/12/2023 09:50, Michal Orzel wrote:
>> On 08/12/2023 10:21, Henry Wang wrote:
>>>> On Dec 8, 2023, at 17:11, Michal Orzel <michal.orzel@xxxxxxx> wrote:
>>>> On 08/12/2023 10:05, Henry Wang wrote:
>>>>> 
>>>>> Hi Michal,
>>>>> 
>>>>>> On Dec 8, 2023, at 16:57, Michal Orzel <michal.orzel@xxxxxxx> wrote:
>>>>>> 
>>>>>> Hi Henry,
>>>>>> 
>>>>>> On 08/12/2023 06:46, Henry Wang wrote:
>>>>>>> diff --git a/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp 
>>>>>>> b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
>>>>>>> new file mode 100755
>>>>>>> index 0000000000..25d9a5f81c
>>>>>>> --- /dev/null
>>>>>>> +++ b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
>>>>>>> @@ -0,0 +1,73 @@
>>>>>>> +#!/usr/bin/expect
>>>>>>> +
>>>>>>> +set timeout 2000
>>>>>> Do we really need such a big timeout (~30 min)?
>>>>>> Looking at your test job, it took 16 mins (quite a lot but I know FVP is 
>>>>>> slow
>>>>>> + send_slow slows things down)
>>>>> 
>>>>> This is a really good question. I did have the same question while 
>>>>> working on
>>>>> the negative test today. The timeout 2000 indeed will fail the job at 
>>>>> about 30min,
>>>>> and waiting for it is indeed not really pleasant.
>>>>> 
>>>>> But my second thought would be - from my observation, the overall time now
>>>>> would vary between 15min ~ 20min, and having a 10min margin is not that 
>>>>> crazy
>>>>> given that we probably will do more testing from the job in the future, 
>>>>> and if the
>>>>> GitLab Arm worker is high loaded, FVP will probably become slower. And 
>>>>> normally
>>>>> we don’t even trigger the timeout as the job will normally pass. So I 
>>>>> decided
>>>>> to keep this.
>>>>> 
>>>>> Mind sharing your thoughts about the better value of the timeout? 
>>>>> Probably 25min?
>>>> From what you said that the average is 15-20, I think we can leave it set 
>>>> to 30.
>>>> But I wonder if we can do something to decrease the average time. ~20 min 
>>>> is a lot
>>>> even for FVP :) Have you tried setting send_slow to something lower than 
>>>> 100ms?
>>>> That said, we don't send too many chars to FVP, so I doubt it would play a 
>>>> major role
>>>> in the overall time.
>>> 
>>> I agree with the send_slow part. Actually I do have the same concern, here 
>>> are my current
>>> understanding and I think you will definitely help with your knowledge:
>>> If you check the full log of Dom0 booting, for example [1], you will find 
>>> that we wasted so
>>> much time in starting the services of the OS (modloop, udev-settle, etc). 
>>> All of these services
>>> are retried many times but in the end they are still not up, and from my 
>>> understanding they
>>> won’t affect the actual test(?) If we can somehow get rid of these services 
>>> from rootfs, I think
>>> we can save a lot of time.
>>> 
>>> And honestly, I noticed that qemu-alpine-arm64-gcc suffers from the same 
>>> problem and it also
>>> takes around 15min to finish. So if we managed to tailor the services from 
>>> the filesystem, we
>>> can save a lot of time.
>> That is not true. Qemu runs the tests relatively fast within few minutes. 
>> The reason you see e.g. 12 mins
>> for some Qemu jobs comes from the timeout we set in Qemu scripts. We don't 
>> have yet the solution (we could
>> do the same as Qubes script) to detect the test success early and exit 
>> before timeout. That is why currently
>> the only way for Qemu tests to finish is by reaching the timeout.
>> So the problem is not with the rootfs and services (the improvement would 
>> not be significant) but with
>> the simulation being slow.
> 
> From my experience with the FVP improvement would be significant. A normal 
> boot distribution will start a lot of services. I end up to write my own 
> initscript doing the bare minimum for creating a guest. This saves me a lot 
> of time everytime I needed to test on FVP.

+1, I feel the same, but I've never done the time measurement though.

> I think we can do the same for the gitlab. Maybe not to the point of writing 
> your initscript but cutting down anything unnecessary.

Yeah I can try to play with removing some of the unnecessary services when 
preparing the rootfs
for Dom0 (see patch 4).

Kind regards,
Henry

> This will avoid the FVP test to become the bottlneck in the gitlab CI.
> 
> Chers,
> 
> -- 
> Julien Grall