Re: [PATCH v2 3/5] automation: Add the expect script with test case for FVP

[Henry Wang <Henry.Wang@xxxxxxx>]

Hi Michal,

> On Dec 8, 2023, at 17:50, Michal Orzel <michal.orzel@xxxxxxx> wrote:
> On 08/12/2023 10:21, Henry Wang wrote:
>> 
>> 
>> Hi Michal,
>> 
>>> On Dec 8, 2023, at 17:11, Michal Orzel <michal.orzel@xxxxxxx> wrote:
>>> On 08/12/2023 10:05, Henry Wang wrote:
>>>> 
>>>> Hi Michal,
>>>> 
>>>>> On Dec 8, 2023, at 16:57, Michal Orzel <michal.orzel@xxxxxxx> wrote:
>>>>> 
>>>>> Hi Henry,
>>>>> 
>>>>> On 08/12/2023 06:46, Henry Wang wrote:
>>>>>> diff --git a/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp 
>>>>>> b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
>>>>>> new file mode 100755
>>>>>> index 0000000000..25d9a5f81c
>>>>>> --- /dev/null
>>>>>> +++ b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
>>>>>> @@ -0,0 +1,73 @@
>>>>>> +#!/usr/bin/expect
>>>>>> +
>>>>>> +set timeout 2000
>>>>> Do we really need such a big timeout (~30 min)?
>>>>> Looking at your test job, it took 16 mins (quite a lot but I know FVP is 
>>>>> slow
>>>>> + send_slow slows things down)
>>>> 
>>>> This is a really good question. I did have the same question while working 
>>>> on
>>>> the negative test today. The timeout 2000 indeed will fail the job at 
>>>> about 30min,
>>>> and waiting for it is indeed not really pleasant.
>>>> 
>>>> But my second thought would be - from my observation, the overall time now
>>>> would vary between 15min ~ 20min, and having a 10min margin is not that 
>>>> crazy
>>>> given that we probably will do more testing from the job in the future, 
>>>> and if the
>>>> GitLab Arm worker is high loaded, FVP will probably become slower. And 
>>>> normally
>>>> we don’t even trigger the timeout as the job will normally pass. So I 
>>>> decided
>>>> to keep this.
>>>> 
>>>> Mind sharing your thoughts about the better value of the timeout? Probably 
>>>> 25min?
>>> From what you said that the average is 15-20, I think we can leave it set 
>>> to 30.
>>> But I wonder if we can do something to decrease the average time. ~20 min 
>>> is a lot
>>> even for FVP :) Have you tried setting send_slow to something lower than 
>>> 100ms?
>>> That said, we don't send too many chars to FVP, so I doubt it would play a 
>>> major role
>>> in the overall time.
>> 
>> I agree with the send_slow part. Actually I do have the same concern, here 
>> are my current
>> understanding and I think you will definitely help with your knowledge:
>> If you check the full log of Dom0 booting, for example [1], you will find 
>> that we wasted so
>> much time in starting the services of the OS (modloop, udev-settle, etc). 
>> All of these services
>> are retried many times but in the end they are still not up, and from my 
>> understanding they
>> won’t affect the actual test(?) If we can somehow get rid of these services 
>> from rootfs, I think
>> we can save a lot of time.
>> 
>> And honestly, I noticed that qemu-alpine-arm64-gcc suffers from the same 
>> problem and it also
>> takes around 15min to finish. So if we managed to tailor the services from 
>> the filesystem, we
>> can save a lot of time.
> That is not true. Qemu runs the tests relatively fast within few minutes. The 
> reason you see e.g. 12 mins
> for some Qemu jobs comes from the timeout we set in Qemu scripts. We don't 
> have yet the solution (we could
> do the same as Qubes script) to detect the test success early and exit before 
> timeout. That is why currently
> the only way for Qemu tests to finish is by reaching the timeout.
> 
> So the problem is not with the rootfs and services (the improvement would not 
> be significant) but with
> the simulation being slow. That said, this is something we all know and I 
> expect FVP to only be used in scenarios
> which cannot be tested using Qemu or real HW.

Ok, you made a point. Let me do some experiments to see if I can improve. 
Otherwise maybe
we can live with this until a better solution.

Kind regards,
Henry

> 
> ~Michal