Re: [PATCH 3/5] automation: Add the expect script with test case for FVP

[Henry Wang <Henry.Wang@xxxxxxx>]

Hi Stefano,

> On Dec 8, 2023, at 10:03, Henry Wang <Henry.Wang@xxxxxxx> wrote:
> 
> Hi Stefano,
> 
>> On Dec 8, 2023, at 09:38, Stefano Stabellini <sstabellini@xxxxxxxxxx> wrote:
>>> +set host_ip $expect_out(0,string)
>>> +
>>> +# Start the FVP and run the test
>>> +spawn bash -c "$runcmd"
>>> +
>>> +test_boot 2000 "$host_ip"
>>> +
>>> +send_user "\nExecution with SUCCESS\n"
>> 
>> Won't this always return SUCCESS even in case of failure?
> 
> IMHO, if things fails, we have various exit code (1-5) for each failure case. 
> For example,
> if the FVP port somehow cannot be found, we exit with error code 5. This will 
> basically lead
> us to the only case where the failure is caused by the script fails to wait 
> for the expected
> string/regexp, and this case we have the timeout failure triggered by my 
> above-mentioned
> expect_after block.

I did a test to see if I break the expect script by adding below hunk:
```
--- a/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
+++ b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
@@ -51,6 +51,7 @@ proc test_boot {{maxline} {host_ip}} {
     send -s "setenv serverip $host_ip; setenv ipaddr $host_ip; tftpb 
0x80200000 boot.scr; source 0x80200000\r"

     # Initial Xen boot
+    expect -re "this is a hack to break the build"
     expect -re "\(XEN\).*Freed .* init memory."

     # Dom0 and DomU
```
The timeout did happen in the expect script after the set timeout, see [1]

However the job still passes, and I believe this is caused by the shell script:
```
./automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp \
    
"/FVP/FVP_Base_RevC-2xAEMvA/Base_RevC_AEMvA_pkg/models/Linux64_armv8l_GCC-9.3/FVP_Base_RevC-2xAEMvA
 \
    -C bp.vis.disable_visualisation=1 \
    -C bp.ve_sysregs.exit_on_shutdown=1 \
    -C bp.secure_memory=0 \
    -C cache_state_modelled=0 \
    -C cluster0.has_arm_v8-4=1 \
    -C cluster1.has_arm_v8-4=1 \
    ${TERM0_CFG} ${TERM1_CFG} ${TERM2_CFG} ${TERM3_CFG} \
    ${VIRTIO_USER_NETWORK_CFG} \
    -C bp.secureflashloader.fname=$(pwd)/binaries/bl1.bin \
    -C bp.flashloader0.fname=$(pwd)/binaries/fip.bin" |& \
        tee smoke.serial

exit 0
```

The “|& tee smoke.serial” hides the error propagated by the expect script. I 
will send a v2 to fix it.

[1] 
https://gitlab.com/xen-project/people/henryw/xen/-/jobs/5708263782/artifacts/file/smoke.serial

Kind regards,
Henry

> 
> Kind regards,
> Henry
> 
>>> +exit 0
>>> -- 
>>> 2.25.1
>>> 
>