Re: [Xen-devel] Xen-unstable staging build broken by pvshim patches.

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 14/08/2019 00:16, Sander Eikelenboom wrote:
> On 13/08/2019 23:05, Andrew Cooper wrote:
>> On 13/08/2019 22:03, Sander Eikelenboom wrote:
>>> On 13/08/2019 15:31, Andrew Cooper wrote:
>>>> On 13/08/2019 12:51, Sander Eikelenboom wrote:
>>>>> On 13/08/2019 13:21, Andrew Cooper wrote:
>>>>>> On 09/08/2019 00:28, Sander Eikelenboom wrote:
>>>>>>> On 09/08/2019 00:44, Andrew Cooper wrote:
>>>>>>>> On 08/08/2019 23:34, Sander Eikelenboom wrote:
>>>>>>>>> On 08/08/2019 23:14, Andrew Cooper wrote:
>>>>>>>>>> On 08/08/2019 22:16, Sander Eikelenboom wrote:
>>>>>>>>>>> On 08/08/2019 23:05, Andrew Cooper wrote:
>>>>>>>>>>>> On 08/08/2019 21:59, Sander Eikelenboom wrote:
>>>>>>>>>>>>> Hi Andrew,
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems the pvshim patches in xen-unstable staging break the 
>>>>>>>>>>>>> build on my machine.
>>>>>>>>>>>>> I cloned a fresh tree to be sure, haven't checked which of the 
>>>>>>>>>>>>> two commits causes it:
>>>>>>>>>>>>> 060f4eee0fb408b316548775ab921e16b7acd0e0 or 
>>>>>>>>>>>>> 32b1d62887d01f85f0c1d2e0103f69f74e1f6fa3
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Sander
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [ -d //usr/local/lib/xen/boot ] || 
>>>>>>>>>>>>> /usr/src/new/xen-unstable/tools/firmware/../../tools/cross-install
>>>>>>>>>>>>>  -d -m0755 -p //usr/local/lib/xen/boot
>>>>>>>>>>>>> [ -d //usr/local/lib/debug/usr/local/lib/xen/boot ] || 
>>>>>>>>>>>>> /usr/src/new/xen-unstable/tools/firmware/../../tools/cross-install
>>>>>>>>>>>>>  -d -m0755 -p //usr/local/lib/debug/usr/local/lib/xen/boot
>>>>>>>>>>>>> [ ! -e hvmloader/hvmloader ] || 
>>>>>>>>>>>>> /usr/src/new/xen-unstable/tools/firmware/../../tools/cross-install
>>>>>>>>>>>>>  -m0644 -p hvmloader/hvmloader //usr/local/lib/xen/boot
>>>>>>>>>>>>> /usr/src/new/xen-unstable/tools/firmware/../../tools/cross-install
>>>>>>>>>>>>>  -m0644 -p seabios-dir/out/bios.bin 
>>>>>>>>>>>>> //usr/local/lib/xen/boot/seabios.bin
>>>>>>>>>>>>> /usr/src/new/xen-unstable/tools/firmware/../../tools/cross-install
>>>>>>>>>>>>>  -m0644 -p xen-dir/xen-shim //usr/local/lib/xen/boot/xen-shim
>>>>>>>>>>>>> install: cannot stat 'xen-dir/xen-shim': No such file or directory
>>>>>>>>>>>>> make[4]: *** [Makefile:52: install] Error 1
>>>>>>>>>>>>> make[4]: Leaving directory 
>>>>>>>>>>>>> '/usr/src/new/xen-unstable/tools/firmware'
>>>>>>>>>>>>> make[3]: *** 
>>>>>>>>>>>>> [/usr/src/new/xen-unstable/tools/../tools/Rules.mk:237: 
>>>>>>>>>>>>> subdir-install-firmware] Error 2
>>>>>>>>>>>>> make[3]: Leaving directory '/usr/src/new/xen-unstable/tools'
>>>>>>>>>>>>> make[2]: *** 
>>>>>>>>>>>>> [/usr/src/new/xen-unstable/tools/../tools/Rules.mk:232: 
>>>>>>>>>>>>> subdirs-install] Error 2
>>>>>>>>>>>>> make[2]: Leaving directory '/usr/src/new/xen-unstable/tools'
>>>>>>>>>>>>> make[1]: *** [Makefile:73: install] Error 2
>>>>>>>>>>>>> make[1]: Leaving directory '/usr/src/new/xen-unstable/tools'
>>>>>>>>>>>>> make: *** [Makefile:131: install-tools] Error 2
>>>>>>>>>>>> That's weird.
>>>>>>>>>>>>
>>>>>>>>>>>> Do you have the full log?  The real failure was somewhere earlier 
>>>>>>>>>>>> where
>>>>>>>>>>>> xen-shim didn't get started.
>>>>>>>>>>>>
>>>>>>>>>>>> ~Andrew
>>>>>>>>>>>>
>>>>>>>>>>> Hmm if forgot and thus forgot to mention my build script disables 
>>>>>>>>>>> some stuff:
>>>>>>>>>>> ./configure --disable-qemu-traditional --disable-stubdom 
>>>>>>>>>>> --disable-docs --disable-rombios
>>>>>>>>>>>
>>>>>>>>>>> Could be that one of those doesn't work anymore.
>>>>>>>>>> The only interesting one would be --disable-rombios, which does make
>>>>>>>>>> changes in this area of the build, but everything I changed was 
>>>>>>>>>> inside
>>>>>>>>>> the xen-dir/ directory so shouldn't interact.>
>>>>>>>>>> ~Andrew
>>>>>>>>>>
>>>>>>>>> It indeed seems to be some interaction with --disable-rombios, with 
>>>>>>>>> just
>>>>>>>>> a plain ./configure it builds fine.
>>>>>>>>> Logs when building with --disable-rombios are attached.
>>>>>>>> Right.  So the build itself works, but the subsequent `make install` 
>>>>>>>> fails.
>>>>>>>>
>>>>>>>> And to confirm, a build of 8d54a6adf (the parent of my first shim
>>>>>>>> commit) works entirely fine?
>>>>>>>>
>>>>>>>> ~Andrew
>>>>>>>>
>>>>>>> Just rechecked, and yes that builds and installs fine (with 
>>>>>>> --disable-rombios).
>>>>>> Which base distro are you using?  I'm unable to reproduce any build
>>>>>> failures locally.
>>>>>>
>>>>>> ~Andrew
>>>>>>
>>>>> Debian 10 / Buster.
>>>> Do you have your full build script available, and is it built fully from
>>>> clean?
>>>>
>>>> How beefy is your build machine?  From the logs it is clearly a parallel
>>>> build but I don't see an explicit -j in the logs.
>>>>
>>>> I still cant reproduce this, even in a buster container.
>>>>
>>>> ~Andrew
>>>>
>>> The machine is not that beefy, but a six core AMD, but no OOMs or anything.
>>>
>>> The script is basically just and some changing of dirs:
>>> make clean && ./configure --disable-qemu-traditional --disable-stubdom 
>>> --disable-docs --disable-rombios && make -j6 && make -j6 install
>>>
>>> I tried some variants just plain from the command line without any scripts:
>>> After a fresh clone of current xen-unstable staging branch.
>>>
>>> Fails:    make clean && ./configure --disable-rombios && make -j6 && make 
>>> -j6 install
>>> Fails:    make clean && ./configure --disable-rombios && make -j4 && make 
>>> -j4 install
>>> Fails:    make clean && ./configure --disable-rombios && make -j2 && make 
>>> -j2 install
>>> Succeeds: make clean && ./configure --disable-rombios && make && make 
>>> install
>>>
>>> Succeeds: make clean && ./configure && make -j2 && make -j2 install
>> What about j4 and j6 of these?
> Urghh I *thought* i have seen j6 work, but the first time rechecking it 
> didn't, second time it did.
> j4 worked first time while rechecking.
>
>>> So to recap:
>>>     Some how on my systeem (Debian buster) with my build tools a parallel 
>>> build with --disable-rombios fails.
>>>     A non parallel build with --disable-rombios succeeds.
>>>     And reverting the "pv shim" patches makes it build again in parallel 
>>> with --disable-rombios.
>>>
>>> And last but not least it seems to work on your systems with your build 
>>> tools. 
>>>
>>> So I would end up with either:
>>>    - a difference in build tools ?
>>>    - and/or some unexpected dependency in the code that can go wrong timing 
>>> wise in a parallel build ?
>> It must be something timing dependent, or I would expect to have
>> reproduced it with your instructions so far.  (Frankly, by this point,
>> I'd also expect someone else to have stumbled over it as well.)
>>
>> It is almost certainly something which isn't quite parallel-safe in
>> tools/firmware, but I'm beginning to suspect that rombios is a bit of a
>> red herring and all it is doing is skewing the timing enough to cause
>> the opposite behaviour.
> I think the above confirms this suspicion.

It seems OSSTest has seen this once.

http://logs.test-lab.xenproject.org/osstest/logs/140088/build-amd64/6.ts-xen-build.log

It is clearly intermittent.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel