dom0less vs xenstored setup race Was: xen | Failed pipeline for staging | 6a47ba2f

[andrew.cooper3@xxxxxxxxxx]

Hello,

After what seems like an unreasonable amount of debugging, we've tracked
down exactly what is going wrong here.

https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/4219721944

Of note is the smoke.serial log around:

io: IN 0xffff90fec250 d0 20230503 14:20:42 INTRODUCE (1 233473 1 )
obj: CREATE connection 0xffff90fff1f0
*** d1 CONN RESET req_cons 00000000, req_prod 0000003a rsp_cons
00000000, rsp_prod 00000000
io: OUT 0xffff9105cef0 d0 20230503 14:20:42 WATCH_EVENT
(@introduceDomain domlist )

XS_INTRODUCE (in C xenstored at least, not checked O yet) always
clobbers the ring pointers.  The added pressure on dom0 that the
xensconsoled adds with it's 4M hypercall bounce buffer occasionally
defers xenstored long enough that the XS_INTRODUCE clobbers the first
message that dom1 wrote into the ring.

The other behaviour seen was xenstored observing a header looking like this:

*** d1 HDR { ty 0x746e6f63, rqid 0x2f6c6f72, txid 0x74616c70, len
0x6d726f66 }

which was rejected as being too long.  That's "control/platform" in
ASCII, so the XS_INTRODUCE intersected dom1 between writing the header
and writing the payload.


Anyway, it is buggy for XS_INTRODUCE to be called on a live an
unsuspecting connection.  It is ultimately init-dom0less's fault for
telling dom1 it's good to go before having waited for XS_INTRODUCE to
complete.

I am going to start by correcting the documentation to make these
details clear, and then figure out what is the best set of steps to
unbreak this.

~Andrew