Re: [Xen-devel] reboot driver domain, vifX.Y = NO-CARRIER?

[Jason Andryuk <jandryuk@xxxxxxxxx>]

Hi Jason,

On Sun, May 6, 2018 at 11:45 AM, Jason Cooper <xen@xxxxxxxxxxxxxx> wrote:
> Hi Marek,
>
> On Sat, May 05, 2018 at 01:03:15AM +0200, Marek Marczykowski-Górecki wrote:
>> On Fri, May 04, 2018 at 06:13:25PM -0400, Rich Persaud wrote:
>> > > On May 1, 2018, at 08:53, Jason Cooper <xen@xxxxxxxxxxxxxx> wrote:
>> > >
>> > > add the link to xen-users thread of me talking to myself.  :-))
>> > >
>> > >> On Tue, May 01, 2018 at 12:37:51PM +0000, Jason Cooper wrote:
>> > >> When I was first digging into this, I started a thread on xen-users [1],
>> > >> I've attached my xl-reboot.sh script here so you can see exactly what
>> > >> I'm attempting to do:
>> > >
>> > > [1] https://marc.info/?l=xen-users&m=152389443206023&w=2
>> >
>> > You may want to look at the code (toolstack and/or frontend-backend
>> > drivers) for Qubes and OpenXT, both of which use network driver
>> > domains and support wired/wireless networks.
>> >
>> > Operational restart of a measured, non-persistent driver domain
>> > (instead of host) is a benefit of Xen disaggregation architectures.
>>
>> In Qubes, on backend restart, we do equivalent of xl network-detach &&
>> xl network-attach (as you do in xl-reboot.sh). xl itself doesn't provide
>> any place to plug such script, but we use libvirt which provide events.
>> Also, we have full control over domain config (libvirt XML), so don't
>> need to extract vif list from xenstore...

OpenXT does the xl network-detach && xl network-attach in its own
daemon: https://github.com/OpenXT/network/blob/master/nwd/Main.hs#L767

>> The problem you describe looks related to
>> https://lkml.org/lkml/2018/2/28/289, but fix is included in 4.16...
>> There was also related libxl patch:
>> https://xen.markmail.org/thread/6qbgmwyjqsshjus7
>> (but it applies to the case where you first shutdown backend and only
>> then do xl network-detach)
>>
>> Do you have xl devd running in your driver domain? Without that xl
>> network-attach wont work (AFAIR udev isn't used here anymore).
>
> Yes, I've now modified the init script (xendomains in Gentoo) to create
> a key /tool/vmstatus/$domname/status, start the domU, loop until it gets
> it's domid, and -chmod the key.  It then does a -watch on that key.  In
> the domU, *after* xl devd is started, it writes "online" to that key.
>
> This allows me to automatically bring up the driver domains, and make
> sure they're ready for connections before proceeding to booting the next
> VM.  This only occurs when the host boots.
>
> After the driver domains are up, the rest of the domains are started in
> parallel.
>
>> Also note that backend shutdown/restart/crash was a source of many
>> problems in frontend kernel and toolstack in the past. Even simple
>> dynamic network-attach/detach sometimes is problematic for the frontend.
>> Links:
>> https://github.com/QubesOS/qubes-issues/issues/3657 (frontend kernel
>> problem)
>> https://github.com/QubesOS/qubes-issues/issues/1426 (toolstack problem,
>> + libvirt)
>> https://github.com/QubesOS/qubes-issues/issues/975 (frontend kernel
>> problem)
>
> Mmm, clearly the state machine and it's implementation needs some
> review.  I'm building v4.16.7 and we'll see how it goes for my usecase.

OpenXT has some patches for reconnecting netfront after the netback
domain is rebooted to a new domid:
https://github.com/OpenXT/xenclient-oe/blob/master/recipes-kernel/linux/4.14/patches/netfront-support-backend-relocate.patch
https://github.com/OpenXT/xenclient-oe/blob/master/recipes-kernel/linux/4.14/patches/xenbus-move-otherend-watches-on-relocate.patch

I'm too familiar with those, so they may be specific to the OpenXT
networking code.

Jason, when you see the vif NO-CARRIER, how do the frontend and
backend XenStore entries look?  Do the domids matchup and is the pair
in state 4 -> XenbusStateConnected?

Regards,
Jason

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel