Re: [Xen-devel] reboot driver domain, vifX.Y = NO-CARRIER?

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 27/04/18 16:03, Jason Cooper wrote:
> All,
>
> On Gentoo Xen 4.9.1, I've been creating minimal Linux DomU's to create a
> virtual, segregated network infrastructure.  This has been going really
> well, and I'm slowly progressing toward a self-updating system.
>
> My main snag has to do with re-attaching VMs to a driver domain after
> rebooting the driver domain.  e.g.
>
>
>                               +-----+
>                             /-| VM1 |
>                            /  +-----+
>        +----+   +-------+ /   +-----+
> ISP ---| SW |---| GW/FW |-----| VM2 |
>        +----+   +-------+ \   +-----+
>         DD        DD       \  +-----+
>                             \-| VMN |
>                               +-----+
>
> So, in this diagram, SW, GW/FW, and VM1 are mini-VMs.  VM2, and the rest
> are full fledged Linux PV VMs.
>
> Only SW, and GW/FW are driver domains.  SW has the physical nic via
> pci-passthrough.  There are actually 7 GW/FW mini-VMs (for 7 public IPs,
> and 7 different networks), and a trunk mini-VM that aren't shown.
>
> The problem occurs when I reboot a driver domain.  Regardless of the
> type of guest attached to it, I'm unable to re-establish connectivity
> between the driver domain and the re-attached guest.  e.g. I reboot
> GW/FW, then re-attach VM1, VM2 and the rest.  No matter how I do it, I
> get:
>
> $ ip link
> ...
> 11: vif20.1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master 
> br10 qlen 32
>     link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
>
> In the driver domain.  At this point, absolutely no packets flow between
> the two VMs.  Not even ARP.  The only solution, so far, is to unnecessarily
> reboot the PV guests.  After that, networking is fine.
>
> Any thoughts?

XenServer found this when we investigated using device driver domains in
a similar way.

The underlying problem is that the frontend/backend setup in xenstore
encodes the domid in path, and changing that isn't transparent to the
guest at all.

The best idea we came up with was to reboot the driver domain and reuse
its old domid, at which point all the xenstore paths would remain
valid.  There is support in Xen for explicitly choosing the domid of a
domain, but I don't think that it is wired up sensibly in xl.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel