Re: [Xen-devel] xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Alex Bligh <alex@xxxxxxxxxxx>]

Ian,

--On 27 October 2011 10:27:08 +0100 Ian Campbell <Ian.Campbell@xxxxxxxxxx> 
wrote:

> > I am not avoiding answering your question (I will get you this) but
> > what is /meant/ to happen in the following scenario:
(now answered in another email) :-)

> Lets call this scenario A:
>
> > * Install on recent kernel (e.g. 2.6.37) running on Xen 3.3
> > * No fancy boot options, xen_emul_unplug not set
> > * No XEN_IOPORT_MAGIC implemented, so check_platform_magic()
> >   returns an error
> Correct, specifically it returns XEN_PLATFORM_ERR_MAGIC.
>
> > * Therefore xen_platform_pci_unplug=0
> Correct. There is some special casing around XEN_PLATFORM_ERR_MAGIC but
> lets assume xen_emul_unplug=unnecessary has not been passed to the
> kernel so it does not take effect.
>
> > * Therefore blkfront etc. don't init (probe returns
> >   -ENODEV)
> > * Therefore OS boots with root=/dev/sda
> sda must be an emulated IDE device since the 2.6.37 kernel does not
> support PV devices with names other than xvd*.
>
> I think this is as expected.
Yes, correct. So an in image created for xen 3.3 will have root=/dev/sda

> > Now Xen 3.3 is upgraded to Xen 4
> Lets call this scenario B:
>
> > * Kernel boots, and XEN_IOPORT_MAGIC now exists
> > * Therefore unplug occurs, and xen_platform_pci_unplug is non zero
> > * Therefore blkfront etc. inits, and PV drivers start
> > * OS still boots with root=/dev/sda
> >
> > Are the PV devices meant to appear as /dev/sdX rather than /dev/xvdX?
> No, there is no support for this in upstream kernels (in general all the
> old behaviours of Xen kernels where they would hijack other drivers
> device names are not upstreamble)
I thought so (from reading the code)

> > If so, how in the code does this happen? If not, won't the boot fail?
> You need to be using root=/dev/xvda here for it to work. Or even better
> use root=UUID=thing or root=LABEL=thong (many distros do one of these by
> default these days).
>
> (I'd forgotten about the UUID=/LABEL= option til just now -- that might
> be the bit of magic which was missing to make this work)
Right, so the same image that worked fine on Xen 3 will not work if
Xen 3 is upgraded to Xen 4 - it will require changes to the image.
It's not only root= on the command line, but also /etc/fstab (we have
partitioned disks in our images).

Sure, using UUIDs works, but unfortunately we have thousands of VMs run by 
users who, not possessing crystal balls, did not know the names of their 
block devices were to change :-) For completeness, there is a also a 
problem with certain versions of grub (pre grub2) where UUID config doesn't 
work;
whilst grub of course loads before the kernel and before the unplug, the
grub config file still references a device which has been unplugged, so
the next time grub is reconfigured, it can't find the disk to write to.

> > > Hmm, yes I think the special treatment of XEN_IOPORT_MAGIC mismatch on
> > > the kernel side is what I was missing.
> > >
> > > It might make sense to have a guest level config option which disables
> > > these magic ports, i.e. makes them return 0xffff like they would have
> > > done in 3.3 (I think 0xffff is what you'll get from an invalid port in
> > > general).
> >
> > Actually I don't think this will work. If we do this,
> > xen_plaftofm_pci_unplug will still be zero (as it's only set on exit
> > of the function after a successful unplug), and that's enough to
> > prevent blkfront and xenbus_probe_frontend from doing anything useful,
> > so will effectively disable PV drivers even where they should be enabled.
> Correct, this will take you back to scenario A, however if that is how
> the guest is configured (to use emulated devices) then this is what you
> wanted (or at least it is what the guest configuration is expecting).
>
> If the guest were configured to use PV devices then it would be using
> root=/dev/xvda. Such a configuration would have needed
> xen_emul_unplug=unnecessary in order to have worked on 3.3 before the
> upgrade and this option would be harmless but unnecessary on 4.1.
Yes.

The trouble is some guests are running Windows, some old xenlinux
kernels, and some new kernels that talk to XEN_IOPORT_MAGIC. I need
one config that works for all of them, as we have no way of telling
them apart.

If I disable XEN_IOPORT_MAGIC then new installs of newer operating
systems will not run with PV drivers. I agree that these won't
run (without tweaked kernel boot lines) on Xen 3.3 with PV drivers
either, but I don't want to be stuck this way for ever.

I /think/ what you are saying is that the on new kernels that talk
to XEN_IOPORT_MAGIC, they will use /dev/sdX (emulated disks). If
we upgrade to Xen4, these will all break, because we can't distinguish
them from the behaviour of a new kernel doing a new install, where
we want the unplug to occur. The fix is either to fix the root
device, or to pass xen_emul_unplug=unnecessary in which case it should
continue to use to the emulated devices.

It would have been really nice to have been able to pass something
through XEN_IOPORT_MAGIC to effectively cause the same effect as
xen_emul_unplug=unnecessary on the command line, but I don't see how
that can be done with code already in the wild - unless I'm wrong?

> If you were using UUID=/LABEL= then I think things would have worked in
> both cases (emulated on 3.3 and pv on 4.1) without additional kernel
> parameters.
>
> One detail worth mentioning is that if the guest is using PV drivers and
> expecting xvd* named devices then prior to the unplug the devices
> xvd[a-d] are also exposed as the IDE devices hd[a-d]. This is how the
> bootloader is still able to access the emulated device.
Sure

--
Alex Bligh

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel