Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough

[Atom2 <ariel.atom2@xxxxxxxxxx>]

Ian, as always - thanks for your reply.
Am 17.03.14 11:00, schrieb Ian Campbell:
> On Sun, 2014-03-16 at 15:41 +0100, Atom2 wrote:
> > Hi list,
> > I am experiencing a substantial delay if I shutdown a PV linux guest
> > which has PCI devices passed-through:
> >
> > I start the domain using
> >         xl create -c domain
> >
> > Then I login from the connected console and do a
> >         shutdown -h now
> > which results in normal shutdown messages (w/o errors) showing on the
> > console screen. The last line of messages then reads:
> >         [   36.535466] reboot: System halted
> >
> > So all seems normal and as expected; the strange thing, however, is that
> > the prompt in dom0 only returns after a substantial amount of time which
> > appears to be _proportional_ to the number of PCI devices passed
> > through: For every device specified in the
> >         pci = [ bb:dd.f, ... ]
> > entry in the guest configuration file the delay is 10 seconds. In other
> > words for 4 PCI devices passed through, it takes 40 seconds before the
> > prompt in dom0 returns (though I did not make any test with more than 4
> > devices).
> >
> > If I issue the command
> >         xl list
> > from an other terminal connected to dom0 while the delay is on-going it
> > shows that the domain in question remains in the state 's', which
> > according to the man page indicates that "the guest OS has shut down
> > (SCHEDOP_shutdown has been called) but the domain is not dying yet".
> > This state does not change until the prompt in dom0 returns (at least
> > repeated 'xl list' commands show the same output) and then the line for
> > the domain is gone.
> >
> > If no devices are passed through, the prompt in dom0 after having shut
> > down the domU appears without any noticeable delay.
> >
> > All tests have been performed using the same environment while only
> > changing the relevant line in the domU configuration file.
> >
> > The xen version used is 4.3.1-r5, xen-tools is also 4.3.1-r5 and the
> > kernel used for both dom0 and domUs is 3.11.7 (hardende gentoo sources).
> Any chance you could try 4.3.2, or even 4.4.0?
Unfortunately neither of these versions are currently available as 
stable ebuilds for my distribution, but I assume it shouldn't be long 
before there's some movement.
> > The system is capable of vt-d and uses a Xeon E3-1260L processor.
> >
> > Do these observations ring a bell with anybody or is this even expected
> > behaviour. If this is not normal - which I would expect as I have not
> > been able to find any information relating to substantial delays during
> > shutdown - how would I go about getting to the grounds of this?
> My guess would be that xl process which is managing the domain destroy
> is waiting for something (perhaps pciback) to confirm shutdown for each
> device and this is timing out in series, leading to the delays. You
> might find something in the logs /var/log/xen pointing to something like
> this.
>
> If not then if you start the guest with "xl -vvv create -F <cfg>" then
> the xl process which is monitoring the domain will stay in the
> foreground and be logging to stdout (I think). If you then issue the
> shutdown from another shell perhaps there will be some obvious gaps in
> the logs as things shutdown which might help.
That worked and there also was some output - please find the log from 
start to finnish attached to this mail. I have marked various points in 
the log: First the point where the startup was done and the domU was 
live and secondly those 4 points in time (or rather output) where the 
10s delay occured.
BTW: I don't know whether it makes any difference, but I am only using 
xen-pciback.hide=(bb:dd.f)(...) on the grub command line for a number of 
devices including those that I pass through to this domU - there's 
nothing else happening in the dom0 with those devices priot to starting 
the domU and there are also no driver modules available for any of the 
hidden hardware (except for one of the hidden USB Controllers of the 
motherboard which is also passed through) in dom0.
> Ian.

Attachment: xl-output
Description: Text document

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users