Re: [Xen-devel] [PATCHv3] QEMU(upstream): Disable xen's use of O_DIRECT by default as it results in crashes.

[Alex Bligh <alex@xxxxxxxxxxx>]

Paolo,

--On 18 March 2013 15:49:48 +0100 Paolo Bonzini <pbonzini@xxxxxxxxxx> wrote:

> > > The problem is that the target's page cache might host image data from a
> > > previous run.  If you do not use O_DIRECT, it will not see the changes
> > > made on the source.
> > I was under the impression that with cache=writeback, qemu doesn't
> > use O_DIRECT, in which case why isn't there the same danger under
> > kvm, i.e. that the target page cache contains data from a previous
> > run?
> KVM in fact only supports migration using cache=none.  This does not
> apply of course if you're using cache-coherent storage, such as rbd or
> gluster; or if you're using one of the userspace backends that bypass
> the page cache, like NBD or libiscsi.
Well right now emulated devices under xen with qemu-upstream
dm do not use O_DIRECT. So that's going to hit this problem anyway.

Let me understand the problem a little more. Let's say a VM has a
disk backed by NFS. It runs on node A, at which point pages are
introduced to the page cache. It then migrates to node B, with
node A flushing its dirty pages to disk before the migration
completed, but not emptying the page cache completely.
It's then migrated back to node A in which case I think you are
saying that node A may still hold the page cache entries from
prior to the first migration.

However, node A has closed and reopened the file. Doesn't NFSv4
notice that the file has changed at this point and invalidate
all page cache entries? Else (re)opening a file on one NFS client
that has been changed on another would never work. I know this
is unpredictable if you haven't closed the file, but in this
instance A has closed the file.

> > Disabling migration seems a bit excessive when migration isn't disabled
> > with cache=unsafe (AFAIK)
> It is not QEMU's task.  There are cases where the cache= options are
> unnecessary or ignored.  But indeed libvirt warns (or blocks, I don't
> remember) in this case.
Well we are kvm user too, and I understand hat kvm (as opposed to
libvirt) does not error or warn if you live migrate with cache=writeback.

I've no problem if xl or libvirt or whatever error or warn. My usage
is API based, rather than xl / libvirt based.

> If libxl does migration without O_DIRECT, then that's a bug in libxl.
> What about blkback?  IIRC it uses bios, so it also bypasses the page
> cache.
Possibly a bug in xl rather than libxl, but as no emulated devices
use O_DIRECT, that bug is already there, and isn't in QEMU.

> > Stefano did ack the patch, and for a one line change it's been
> > through a pretty extensive discussion on xen-devel ...
> It may be a one-line change, but it completely changes the paths that
> I/O goes through.  Apparently the discussion was not enough.
What would you suggest?

--
Alex Bligh

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel