[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] null domains after xl destroy



On 2017-05-16 10:49, Glenn Enright wrote:
On 15/05/17 21:57, Juergen Gross wrote:
On 13/05/17 06:02, Glenn Enright wrote:
On 09/05/17 21:24, Roger Pau Monné wrote:
On Mon, May 08, 2017 at 11:10:24AM +0200, Juergen Gross wrote:
On 04/05/17 00:17, Glenn Enright wrote:
On 04/05/17 04:58, Steven Haigh wrote:
On 04/05/17 01:53, Juergen Gross wrote:
On 03/05/17 12:45, Steven Haigh wrote:
Just wanted to give this a little nudge now people seem to be
back on
deck...

Glenn, could you please give the attached patch a try?

It should be applied on top of the other correction, the old debug
patch should not be applied.

I have added some debug output to make sure we see what is happening.

This patch is included in kernel-xen-4.9.26-1

It should be in the repos now.


Still seeing the same issue. Without the extra debug patch all I see in
the logs after destroy is this...

xen-blkback: xen_blkif_disconnect: busy
xen-blkback: xen_blkif_free: delayed = 0

Hmm, to me it seems as if some grant isn't being unmapped.

Looking at gnttab_unmap_refs_async() I wonder how this is supposed to
work:

I don't see how a grant would ever be unmapped in case of
page_count(item->pages[pc]) > 1 in __gnttab_unmap_refs_async(). All it does is deferring the call to the unmap operation again and again. Or
am I missing something here?

No, I don't think you are missing anything, but I cannot see how this
can be
solved in a better way, unmapping a page that's still referenced is
certainly
not the best option, or else we risk triggering a page-fault elsewhere.

IMHO, gnttab_unmap_refs_async should have a timeout, and return an
error at
some point. Also, I'm wondering whether there's a way to keep track of
who has
references on a specific page, but so far I haven't been able to
figure out how
to get this information from Linux.

Also, I've noticed that __gnttab_unmap_refs_async uses page_count,
shouldn't it
use page_ref_count instead?

Roger.


In case it helps, I have continued to work on this. I notices processed
left behind (under 4.9.27). The same issue is ongoing.

# ps auxf | grep [x]vda
root      2983  0.0  0.0      0     0 ?        S    01:44   0:00  \_
[1.xvda1-1]
root      5457  0.0  0.0      0     0 ?        S    02:06   0:00  \_
[3.xvda1-1]
root      7382  0.0  0.0      0     0 ?        S    02:36   0:00  \_
[4.xvda1-1]
root      9668  0.0  0.0      0     0 ?        S    02:51   0:00  \_
[6.xvda1-1]
root     11080  0.0  0.0      0     0 ?        S    02:57   0:00  \_
[7.xvda1-1]

# xl list
Name                              ID   Mem VCPUs      State   Time(s)
Domain-0                          0  1512     2     r-----     118.5
(null)                            1     8     4     --p--d      43.8
(null)                            3     8     4     --p--d       6.3
(null)                            4     8     4     --p--d      73.4
(null)                            6     8     4     --p--d      14.7
(null)                            7     8     4     --p--d      30

Those all have...

[root 11080]# cat wchan
xen_blkif_schedule

[root 11080]# cat stack
[<ffffffff814eaee8>] xen_blkif_schedule+0x418/0xb40
[<ffffffff810a0555>] kthread+0xe5/0x100
[<ffffffff816f1c45>] ret_from_fork+0x25/0x30
[<ffffffffffffffff>] 0xffffffffffffffff

And found another reference count bug. Would you like to give the
attached patch (to be applied additionally to the previous ones) a try?


Juergen


This seems to have solved the issue in 4.9.28, with all three patches
applied. Awesome!

On my main test machine I can no longer replicate what I was
originally seeing, and in dmesg I now see this flow...

xen-blkback: xen_blkif_disconnect: busy
xen-blkback: xen_blkif_free: delayed = 1
xen-blkback: xen_blkif_free: delayed = 0

xl list is clean, xenstore looks right. No extraneous processes left over.

Thankyou Juergen, so much. Really appreciate your persistence with
this. Anything I can do to help push this upstream please let me know.
Feel free to add a reported-by line with my name if you think it
appropriate.

This is good news.

Juergen, Can I request a full patch set posted to the list (plz CC me) - and I'll ensure we can build the kernel with all 3 (?) patches applied and test properly.

I'll build up a complete kernel with those patches and give a tested-by if all goes well.

--
Steven Haigh

Email: netwiz@xxxxxxxxx
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.