[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] çåïlive-migration restore failed error

åääïAndrew Cooper <andrew.cooper3@xxxxxxxxxx>
åéæéï2014å9æ15æ(ææä) 22:01
æääïåæé(äæ) <huaixin.chx@xxxxxxxxxxxxxxx>ïkeir <keir@xxxxxxx>ïIan.Campbell <Ian.Campbell@xxxxxxxxxx>ïstefano.stabellini <stefano.stabellini@xxxxxxxxxxxxx>ïxen-devel <xen-devel@xxxxxxxxxxxxxxxxxxx>ïIan.Jackson <Ian.Jackson@xxxxxxxxxxxxx>ïgeorge.dunlap <george.dunlap@xxxxxxxxxxxxx>
æãéïååæ(åè) <jinsong.liu@xxxxxxxxxxxxxxx>
äãéïRe: live-migration restore failed error

On 15/09/2014 10:41, åæé(äæ) wrote:
We are working on live-migration based on Xen-4.0.1(For history reason, and meantime we are upgrading our Xen to very latest version). Restore failed when live migrating ubuntu12.04 on xen-4.0.1. To be more specific, error occurred when populating memory. Error messages are as follow:

[2014-09-12 22:40:40 7331 1189091648] DEBUG (XendCheckpoint:307) [xc_restore]: /usr/lib64/xen/bin/xc_restore 4 2763 3 4 1 1 1 0
[2014-09-12 22:40:40 7331 1189091648] DEBUG (XendCheckpoint:428) Thread-40188
[2014-09-12 22:40:40 7331 1172306240] INFO (XendCheckpoint:476) Thread-40188:xc_domain_restore start: p2m_size = fefff
[2014-09-12 22:40:40 7331 1172306240] INFO (XendCheckpoint:476) Thread-40188:Reloading memory pages:   0%
[2014-09-12 22:40:50 7331 1172306240] INFO (XendCheckpoint:476) Thread-40188:Failed allocation for dom 2763: 128 extents of order 0
[2014-09-12 22:40:50 7331 1172306240] INFO (XendCheckpoint:476) Thread-40188:ERROR Internal error: Failed to allocate memory for batch.!
[2014-09-12 22:40:50 7331 1172306240] INFO (XendCheckpoint:476) Thread-40188:
[2014-09-12 22:40:50 7331 1172306240] INFO (XendCheckpoint:476) Thread-40188:Restore exit with rc=1
[2014-09-12 22:40:50 7331 1189091648] DEBUG (XendCheckpoint:462) /usr/lib64/xen/bin/xc_restore 4 2763 3 4 1 1 1 0 failed status 256
[2014-09-12 22:40:50 7331 1189091648] DEBUG (XendDomainInfo:3845) XendDomainInfo.destroy: domid=2763

In this case, populate_physmap terminated with nr_done 127.  So xc_memory_op return 127 while nr_extents equals 128.

This problem happends once every 1770th live migration or so. As I am debugging this issue, I'm sending this email to ask for suggestions on this issue.

Huaixin Chang

Xen is unable to fulfil the allocation request.  You have run out of host memory.


Here are some more clues.

I'm migrating ubuntu12.04(with 1G or 512M memory) on two machines with around 96G of memory, back and forth. The issue occurs around 1770 times of migration erery time, whether guest memory is 512M or 1G.

In the pasted xend log, a request of 128 pages of non-contiguous memory failed. Currently, I am conducting another round of migration test, which has completed 230 times of migration, and hopefully will terminate after about one day. So far, I do not see a major decrese of hypervisor memory. I will check whether there is memory issues when the problem shows up.
total_memory           : 98276
free_memory            : 84454

Sorry for not being able to provide a hypervisor log at the moment. Previously I printed too many of messages, most of them were suppressed and no helpful message could be found. I will also check whether this round will help. 

Huaixin Chang

Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.