[Xen-users] XEN 3.1: critical bug: vif init failure after creating 15-17 VMs (XENBUS: Timeout connecting to device: device/vif)

[Eric Tessler <maiden1134@xxxxxxxxx>]

 

We have found a critical problem with the XEN 3.1 release (for those who are running 15-20 VMs on a single server). We are using the official XEN 3.1 release on a rackable server (Dual-Core AMD Opteron, 8GB RAM).

 

The problem we are seeing is that intermittently vifs fail to work properly in VMs after we create around 15-17 VMs on our server (all running at the same time, created one by one). Sometimes we can create up to 40 VMs w/o a problem, other times vifs begin to fail on the 15th-17th VM (each VM has 4 vifs, 1 block device, 64MB memory), we see the following error message in the VM (domU) on its console:

"XENBUS: Timeout connecting to device: device/vif/3 (state 6)"

 

At the same time in dom0, we see the following error message in /var/log/messages:

"vif vif-16-3: 1 mapping shared-frames 2310/2311 port 11"

(the error message above means that netif_map failed for some reason in XenBus)

 

If we repeat this same exact test using XEN 3.0.4, we never have any problems. All vifs in all VMs work correctly. This problem must be specific to XEN 3.1.

 

I have searched the web and this user list and I have not been able to find out if anyone else has observed this same problem or if a fix for this problem already exists (if there is a fix, please post info about it here). If there is no fix for this yet, I will be looking into this bug to solve it, any pointers on where to concentrate my debugging efforts would be appreciated (I don't know the XEN code that well).

 

One other strange note about this issue: If we leave the failed VM alone, we actually can create another VM w/o any problem (vifs come up correctly). Afterwards, we can then destroy and create the VM that used to fail and now it boots w/o any problems (its vif comes up correctly). This smells like a race condition bug in the XEN code (this proves that it is not due to low resources or something like that).

 

Any help on this issue would be greatly appreciated,

 

Thank you,

 

Eric

 

---

Get the free Yahoo! toolbar and rest assured with the added security of spyware protection.
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users