[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-users] Infinite loop shutting down xendomains
More info on the network script at the bottom - John On Sat, 2006-01-28 at 23:14 -0500, John A. Sullivan III wrote: > Pardon me for a long detailed post but this looks pretty serious. The > bottom line is that, when I attempt to shut down or reboot my simple > xen-2.0.7 server running FC3 with two domUs, shutting down xendomains > appears to send python into an infinite loop. This may be due to a > serious misconfiguration on my part but it is certainly a big problem. > > If we shutdown the domUs manually first, the system reboots perfectly > fine. > > I'll describe some history and include lots of log information. The > configuration is slightly unusual. Both domUs are VPN gateways running > the 2.4.30 kernel with openswan and openvpn. Because of the openvpn, > but are privileged domains but the only device driver configured in the > kernel is the openvpn tun/tap adapter. > > We first noticed problems during shutdown in that the last domU to > shutdown crashed very nastily when closing openvpn. We could see this > even when manually stopping the domUs. We surmised that the problem was > we had tun0 and tun1 defined in both domains and killing them in one > must be killing them in another. We renamed those tun adapters in the > second domain to tun10 and tun11 and that problem went away. > > We still noticed very long reboot times. The hang seemed to be at > killall but the "logjam" seemed to break after xendomains errored out. > We assumed this was because killall killed something that xendomains > needed to shutdown gracefully so we added explicit K scripts in rc0.d > and rc6.d for xend and xendomains. > > At first this seemed to work but it must have been only coincidental > because the problem returned. We then realized we had brain cramped and > shutdown xend before xendomains. We thought for sure we had found the > problem and put them in the proper order but, alas, the problem did not > go away. > > To gain a little more insight into exactly what was happening, we > manually ran each kill script in rc6.d. When we ran K78xendomains stop, > we saw the problem, or at least many aspect of it. > > >From a xen console, we saw that the first domU had cleanly shutdown but > the second one was still listed howbeit with zero memory. > > The hard disks were thrashing madly and top showed python consuming 99.9 > percent CPU utilization. Most of the time, during reboot, we do not see > the madly thrashing disks. They do so for a moment as one might expect > when shutting down the domains but then just blip occasionally > thereafter. > > Here is what we see in the logs: > > >From messages (ntpd was the last service stopped before xendomains): > Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15 > Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded > Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode > Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering > disabled state > Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering > disabled state > Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode > Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering > disabled state > Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering > disabled state > Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled > Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled > Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled > Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled > Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode > Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering > disabled state > Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering > disabled state > Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode > Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering > disabled state > Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering > disabled state > Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled > Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled > Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled > Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled > > Here is the complete xend-debug.log: > network start bridge=xen-br0 netdev=eth0 antispoof=no > VIRTUAL MEMORY ARRANGEMENT: > Loaded kernel: c0100000->c0319394 > Init. ramdisk: c031a000->c031a000 > Phys-Mach map: c031a000->c0332000 > Page tables: c0332000->c0334000 > Start info: c0334000->c0335000 > Boot stack: c0335000->c0336000 > TOTAL: c0000000->c0400000 > ENTRY ADDRESS: c0100000 > vif-bridge up vif=vif1.0 domain=niagras1 mac=02:00:00:01:02:00 > bridge=xen-br0 > vif-bridge up vif=vif1.1 domain=niagras1 mac=aa:00:00:58:4a:a7 > bridge=xen-br0 > VIRTUAL MEMORY ARRANGEMENT: > Loaded kernel: c0100000->c0319394 > Init. ramdisk: c031a000->c031a000 > Phys-Mach map: c031a000->c0332000 > Page tables: c0332000->c0334000 > Start info: c0334000->c0335000 > Boot stack: c0335000->c0336000 > TOTAL: c0000000->c0400000 > ENTRY ADDRESS: c0100000 > vif-bridge up vif=vif2.0 domain=vpn1dc1 mac=02:00:00:01:01:00 > bridge=xen-br0 > vif-bridge up vif=vif2.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > recv_fe_driver_status> {'status': 1} > > recv_fe_driver_status> > > recv_fe_interface_connect {'tx_shmem_frame': 44763, 'rx_shmem_frame': > 44764, 'handle': 0} > > recv_fe_interface_connect {'tx_shmem_frame': 44767, 'rx_shmem_frame': > 44768, 'handle': 1} > recv_fe_driver_status> {'status': 1} > > recv_fe_driver_status> > > recv_fe_interface_connect {'tx_shmem_frame': 69339, 'rx_shmem_frame': > 69340, 'handle': 0} > > recv_fe_interface_connect {'tx_shmem_frame': 69343, 'rx_shmem_frame': > 69344, 'handle': 1} > {} > {} > VirqClient.virqReceived> 4 > vif-bridge down vif=vif2.0 domain=vpn1dc1 mac=02:00:00:01:01:00 > bridge=xen-br0 > vif-bridge down vif=vif2.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > VirqClient.virqReceived> 4 > vif-bridge down vif=vif1.0 domain=niagras1 mac=02:00:00:01:02:00 > bridge=xen-br0 > vif-bridge down vif=vif1.1 domain=niagras1 mac=aa:00:00:26:24:c4 > bridge=xen-br0 > Traceback (most recent call last): > File "/usr/lib/python2.3/site-packages/twisted/protocols/basic.py", > line 229, in dataReceived > return self.rawDataReceived(data) > File "/usr/lib/python2.3/site-packages/twisted/protocols/http.py", > line 1015, in rawDataReceived > self.allContentReceived() > File "/usr/lib/python2.3/site-packages/twisted/protocols/http.py", > line 1006, in allContentReceived > req.requestReceived(command, path, version) > File "/usr/lib/python2.3/site-packages/twisted/protocols/http.py", > line 557, in requestReceived > self.process() > --- <exception caught here> --- > File "/usr/lib/python2.3/site-packages/twisted/web/server.py", line > 165, in process > self.render(resrc) > File "/usr/lib/python2.3/site-packages/twisted/web/server.py", line > 172, in render > body = resrc.render(self) > File "/usr/lib/python2.3/site-packages/twisted/web/resource.py", line > 201, in render > return m(request) > File "/usr/lib/python/xen/xend/server/SrvDomain.py", line 180, in > render_POST > return self.perform(req) > File "/usr/lib/python/xen/xend/server/SrvBase.py", line 103, in > perform > return self._perform(op, op_method, req) > File "/usr/lib/python/xen/xend/server/SrvBase.py", line 109, in > _perform > return self._perform_err(err, op, req) > File "/usr/lib/python/xen/xend/server/SrvBase.py", line 107, in > _perform > val = op_method(op, req) > File "/usr/lib/python/xen/xend/server/SrvDomain.py", line 47, in > op_shutdown > val = fn(req.args, {'dom': self.dom.id}) > File "/usr/lib/python/xen/xend/Args.py", line 145, in __call__ > return self.call_with_form_args(self.fn, fargs, xargs=xargs) > File "/usr/lib/python/xen/xend/Args.py", line 121, in > call_with_form_args > return fn(*params, **keys) > File "/usr/lib/python/xen/xend/XendDomain.py", line 477, in > domain_shutdown > val = xend.domain_shutdown(dominfo.id, reason, key) > File "/usr/lib/python/xen/xend/server/SrvDaemon.py", line 723, in > domain_shutdown > ctrl = self.domainCF.getController(dom) > File "/usr/lib/python/xen/xend/server/controller.py", line 323, in > getController > ctrl = self.createController(dom) > File "/usr/lib/python/xen/xend/server/domain.py", line 20, in > createController > return DomainController(self, dom) > File "/usr/lib/python/xen/xend/server/domain.py", line 38, in __init__ > self.registerChannel() > File "/usr/lib/python/xen/xend/server/controller.py", line 220, in > registerChannel > self.channel = self.channelFactory.domChannel(self.dom) > File "/usr/lib/python/xen/xend/server/channel.py", line 83, in > domChannel > remote_port=remote_port) > File "/usr/lib/python/xen/xend/server/channel.py", line 253, in > __init__ > remote_port=remote_port) > File "/usr/lib/python/xen/xend/server/channel.py", line 137, in > createPort > remote_port=int(remote_port)) > xen.lowlevel.xu.PortError: Failed to map domain control interface > /usr/lib/python2.3/site-packages/twisted/web/util.py:290: FutureWarning: > hex()/oct() of negative int will return a signed string in Python 2.4 > and up > return '''<div class="instance"><span class="instanceName">%s instance > @ %s</span> > /usr/lib/python2.3/site-packages/twisted/web/util.py:266: FutureWarning: > hex()/oct() of negative int will return a signed string in Python 2.4 > and up > w('<div class="dict"><span class="heading">Dictionary instance @ % > s</span>' % hex(id(d))) > /usr/lib/python2.3/site-packages/twisted/web/util.py:279: FutureWarning: > hex()/oct() of negative int will return a signed string in Python 2.4 > and up > w('<div class="list"><span class="heading">List instance @ %s</span>' > % hex(id(l))) > > Init. ramdisk: c031a000->c031a000 > Phys-Mach map: c031a000->c0332000 > Page tables: c0332000->c0334000 > Start info: c0334000->c0335000 > Boot stack: c0335000->c0336000 > TOTAL: c0000000->c0400000 > ENTRY ADDRESS: c0100000 > vif-bridge up vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00 > bridge=xen-br0 > vif-bridge up vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > recv_fe_driver_status> {'status': 1} > > recv_fe_driver_status> > > recv_fe_interface_connect {'tx_shmem_frame': 88257, 'rx_shmem_frame': > 88258, 'handle': 0} > > recv_fe_interface_connect {'tx_shmem_frame': 88261, 'rx_shmem_frame': > 88262, 'handle': 1} > VIRTUAL MEMORY ARRANGEMENT: > Loaded kernel: c0100000->c0319394 > Init. ramdisk: c031a000->c031a000 > Phys-Mach map: c031a000->c0332000 > Page tables: c0332000->c0334000 > Start info: c0334000->c0335000 > Boot stack: c0335000->c0336000 > TOTAL: c0000000->c0400000 > ENTRY ADDRESS: c0100000 > vif-bridge up vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00 > bridge=xen-br0 > vif-bridge up vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b > bridge=xen-br0 > recv_fe_driver_status> {'status': 1} > > recv_fe_driver_status> > > recv_fe_interface_connect {'tx_shmem_frame': 185025, 'rx_shmem_frame': > 185026, 'handle': 0} > > recv_fe_interface_connect {'tx_shmem_frame': 185029, 'rx_shmem_frame': > 185030, 'handle': 1} > {} > {} > VirqClient.virqReceived> 4 > vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00 > bridge=xen-br0 > vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b > bridge=xen-br0 > VirqClient.virqReceived> 4 > vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00 > bridge=xen-br0 > vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > -bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > interface vif4.1 does not exist! > VIRTUAL MEMORY ARRANGEMENT: > Loaded kernel: c0100000->c02c9df4 > Init. ramdisk: c02ca000->c02ca000 > Phys-Mach map: c02ca000->c02e2000 > Page tables: c02e2000->c02e4000 > Start info: c02e4000->c02e5000 > Boot stack: c02e5000->c02e6000 > TOTAL: c0000000->c0400000 > ENTRY ADDRESS: c0100000 > vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00 > bridge=xen-br0 > interface vif5.0 does not exist! > vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > interface vif5.1 does not exist! > recv_fe_driver_status> {'status': 1} > > recv_fe_driver_status> > > recv_fe_interface_connect {'tx_shmem_frame': 202459, 'rx_shmem_frame': > 202460, 'handle': 0} > > recv_fe_interface_connect {'tx_shmem_frame': 202463, 'rx_shmem_frame': > 202464, 'handle': 1} > VirqClient.virqReceived> 4 > vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00 > bridge=xen-br0 > interface vif5.0 does not exist! > vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > interface vif5.1 does not exist! > > Here is the information from xend.log: > Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15 > Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded > Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode > Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering > disabled state > Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering > disabled state > Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode > Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering > disabled state > Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering > disabled state > Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled > Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled > Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled > Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled > Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode > Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering > disabled state > Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering > disabled state > Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode > Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering > disabled state > Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering > disabled state > Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled > Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled > Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled > Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled > Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot > Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6 > Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for > user root > Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for > user root > Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for > user root > Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains: > Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error > Jan 28 22:18:05 xenpub1dc1 xendomains: failed > Jan 28 22:18:05 xenpub1dc1 xendomains: > Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains: failed > Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped. > Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating. > Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded > Jan 28 22:18:06 xenpub1dc1 exiting on signal 15 > Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart. > Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded > Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg > started. > Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0 > (root@xxxxxxxxxxxxxxxxxxxxxxxxx) (gcc version 3.4.4 20050721 (Red Hat > 3.4.4-2)) #4 Thu Jan > File messages not changed so no update needed. > [root@xenpub1dc1 log]# ls > acpid boot.log.1 cron.2 mail messages prelink.log > samba spooler wtmp xfrd.log > anaconda.log boot.log.2 cups maillog messages.1 rpmpkgs > secure spooler.1 xend-debug.log Xorg.0.log > anaconda.syslog cron dmesg maillog.1 messages.2 rpmpkgs.1 > secure.1 spooler.2 xend.log Xorg.0.log.old > boot.log cron.1 lastlog maillog.2 ppp rpmpkgs.2 > secure.2 vbox xend.log.1 yum.log > [root@xenpub1dc1 log]# joe xend-debug.log > Processing '/etc/joe/joerc'...done > Processing '/etc/joe/joerc'...done > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I xend-debug.log > Row 171 Col 1 11:02 Ctrl-K H for help > recv_fe_driver_status> > > recv_fe_interface_connect {'tx_shmem_frame': 185025, 'rx_shmem_frame': > 185026, 'handle': 0} > > recv_fe_interface_connect {'tx_shmem_frame': 185029, 'rx_shmem_frame': > 185030, 'handle': 1} > {} > {} > VirqClient.virqReceived> 4 > vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00 > bridge=xen-br0 > vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b > bridge=xen-br0 > VirqClient.virqReceived> 4 > vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00 > bridge=xen-br0 > vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > -bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > interface vif4.1 does not exist! > VIRTUAL MEMORY ARRANGEMENT: > Loaded kernel: c0100000->c02c9df4 > Init. ramdisk: c02ca000->c02ca000 > Phys-Mach map: c02ca000->c02e2000 > Page tables: c02e2000->c02e4000 > Start info: c02e4000->c02e5000 > Boot stack: c02e5000->c02e6000 > TOTAL: c0000000->c0400000 > ENTRY ADDRESS: c0100000 > vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00 > bridge=xen-br0 > interface vif5.0 does not exist! > vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > interface vif5.1 does not exist! > recv_fe_driver_status> {'status': 1} > > recv_fe_driver_status> > > recv_fe_interface_connect {'tx_shmem_frame': 202459, 'rx_shmem_frame': > 202460, 'handle': 0} > > recv_fe_interface_connect {'tx_shmem_frame': 202463, 'rx_shmem_frame': > 202464, 'handle': 1} > VirqClient.virqReceived> 4 > vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00 > bridge=xen-br0 > interface vif5.0 does not exist! > vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > interface vif5.1 does not exist! > > File xend-debug.log not changed so no update needed. > [root@xenpub1dc1 log]# ls > acpid boot.log.1 cron.2 mail messages prelink.log > samba spooler wtmp xfrd.log > anaconda.log boot.log.2 cups maillog messages.1 rpmpkgs > secure spooler.1 xend-debug.log Xorg.0.log > anaconda.syslog cron dmesg maillog.1 messages.2 rpmpkgs.1 > secure.1 spooler.2 xend.log Xorg.0.log.old > boot.log cron.1 lastlog maillog.2 ppp rpmpkgs.2 > secure.2 vbox xend.log.1 yum.log > [root@xenpub1dc1 log]# joe xend.log > Processing '/etc/joe/joerc'...done > Processing '/etc/joe/joerc'...done > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I xend.log > Row 6079 Col 1 11:04 Ctrl-K H for help > [2006-01-28 22:06:09 xend] INFO (XendRoot:113) EVENT> > xend.domain.shutdown ['vpn1dc1', '2', 'halt'] > [2006-01-28 22:06:09 xend] INFO (XendRoot:113) EVENT> > xend.domain.shutdown ['niagras1', '1', 'halt'] > [2006-01-28 22:06:22 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=niagras1 id=1 > [2006-01-28 22:06:22 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=1 reason=poweroff > [2006-01-28 22:06:22 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 1 poweroff 0 > [2006-01-28 22:06:22 xend] INFO (XendDomain:568) Destroying domain: > name=niagras1 > [2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:665) Destroying vifs > for domain 1 > [2006-01-28 22:06:22 xend] DEBUG (netif:305) Destroying vif domain=1 > vif=0 > [2006-01-28 22:06:22 xend] DEBUG (netif:305) Destroying vif domain=1 > vif=1 > [2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:674) Destroying vbds > for domain 1 > [2006-01-28 22:06:22 xend] DEBUG (blkif:552) Destroying blkif domain=1 > [2006-01-28 22:06:22 xend] DEBUG (blkif:408) Destroying vbd domain=1 > idx=0 > [2006-01-28 22:06:22 xend] DEBUG (blkif:408) Destroying vbd domain=1 > idx=1 > [2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 1 > [2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:622) Closing channel to > domain 1 > [2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT> xend.virq 4 > [2006-01-28 22:06:22 xend] DEBUG (blkif:363) Unbinding vbd (type phy) > from VG1/niagras1root > [2006-01-28 22:06:22 xend] DEBUG (blkif:363) Unbinding vbd (type phy) > from VG1/niagras1swap > [2006-01-28 22:06:22 xend] INFO (console:58) Console disconnected 15 > 127.0.0.1 3484 > [2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['niagras1', '1', 'poweroff'] > [2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['niagras1', '1'] > [2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT> > xend.console.disconnect [15, '127.0.0.1', 3484] > [2006-01-28 22:06:23 xend] INFO (XendRoot:113) EVENT> xend.domain.died > ['niagras1', '1'] > [2006-01-28 22:06:24 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:24 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:24 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:24 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:665) Destroying vifs > for domain 2 > [2006-01-28 22:06:24 xend] DEBUG (netif:305) Destroying vif domain=2 > vif=0 > [2006-01-28 22:06:24 xend] DEBUG (netif:305) Destroying vif domain=2 > vif=1 > [2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:674) Destroying vbds > for domain 2 > [2006-01-28 22:06:24 xend] DEBUG (blkif:552) Destroying blkif domain=2 > [2006-01-28 22:06:24 xend] DEBUG (blkif:408) Destroying vbd domain=2 > idx=0 > [2006-01-28 22:06:24 xend] DEBUG (blkif:408) Destroying vbd domain=2 > idx=1 > [2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:622) Closing channel to > domain 2 > [2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.virq 4 > [2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy) > from VG1/vpn1dc1root > [2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy) > from VG1/vpn1dc1swap > [2006-01-28 22:06:24 xend] INFO (console:58) Console disconnected 16 > 127.0.0.1 3986 > [2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15 > Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded > Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode > Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering > disabled state > Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering > disabled state > Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode > Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering > disabled state > Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering > disabled state > Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled > Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled > Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled > Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled > Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode > Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering > disabled state > Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering > disabled state > Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode > Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering > disabled state > Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering > disabled state > Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled > Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled > Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled > Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled > Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot > Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6 > Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for > user root > Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for > user root > Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for > user root > Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains: > Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error > Jan 28 22:18:05 xenpub1dc1 xendomains: failed > Jan 28 22:18:05 xenpub1dc1 xendomains: > Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains: failed > Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped. > Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating. > Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded > Jan 28 22:18:06 xenpub1dc1 exiting on signal 15 > Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart. > Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded > Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg > started. > Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0 > (root@xxxxxxxxxxxxxxxxxxxxxxxxx) (gcc version 3.4.4 20050721 (Red Hat > 3.4.4-2)) #4 Thu Jan > File messages not changed so no update needed. > [root@xenpub1dc1 log]# ls > acpid boot.log.1 cron.2 mail messages prelink.log > samba spooler wtmp xfrd.log > anaconda.log boot.log.2 cups maillog messages.1 rpmpkgs > secure spooler.1 xend-debug.log Xorg.0.log > anaconda.syslog cron dmesg maillog.1 messages.2 rpmpkgs.1 > secure.1 spooler.2 xend.log Xorg.0.log.old > boot.log cron.1 lastlog maillog.2 ppp rpmpkgs.2 > secure.2 vbox xend.log.1 yum.log > [root@xenpub1dc1 log]# joe xend-debug.log > Processing '/etc/joe/joerc'...done > Processing '/etc/joe/joerc'...done > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I xend-debug.log > Row 171 Col 1 11:02 Ctrl-K H for help > recv_fe_driver_status> > > recv_fe_interface_connect {'tx_shmem_frame': 185025, 'rx_shmem_frame': > 185026, 'handle': 0} > > recv_fe_interface_connect {'tx_shmem_frame': 185029, 'rx_shmem_frame': > 185030, 'handle': 1} > {} > {} > VirqClient.virqReceived> 4 > vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00 > bridge=xen-br0 > vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b > bridge=xen-br0 > VirqClient.virqReceived> 4 > vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00 > bridge=xen-br0 > vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > -bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > interface vif4.1 does not exist! > VIRTUAL MEMORY ARRANGEMENT: > Loaded kernel: c0100000->c02c9df4 > Init. ramdisk: c02ca000->c02ca000 > Phys-Mach map: c02ca000->c02e2000 > Page tables: c02e2000->c02e4000 > Start info: c02e4000->c02e5000 > Boot stack: c02e5000->c02e6000 > TOTAL: c0000000->c0400000 > ENTRY ADDRESS: c0100000 > vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00 > bridge=xen-br0 > interface vif5.0 does not exist! > vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > interface vif5.1 does not exist! > recv_fe_driver_status> {'status': 1} > > recv_fe_driver_status> > > recv_fe_interface_connect {'tx_shmem_frame': 202459, 'rx_shmem_frame': > 202460, 'handle': 0} > > recv_fe_interface_connect {'tx_shmem_frame': 202463, 'rx_shmem_frame': > 202464, 'handle': 1} > VirqClient.virqReceived> 4 > vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00 > bridge=xen-br0 > interface vif5.0 does not exist! > vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > interface vif5.1 does not exist! > > File xend-debug.log not changed so no update needed. > [root@xenpub1dc1 log]# ls > acpid boot.log.1 cron.2 mail messages prelink.log > samba spooler wtmp xfrd.log > anaconda.log boot.log.2 cups maillog messages.1 rpmpkgs > secure spooler.1 xend-debug.log Xorg.0.log > anaconda.syslog cron dmesg maillog.1 messages.2 rpmpkgs.1 > secure.1 spooler.2 xend.log Xorg.0.log.old > boot.log cron.1 lastlog maillog.2 ppp rpmpkgs.2 > secure.2 vbox xend.log.1 yum.log > [root@xenpub1dc1 log]# joe xend.log > Processing '/etc/joe/joerc'...done > Processing '/etc/joe/joerc'...done > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I xend.log > Row 6077 Col 91 11:06 Ctrl-K H for help > [2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy) > from VG1/vpn1dc1swap > [2006-01-28 22:06:24 xend] INFO (console:58) Console disconnected 16 > 127.0.0.1 3986 > [2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> > xend.console.disconnect [16, '127.0.0.1', 3986] > [2006-01-28 22:06:26 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:26 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:26 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:26 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:26 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:26 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:26 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:28 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:28 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:28 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:28 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:28 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:28 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:28 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:30 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:30 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:30 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:30 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:30 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:30 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:30 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:32 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:32 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:32 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:32 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:32 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:32 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:32 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:34 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:34 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:34 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:34 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:34 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:34 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:34 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:36 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15 > Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded > Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode > Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering > disabled state > Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering > disabled state > Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode > Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering > disabled state > Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering > disabled state > Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled > Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled > Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled > Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled > Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode > Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering > disabled state > Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering > disabled state > Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode > Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering > disabled state > Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering > disabled state > Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled > Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled > Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled > Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled > Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot > Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6 > Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for > user root > Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for > user root > Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for > user root > Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains: > Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error > Jan 28 22:18:05 xenpub1dc1 xendomains: failed > Jan 28 22:18:05 xenpub1dc1 xendomains: > Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains: failed > Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped. > Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating. > Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded > Jan 28 22:18:06 xenpub1dc1 exiting on signal 15 > Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart. > Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded > Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg > started. > Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0 > (root@xxxxxxxxxxxxxxxxxxxxxxxxx) (gcc version 3.4.4 20050721 (Red Hat > 3.4.4-2)) #4 Thu Jan > File messages not changed so no update needed. > [root@xenpub1dc1 log]# ls > acpid boot.log.1 cron.2 mail messages prelink.log > samba spooler wtmp xfrd.log > anaconda.log boot.log.2 cups maillog messages.1 rpmpkgs > secure spooler.1 xend-debug.log Xorg.0.log > anaconda.syslog cron dmesg maillog.1 messages.2 rpmpkgs.1 > secure.1 spooler.2 xend.log Xorg.0.log.old > boot.log cron.1 lastlog maillog.2 ppp rpmpkgs.2 > secure.2 vbox xend.log.1 yum.log > [root@xenpub1dc1 log]# joe xend-debug.log > Processing '/etc/joe/joerc'...done > Processing '/etc/joe/joerc'...done > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I xend-debug.log > Row 171 Col 1 11:02 Ctrl-K H for help > recv_fe_driver_status> > > recv_fe_interface_connect {'tx_shmem_frame': 185025, 'rx_shmem_frame': > 185026, 'handle': 0} > > recv_fe_interface_connect {'tx_shmem_frame': 185029, 'rx_shmem_frame': > 185030, 'handle': 1} > {} > {} > VirqClient.virqReceived> 4 > vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00 > bridge=xen-br0 > vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b > bridge=xen-br0 > VirqClient.virqReceived> 4 > vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00 > bridge=xen-br0 > vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > -bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > interface vif4.1 does not exist! > VIRTUAL MEMORY ARRANGEMENT: > Loaded kernel: c0100000->c02c9df4 > Init. ramdisk: c02ca000->c02ca000 > Phys-Mach map: c02ca000->c02e2000 > Page tables: c02e2000->c02e4000 > Start info: c02e4000->c02e5000 > Boot stack: c02e5000->c02e6000 > TOTAL: c0000000->c0400000 > ENTRY ADDRESS: c0100000 > vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00 > bridge=xen-br0 > interface vif5.0 does not exist! > vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > interface vif5.1 does not exist! > recv_fe_driver_status> {'status': 1} > > recv_fe_driver_status> > > recv_fe_interface_connect {'tx_shmem_frame': 202459, 'rx_shmem_frame': > 202460, 'handle': 0} > > recv_fe_interface_connect {'tx_shmem_frame': 202463, 'rx_shmem_frame': > 202464, 'handle': 1} > VirqClient.virqReceived> 4 > vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00 > bridge=xen-br0 > interface vif5.0 does not exist! > vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01 > bridge=xen-br1 > interface vif5.1 does not exist! > > File xend-debug.log not changed so no update needed. > [root@xenpub1dc1 log]# ls > acpid boot.log.1 cron.2 mail messages prelink.log > samba spooler wtmp xfrd.log > anaconda.log boot.log.2 cups maillog messages.1 rpmpkgs > secure spooler.1 xend-debug.log Xorg.0.log > anaconda.syslog cron dmesg maillog.1 messages.2 rpmpkgs.1 > secure.1 spooler.2 xend.log Xorg.0.log.old > boot.log cron.1 lastlog maillog.2 ppp rpmpkgs.2 > secure.2 vbox xend.log.1 yum.log > [root@xenpub1dc1 log]# joe xend.log > Processing '/etc/joe/joerc'...done > Processing '/etc/joe/joerc'...done > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I xend.log > Row 6157 Col 100 11:07 Ctrl-K H for help > [2006-01-28 22:06:36 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:36 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:36 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:36 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:36 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:36 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:36 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:38 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:38 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:38 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:38 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:38 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:38 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:38 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:40 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:40 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:40 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:40 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:40 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:40 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:40 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:42 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:42 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:42 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:42 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:42 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:42 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:42 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:44 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:44 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:44 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:44 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:44 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:44 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:44 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:46 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:46 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:46 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:46 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:46 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:46 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:46 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:48 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:48 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:48 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:48 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:48 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:48 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:48 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:50 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:50 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:50 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:50 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:50 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:50 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:50 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:52 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:52 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:52 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:52 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:52 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:52 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:52 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:54 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:54 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:54 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:54 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:54 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:54 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:54 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:56 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:56 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:56 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:56 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > [2006-01-28 22:06:56 xend] DEBUG (XendDomainInfo:634) Closing console, > domain 2 > [2006-01-28 22:06:56 xend] INFO (XendRoot:113) EVENT> xend.domain.exit > ['vpn1dc1', '2', 'poweroff'] > [2006-01-28 22:06:56 xend] INFO (XendRoot:113) EVENT> > xend.domain.destroy ['vpn1dc1', '2'] > [2006-01-28 22:06:57 xend] DEBUG (XendDomain:244) XendDomain>reap> > domain died name=vpn1dc1 id=2 > [2006-01-28 22:06:57 xend] DEBUG (XendDomain:247) XendDomain>reap> > shutdown id=2 reason=poweroff > [2006-01-28 22:06:57 xend] DEBUG (XendDomain:487) > domain_restart_schedule> 2 poweroff 0 > [2006-01-28 22:06:57 xend] INFO (XendDomain:568) Destroying domain: > name=vpn1dc1 > etc., etc., etc. > > Here are the configuration files for the two domUs: > # -*- mode: python; -*- > #============================================================================ > # Python configuration setup for 'xm create'. > # This script sets the parameters used when a domain is created using > 'xm create'. > # You use a separate script for each domain you want to create, or > # you can set the parameters for the domain on the xm command line. > #============================================================================ > > #---------------------------------------------------------------------------- > # Kernel image file. > kernel = "/boot/vmlinuz-2.4.30-xenUVPN" > > # Optional ramdisk. > #ramdisk = "/boot/initrd.gz" > > # The domain build function. Default is 'linux'. > #builder='linux' > > # Initial memory allocation (in megabytes) for the new domain. > memory = 96 > > # A name for your domain. All domains must have different names. > name = "niagras1" > > # Which CPU to start domain on? > #cpu = -1 # leave to Xen to pick > > #---------------------------------------------------------------------------- > # Define network interfaces. > > # Number of network interfaces. Default is 1. > nics=2 > > # Optionally define mac and/or bridge for the network interfaces. > # Random MACs are assigned if not given. > #vif = [ 'mac=aa:00:00:00:00:11, bridge=xen-br0' ] > > vif = ['mac=02:00:00:01:02:00, bridge=xen-br0' ] > > #---------------------------------------------------------------------------- > # Define the disk devices you want the domain to have access to, and > # what you want them accessible as. > # Each disk entry is of the form phy:UNAME,DEV,MODE > # where UNAME is the device, DEV is the device name the domain will see, > # and MODE is r for read-only, w for read-write. > > #disk = [ 'phy:hda1,hda1,w' ] > > disk = [ 'phy:VG1/niagras1root,sda1,w', 'phy:VG1/niagras1swap,sda2,w' ] > > #---------------------------------------------------------------------------- > # Set the kernel command line for the new domain. > # You only need to define the IP parameters and hostname if the domain's > # IP config doesn't, e.g. in ifcfg-eth0 or via DHCP. > # You can use 'extra' to set the runlevel and custom environment > # variables used by custom rc scripts (e.g. VMID=, usr= ). > > # Set if you want dhcp to allocate the IP address. > #dhcp="dhcp" > # Set netmask. > #netmask= > # Set default gateway. > #gateway= > # Set the hostname. > #hostname= "vm%d" % vmid > > # Set root device. > root = "/dev/sda1 ro" > > # Root device for nfs. > #root = "/dev/nfs" > # The nfs server. > #nfs_server = '169.254.1.0' > # Root directory on the nfs server. > #nfs_root = '/full/path/to/root/directory' > > # Sets runlevel 4. > #extra = "4" > > #---------------------------------------------------------------------------- > # Set according to whether you want the domain restarted when it exits. > # The default is 'onreboot', which restarts the domain when it shuts > down > # with exit code reboot. > # Other values are 'always', and 'never'. > > #restart = 'onreboot' > > #============================================================================ > > > # -*- mode: python; -*- > #============================================================================ > # Python configuration setup for 'xm create'. > # This script sets the parameters used when a domain is created using > 'xm create'. > # You use a separate script for each domain you want to create, or > # you can set the parameters for the domain on the xm command line. > #============================================================================ > > #---------------------------------------------------------------------------- > # Kernel image file. > kernel = "/boot/vmlinuz-2.4.30-xenUVPN" > > # Optional ramdisk. > #ramdisk = "/boot/initrd.gz" > > # The domain build function. Default is 'linux'. > #builder='linux' > > # Initial memory allocation (in megabytes) for the new domain. > memory = 96 > > # A name for your domain. All domains must have different names. > name = "vpn1dc1" > > # Which CPU to start domain on? > #cpu = -1 # leave to Xen to pick > > #---------------------------------------------------------------------------- > # Define network interfaces. > > # Number of network interfaces. Default is 1. > nics=2 > > # Optionally define mac and/or bridge for the network interfaces. > # Random MACs are assigned if not given. > #vif = [ 'mac=aa:00:00:00:00:11, bridge=xen-br0' ] > > vif = ['mac=02:00:00:01:01:00, bridge=xen-br0', 'mac=02:00:00:01:01:01, > bridge=xen-br1' ] > > #---------------------------------------------------------------------------- > # Define the disk devices you want the domain to have access to, and > # what you want them accessible as. > # Each disk entry is of the form phy:UNAME,DEV,MODE > # where UNAME is the device, DEV is the device name the domain will see, > # and MODE is r for read-only, w for read-write. > > #disk = [ 'phy:hda1,hda1,w' ] > > disk = [ 'phy:VG1/vpn1dc1root,sda1,w', 'phy:VG1/vpn1dc1swap,sda2,w' ] > > #---------------------------------------------------------------------------- > # Set the kernel command line for the new domain. > # You only need to define the IP parameters and hostname if the domain's > # IP config doesn't, e.g. in ifcfg-eth0 or via DHCP. > # You can use 'extra' to set the runlevel and custom environment > # variables used by custom rc scripts (e.g. VMID=, usr= ). > > # Set if you want dhcp to allocate the IP address. > #dhcp="dhcp" > # Set netmask. > #netmask= > # Set default gateway. > #gateway= > # Set the hostname. > #hostname= "vm%d" % vmid > > # Set root device. > root = "/dev/sda1 ro" > > # Root device for nfs. > #root = "/dev/nfs" > # The nfs server. > #nfs_server = '169.254.1.0' > # Root directory on the nfs server. > #nfs_root = '/full/path/to/root/directory' > > # Sets runlevel 4. > #extra = "4" > > #---------------------------------------------------------------------------- > # Set according to whether you want the domain restarted when it exits. > # The default is 'onreboot', which restarts the domain when it shuts > down > # with exit code reboot. > # Other values are 'always', and 'never'. > > #restart = 'onreboot' > > #============================================================================ > > > I'm stumped but it sure looks serious. Have I produced some dreadful > misconfiguration? This host is being shipped to a remote data center on > Monday and goes into production in a week. Any help would be greatly > appreciated. Thanks - John We are also using a very slightly customized network script since we have three interfaces. Here is the network script: # Exit if anything goes wrong. set -e # First arg is the operation. OP=$1 shift # Pull variables in args in to environment. for arg ; do export "${arg}" ; done bridge=${bridge:-xen-br0} netdev=${netdev:-eth0} antispoof=${antispoof:-yes} SCRIPTDIR=/etc/xen/scripts echo "network $OP bridge=$bridge netdev=$netdev antispoof=$antispoof" # Usage: transfer_addrs src dst # Copy all IP addresses (including aliases) from device $src to device $dst. transfer_addrs () { local src=$1 local dst=$2 # Don't bother if $dst already has IP addresses. if ip addr show dev ${dst} | egrep -q '^ *inet ' ; then return fi # Address lines start with 'inet' and have the device in them. # Replace 'inet' with 'ip addr add' and change the device name $src # to 'dev $src'. Remove netmask as we'll add routes later. ip addr show dev ${src} | egrep '^ *inet ' | sed -e " s/inet/ip addr add/ s@\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+\)/[0-9]\+@\1@ s/${src}/dev ${dst}/ " | sh -e } # Usage: del_addrs src del_addrs () { local src=$1 ip addr show dev ${src} | egrep '^ *inet ' | sed -e " s/inet/ip addr del/ s@\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+\)/[0-9]\+@\1@ s/${src}/dev ${src}/ " | sh -e } # Usage: transfer_routes src dst # Get all IP routes to device $src, delete them, and # add the same routes to device $dst. # The original routes have to be deleted, otherwise adding them # for $dst fails (duplicate routes). transfer_routes () { local src=$1 local dst=$2 # List all routes and grep the ones with $src in. # Stick 'ip route del' on the front to delete. # Change $src to $dst and use 'ip route add' to add. ip route list | grep ${src} | sed -e " h s/^/ip route del / P g s/${src}/${dst}/ s/^/ip route add / P d " | sh -e } # Usage: create_bridge bridge create_bridge () { local bridge=$1 # Don't create the bridge if it already exists. if ! brctl show | grep -q ${bridge} ; then brctl addbr ${bridge} brctl stp ${bridge} off brctl setfd ${bridge} 0 fi ifconfig ${bridge} up } # Usage: add_to_bridge bridge dev add_to_bridge () { local bridge=$1 local dev=$2 # Don't add $dev to $bridge if it's already on a bridge. if ! brctl show | grep -q ${dev} ; then brctl addif ${bridge} ${dev} fi } # Usage: antispoofing dev bridge # Set the default forwarding policy for $dev to drop. # Allow forwarding to the bridge. antispoofing () { local dev=$1 local bridge=$2 iptables -P FORWARD DROP iptables -A FORWARD -m physdev --physdev-in ${dev} -j ACCEPT } # Usage: show_status dev bridge # Print ifconfig and routes. show_status () { local dev=$1 local bridge=$2 echo '============================================================' ifconfig ${dev} ifconfig ${bridge} echo ' ' ip route list echo ' ' route -n echo '============================================================' } op_start () { if [ "${bridge}" == "null" ] ; then return fi # Create the bridge and give it the interface IP addresses. # Move the interface routes onto the bridge. while read bridge netdev do create_bridge ${bridge} if [ -n "$(ip add ls dev $netdev | grep 'inet ')" ]; then transfer_addrs ${netdev} ${bridge} transfer_routes ${netdev} ${bridge} fi # Don't add $dev to $bridge if it's already on a bridge. if ! brctl show | grep -q ${netdev} ; then brctl addif ${bridge} ${netdev} fi if [ ${antispoof} == 'yes' ] ; then antispoofing ${netdev} ${bridge} fi done < ${SCRIPTDIR}/bridges.txt } op_stop () { if [ "${bridge}" == "null" ] ; then return fi # Remove the interface from the bridge. # Move the routes back to the interface. while read bridge netdev do brctl delif ${bridge} ${netdev} if [ -n "$(ip add ls dev $netdev | grep 'inet ')" ]; then transfer_routes ${bridge} ${netdev} fi done < ${SCRIPTDIR}/bridges.txt # It's not our place to be enabling forwarding... } case ${OP} in start) op_start ;; stop) op_stop ;; status) show_status ${netdev} ${bridge} ;; *) echo 'Unknown command: ' ${OP} echo 'Valid commands are: start, stop, status' exit 1 esac Here is the bridges.txt file: xen-br0 eth0 xen-br1 eth1 xen-br2 eth2 -- John A. Sullivan III Open Source Development Corporation +1 207-985-7880 jsullivan@xxxxxxxxxxxxxxxxxxx Financially sustainable open source development http://www.opensourcedevel.com _______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-users
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |