[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] newbie in trouble with CentOS Xen

Quoting Alexandre Kouznetsov <alk@xxxxxxxxxx>:


El 04/04/13 10:13, Dave Stevens escribió:
[root@skeena ~]# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     9879     8 r-----    870.8
bulkley                                    1     2048     4 -b----   3283.0
[root@skeena ~]#

[root@skeena ~]# brctl show
bridge name    bridge id        STP enabled    interfaces
virbr0        8000.000000000000    yes
xenbr0        8000.feffffffffff    no        vif1.0
xenbr1        8000.feffffffffff    no        vif1.1

This tells us what Domains have network interfaces in what bridges.
It seems like there is a mix of configurations. Looks fine, for a bridged networking (not NATted, see below), except there might be a problem with peth0 and peth1.

[root@skeena ~]# ifconfig
eth1      Link encap:Ethernet  HWaddr 00:30:48:CE:84:7F
          inet addr:  Bcast:  Mask:
          inet6 addr: fe80::230:48ff:fece:847f/64 Scope:Link
          RX packets:81102 errors:0 dropped:0 overruns:0 frame:0
          TX packets:34828 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10772813 (10.2 MiB)  TX bytes:2721383 (2.5 MiB)

lo        Link encap:Local Loopback
          inet addr:  Mask:
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:3086 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3086 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4816900 (4.5 MiB)  TX bytes:4816900 (4.5 MiB)

peth0     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          UP BROADCAST NOARP  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:250 Base address:0xc000

peth1     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:42346 errors:0 dropped:0 overruns:0 frame:0
          TX packets:73652 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:9145360 (8.7 MiB)  TX bytes:5504883 (5.2 MiB)
          Interrupt:251 Base address:0xe000
This is weired.

The common way to give network to DomUs is to define a bridge, put the physical interface into it permanently, and plug DomU's interfaces there ion demand (you probably found that in the Wiki).
The very old network-bridge xen script was defining the bridges as following:
save IP configuration from eth0,
rename eth0 to peth0 (aka "physical eth0"),
create bridge called "eth0" (now that the name is free),
set IP configuration it took from the physical eth0 on the new bridge.

That way, before xend was brought up you had a plain eth0 to your network. After xend startup you had the same eth0 to your network. This proved to be little bit confusing and the configuration scripts produced unstable results, so late Xen documentation suggests to set up bridges using you platform means, and just tell Xen what bridge to use for what DomU's NIC.

Check this oldier wiki http://wiki.xensource.com/xenwiki/XenNetworking. It probably reflect better your scenario (and imho has nicer pictures)

What I see here are peth0 and peth1, they are supposed to be physical interfaces with a valid MAC, and be of the same nature. Actually, they have HWaddr FE:FF:FF:FF:FF:FF and one of them have a inet6 addr. eth1 seems to be a plain physical interface (I bet it's the one you use to connect to your server) while a bridge called "eth1" is expected. eth0 is simply missing.

Please tell me, how many physical network cards does your server have? How are they supposed to be used? Was any physical NIC added or removed recently? Maybe some NIC just died? The change of NIC number could confuse the network-script that badly. Did I told it was unstable?

well the MB is a Supermicro 2P with 2 nics on it, eth0 and 1. The incoming line goes into a Cisco router and that sends packets to eth1 on the MB. The nic hasn't changed recently, still using the usual one. There is no add-in nic.

It would be of great reference if you attach your DomU .cfg (under /etc/xen) file and /etc/xen/xend-config.sxp (I took the path form a Debian, it may be different in CentOS).


[root@skeena xen]# cat xend-config.sxp
# -*- sh -*-

# Xend configuration file.

# This example configuration is appropriate for an installation that
# utilizes a bridged network configuration. Access to xend via http
# is disabled.

# Commented out entries show the default for that entry, unless otherwise
# specified.

#(logfile /var/log/xen/xend.log)
#(loglevel DEBUG)

#(xend-http-server no)
(xend-unix-server yes)
#(xend-tcp-xmlrpc-server no)
#(xend-unix-xmlrpc-server yes)
#(xend-relocation-server no)
# The relocation server should be kept desactivated unless using a trusted
# network, the domain virtual memory will be exchanged in raw form without
# encryption of the communication. See also xend-relocation-hosts-allow option

(xend-unix-path /var/lib/xend/xend-socket)

# Port xend should use for the HTTP interface, if xend-http-server is set.
#(xend-port            8000)

# Port xend should use for the relocation interface, if xend-relocation-server
# is set.
#(xend-relocation-port 8002)

# Address xend should listen on for HTTP connections, if xend-http-server is
# set.
# Specifying 'localhost' prevents remote connections.
# Specifying the empty string '' (the default) allows all connections.
#(xend-address '')
#(xend-address localhost)

# Address xend should listen on for relocation-socket connections, if
# xend-relocation-server is set.
# Meaning and default as for xend-address above.
#(xend-relocation-address '')

# The hosts allowed to talk to the relocation port.  If this is empty (the
# default), then all connections are allowed (assuming that the connection
# arrives on a port and interface on which we are listening; see
# xend-relocation-port and xend-relocation-address above).  Otherwise, this
# should be a space-separated sequence of regular expressions.  Any host with
# a fully-qualified domain name or an IP address that matches one of these
# regular expressions will be accepted.
# For example:
#  (xend-relocation-hosts-allow '^localhost$ ^.*\.example\.org$')
#(xend-relocation-hosts-allow '')
(xend-relocation-hosts-allow '^localhost$ ^localhost\\.localdomain$')

# The limit (in kilobytes) on the size of the console buffer
#(console-limit 1024)

# To bridge network traffic, like this:
# dom0: fake eth0 -> vif0.0 -+
#                            |
#                          bridge -> real eth0 -> the network
#                            |
# domU: fake eth0 -> vifN.0 -+
# use
# (network-script network-bridge)
# Your default ethernet device is used as the outgoing interface, by default.
# To use a different one (e.g. eth1) use
# (network-script 'network-bridge netdev=eth1')
# The bridge is named xenbr0, by default.  To rename the bridge, use
# (network-script 'network-bridge bridge=<name>')
# It is possible to use the network-bridge script in more complicated
# scenarios, such as having two outgoing interfaces, with two bridges, and
# two fake interfaces per guest domain.  To do things like this, write
# yourself a wrapper script, and call network-bridge from it, as appropriate.
(network-script network-bridge-multi)

# The script used to control virtual interfaces.  This can be overridden on a
# per-vif basis when creating a domain or a configuring a new vif.  The
# vif-bridge script is designed for use with the network-bridge script, or
# similar configurations.
# If you have overridden the bridge name using
# (network-script 'network-bridge bridge=<name>') then you may wish to do the
# same here.  The bridge name can also be set when creating a domain or
# configuring a new vif, but a value specified here would act as a default.
# If you are using only one bridge, the vif-bridge script will discover that,
# so there is no need to specify it explicitly.
(vif-script vif-bridge)

## Use the following if network traffic is routed, as an alternative to the
# settings for bridged networking given above.
#(network-script network-route)
#(vif-script     vif-route)

## Use the following if network traffic is routed with NAT, as an alternative
# to the settings for bridged networking given above.
#(network-script network-nat)
#(vif-script     vif-nat)

# Dom0 will balloon out when needed to free memory for domU.
# dom0-min-mem is the lowest memory level (in MB) dom0 will get down to.
# If dom0-min-mem=0, dom0 will never balloon out.
(dom0-min-mem 256)

# In SMP system, dom0 will use dom0-cpus # of CPUS
# If dom0-cpus = 0, dom0 will take all cpus available
(dom0-cpus 0)

# Whether to enable core-dumps when domains crash.
#(enable-dump no)

# The tool used for initiating virtual TPM migration
#(external-migration-tool '')

# The interface for VNC servers to listen on. Defaults
# to  To restore old 'listen everywhere' behaviour
# set this to
#(vnc-listen '')

# The default password for VNC console on HVM domain.
# Empty string is no authentication.
(vncpasswd '')

# The default keymap to use for the VM's virtual keyboard
# when not specified in VM's configuration
(keymap 'en-us')

# The VNC server can be told to negotiate a TLS session
# to encryption all traffic, and provide x509 cert to
# clients enalbing them to verify server identity. The
# GTK-VNC widget, virt-viewer, virt-manager and VeNCrypt
# all support the VNC extension for TLS used in QEMU. The
# TightVNC/RealVNC/UltraVNC clients do not.
# To enable this create x509 certificates / keys in the
# directory /etc/xen/vnc
#  ca-cert.pem       - The CA certificate
#  server-cert.pem   - The Server certificate signed by the CA
#  server-key.pem    - The server private key
# and then uncomment this next line
# (vnc-tls 1)
# The certificate dir can be pointed elsewhere..
# (vnc-x509-cert-dir /etc/xen/vnc)
# The server can be told to request & validate an x509
# certificate from the client. Only clients with a cert
# signed by the trusted CA will be able to connect. This
# is more secure the password auth alone. Passwd auth can
# used at the same time if desired. To enable client cert
# checking uncomment this:
# (vnc-x509-verify 1)

# Allow probing of disk image file format.  This is insecure!  It lets
# a malicious domU read any file in dom0.  Applies only to fully
# virtual domUs.  Required for using formats other than raw.
#(enable-image-format-probing no)

# Number of seconds xend will wait for device creation
#(device-create-timeout 100)

# Strict checking when doing PCI passthrough; enabled by default
#(pci-dev-assign-strict-check yes)

[root@skeena xen]#

-- ends ---

Please check what version of this components do you have installed:
Xen hypervisor

[root@skeena ~]# xm info
host                   : skeena.bvserver.ca
release                : 2.6.18-308.20.1.el5xen
version                : #1 SMP Tue Nov 13 11:03:56 EST 2012
machine                : x86_64
nr_cpus                : 8
nr_nodes               : 1
sockets_per_node       : 2
cores_per_socket       : 4
threads_per_core       : 1
cpu_mhz                : 2311
hw_caps : 178bfbff:efd3fbff:00000000:00000110:00802009:00000000:000037ff
total_memory           : 16383
free_memory            : 4123
node_to_cpu            : node0:0-7
xen_major              : 3
xen_minor              : 1
xen_extra              : .2-308.20.1.el5
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20080704 (Red Hat 4.1.2-52)
cc_compile_by          : mockbuild
cc_compile_domain      : centos.org
cc_compile_date        : Tue Nov 13 10:06:48 EST 2012
xend_config_format     : 2
[root@skeena ~]#

Linux kernel

[root@skeena ~]# uname -a
Linux skeena.bvserver.ca 2.6.18-308.20.1.el5xen #1 SMP Tue Nov 13 11:03:56 EST 2012 x86_64 x86_64 x86_64 GNU/Linux
[root@skeena ~]#

xenutils (the name of package may be different in CentOS)

xentools (the name of package may be different in CentOS)

can't find an applicable name for the toolsets


[root@skeena ~]# iptables -L -v
    0     0 ACCEPT     all  --  any    virbr0  anywhere    state RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  virbr0 any
    0     0 ACCEPT     all  --  virbr0 virbr0  anywhere
    0     0 REJECT     all  --  any    virbr0  anywhere
anywhere            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  virbr0 any     anywhere
anywhere            reject-with icmp-port-unreachable

This must be the rules to restrict traffic within a bridge, dedicated to internal communication between Dom0 and DomU's. Unused, apparently. The rest of ipfilter configuration is permissive, so it shall not be the issue.

As the general solution to your problem, in case you just want to "make it work back" and not doing any new implementation (Dom0 OS and Xe upgrade would be probably cleaner if reinstalled), I would suggest to re-do the networking configuration.

I'm agreeable, just not sure how

Probably your DomU was configured with NATted networking, when it broke Dom0 kept it's external IP address. The bridged networking is simpler and easier to debug.


Thank you.

Alexandre Kouznetsov

Xen-users mailing list

The problem with being cynical is you can't keep up!

-- anon. philosopher

Xen-users mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.