RE: [Xen-devel] RE: [Xen-users] How to: Improve Network Throughput of a XEN kernel?

[Thomas Halinka <lists@xxxxxxxxx>]

Hi,

Am Mittwoch, den 03.03.2010, 10:44 -0500 schrieb Mike Viau:
> > 
> > On Tue, Mar 02, 2010 at 07:50:59PM -0500, Mike Viau wrote:
> > > >Any 
> just first argument dummy=dummy should on kenel line.
> > > > Read Wiki one more time. It works.
> > > 
> > > I believe you that it does work. I was able to see (XEN) messages
> coming from the hypervisor, but no matter how I changed the option I
> kept getting a kernel fault.
> > > 
> > > I feel like it was time to stop mucking around with getting Ubuntu
> and the grub2 boot loader to work with XEN so I decided to install a
> headless OpenSUSE 11.2 system.
> > > 
> > > The openSUSE distribution supported Xen 3.4.1 tools, and had a
> 2.6.31.12 xenified kernel. On this setup I experience identical
> network performance.
> > > 
> > > ~1MB/s - w/ xen kernel
> > > ~4.5 MB/s - w/ non-xen kernel
> > > 
> > > I use the e1000e kernel module driver for my Ethernet card.
> > > 
> > > Is this typical Dom0 (or Xen Host) network performance with a
> xenified kernel or is just in my two setups that one encounters such a
> network performance hit to the Dom0?
> > >
> > 
> > No, it's not typical.
> > 
> > How are you measuring the performance? In a gigabit network you
> should be able
> > to push around 110 MB/sec.. 4.5 MB/sec sounds really slow.
> 
> 
> I am using Fast Ethernet (100MB). What sort of throughput should I
> expect becasue I also believe ~1MB is too slow, even just to/from the
> Dom0, without any network emulation to the DomUs.

10 - 12 MB/s should be expected....

> 
> I simply used SCP, HTTP, and FTP file transfer protocols on the Xen
> host to another computer on the LAN. 

Please do something like

Start netCat on a "server"

Server: # nc -l 1111 > /dev/null

and start dd+nc on your XEN-Box/Client

Client: # dd if=/dev/zero bs=1M count=10000 | nc $SERVER-IP 1111

This should show up raw network throughput....


> The ~4.5MB/s bottleneck is the hard drive on the other computer so I
> suspect the true bandwidth would be higher, but getting past the
> ~1MB/s roadblock is my major concern.

use /dev/null instead of HDDs

> I have tried two linux distros and both forward ported xenified and
> parvirtualized xen kernel.
> 
> Does the Xen hypervisor have any effect on the network throughput?

No, but the underlying dom0-Kernel could affect the NIC-Performance...
Had some Issues with running Intel 10GB-NICs (ixgbe) on a 2.6.18-Kernel,
while 2.6.27/29 performed nice.

>  Would updating the Intel E1000E kernel module driver affect the
> network throughput?
> 
> 
> -M


hth,

thomas

> 
> 
> > 
> > -- Pasi
> > 
> > > -M
> > > 
> > > --- On Mon, 3/1/10, Mike Viau <viaum@xxxxxxxxxxxxxxx> wrote:
> > > 
> > > From: Mike Viau <viaum@xxxxxxxxxxxxxxx>
> > > Subject: RE: [Xen-devel] RE: [Xen-users] How to: Improve Network
> Throughput of a XEN kernel?
> > > To: bderzhavets@xxxxxxxxx
> > > Cc: xen-devel@xxxxxxxxxxxxxxxxxxx, xen-users@xxxxxxxxxxxxxxxxxxx
> > > Date: Monday, March 1, 2010, 2:52 PM
> > > 
> > > Why using grub2 vs grub is baffling am not sure....
> > > 
> > > I think I'd prefer to use the kernel command line in grub2 so that
> the options may be changed easier laster on. However I still have been
> having so success with boot
> > > XEN with Ubuntu's grub2 boot loader.
> > > 
> > > Am I correct to conclude CONFIG_CMDLINE_BOL must = N to allow
> GRUB2 to pass kernel options?
> > > [1]http://cateee.net/lkddb/web-lkddb/CMDLINE_BOOL.html
> > > 
> > > I have read through the Wiki page at
> [2]http://wiki.xensource.com/xenwiki/XenCommonProblems. I am running a
> 64-bit Xen Hypervisor.
> > > 
> > > Thus far I have been having no success with using the example
> entry found at
> [3]http://old.nabble.com/Strange-interaction-from-grub2-and-XEN-td26464067.html
> > > 
> > > A question regarding that entry is what would be the root device
> when there are separate /boot and root partitions? I am using the
> partition with the /boot (grub2) as
> > > my 'set root' device right?
> > > 
> > > I am also using the dummy=dummy as the first option on the first
> module line, again as discovered on the wiki.
> > > 
> > > Additionally posted on another thread here in the Xen mailing list
> was:
> > > 
> > > > On 1 March 2010 08:09, Luke Carrier <luke.carrier@xxxxxxxxx>
> wrote:
> > > >>
> > > >> Not sure about Debian, but I know Ubuntu isn*t ideal for Xen
> any more
> > > >> because of the presence of GRUB2 * the new boot loader doesn*t
> support
> > > >> passing certain command lines and the current multiboot
> implementation
> > > >> breaks the method Xen uses to load the
> > > Dom0 kernel * you can*t use Ubuntu
> > > >> 9.10 Server with Xen.
> > > 
> > > I am confused now because Boris clearly created a wordpress.com
> weblog in where Ubuntu 9.10 Server was
> > > used with XEN.
> > > 
> > > Lastly, out of curiosity, what (or what kind) of commands cannot
> be passed with GRUB2?
> > > 
> > > ThX!
> > > 
> > > 
> > > -M
> > > 
> > >
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > > 
> > > Date: Sat, 27 Feb 2010 11:20:31 -0800
> > > From: bderzhavets@xxxxxxxxx
> > > Subject: Re: [Xen-devel] RE: [Xen-users] How to: Improve Network
> Throughput of a XEN kernel?
> > > To: mike.viau@xxxxxxxxxxxxxxx
> > > CC: xen-devel@xxxxxxxxxxxxxxxxxxx; xen-users@xxxxxxxxxxxxxxxxxxx
> > > 
> > > In Xen Wiki is mentioned , that parameters for kernel may be set
> via command line as follows:-
> > >
> ************************************************************************************************************
> > > module (hd1,10)/vmlinuz-2.6.31.8 dummy=dummy root=/dev/sda7 ro
> console=tty0
> > >
> ************************************************************************************************************
> > > # cat /proc/cmdline
> > > 
> > > shows that GRUB2 skips first argument, what causes problem.
> > > 
> > > Compiling built-in CMD_LINE is just an option.
> > > Read all Wiki written by Pasi and you would never be surprised ;)
> > > 
> > > > P.S: What is your feeling on using a recent Ubuntu system with
> > > > grub-legacy as the boot loader? I noticed that Squeeze has
> adopted grub2 > package as well too...
> > > 
> > > I have development Xen 3.4.3 instance on top Ubuntu 9.10 Server
> with GRUB2 ( baremetall) loading any from 3 mentioned bellow kernels
> under Xen 3.4.3
> > > 
> > > 1.) 2.6.32.9 PVOPS
> > > 2.) 2.6.31.6 PVOPS
> > > 3.) 2.6.31.12 xenified
> > > 
> > > with no problems with dummy=dummy as first parameter ( thanks
> Pasi).
> > > It may also may be done via compiling kernels with built-in
> CMD_LINE.
> > > It works fine as well.
> > > 
> > > Boris.
> > > 
> > > --- On Sat, 2/27/10, Mike Viau <viaum@xxxxxxxxxxxxxxx> wrote:
> > > 
> > > From: Mike Viau <viaum@xxxxxxxxxxxxxxx>
> > > Subject: [Xen-devel] RE: [Xen-users] How to: Improve Network
> Throughput of a XEN kernel?
> > > To: bderzhavets@xxxxxxxxx
> > > Cc: xen-devel@xxxxxxxxxxxxxxxxxxx, xen-users@xxxxxxxxxxxxxxxxxxx
> > > Date: Saturday, February 27, 2010, 12:03 PM
> > > 
> > > Yes, thank you Boris for pointing that out. I found that was
> defiantly true. The 3.2.1 xen hypervisor could only boot the 2.6.26-2
> xen kernel and the xenified
> > > 2.6.31.12.
> > > 
> > > Additionally I found that:
> > > 
> > > Xend does not start when using pv_ops dom0 kernel?
> > > 
> > > In December 2009 pv_ops dom0 kernel modules were renamed to have a
> "xen-" prefix in them, ie. "evtchn.ko" became "xen-evtchn.ko".
> > > 
> > > This makes Xen 3.4.x xend fail to start, because it tried to load
> "evtchn.ko", but that doesn't exist. You need to load "xen-evtchn.ko"
> and then start xend. Fedora 12
> > > xen-3.4.2-2 rpms have this problem fixed.
> > > 
> > > Also make sure you have xenfs mounted to "/proc/xen", that's
> needed aswell.
> > > 
> > > [Source: [4]http://wiki.xensource.com/xenwiki/XenParavirtOps]
> > > 
> > > And it appears the Debian xen-3.4.2-2 debs (in squeeze) have not
> fixed this problem.
> > > 
> > > Lastly I am still doing some reading on the grub2 package and how
> to configure it. It turns out that I have been using grub-legacy (or
> just good old grub) for so long
> > > that configuring the grub2 boot loader is turning out to be my own
> personal kryptonite.
> > > 
> > > 
> > > 
> > > I am still working on getting the hypervisor to load the dom0
> kernel in a Ubuntu system.
> > > 
> > > 
> > > 
> > > I have noticed an example in your wordpress @ [5]Set up Xen
> 3.4.3-rc2 & Libvirt 0.7.0 Dom0 (with 2.6.31.8 xenified aka Suse
> kernel) on top of Ubuntu 9.10 Server
> > > 
> > > 
> > > 
> > > With CONFIG_CMDLINE="root=/dev/sdb11 ro console=tty0"
> > > 
> > > 
> > > 
> > > menuentry "Xen 3.4 / Ubuntu 9.10 kernel 2.6.31.8 xenified" {
> > > insmod ext2
> > > set root=(hd1,10)
> > > multiboot (hd1,10)/xen-3.4.gz
> > > module (hd1,10)/vmlinuz-2.6.31.8
> > > module (hd1,10)/initrd-2.6.31.8.img
> > > }
> > > 
> > > 
> > > 
> > > I am sure you wouldn't of mentioned the CONFIG_CMDLINE kernel
> option is it was not important for using with grub2 right? I have not
> have to compile a kernel with the
> > > command line previous as I was just able to supply the arguments
> on the kernel line of my menu.lst (grub configuration file).
> > > 
> > > P.S: What is your feeling on using a recent Ubuntu system with
> grub-legacy as the boot loader? I noticed that Squeeze has adopted
> grub2 package as well too...
> > > 
> > > 
> > > -M
> > > 
> > >
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > > 
> > > Date: Sat, 27 Feb 2010 04:32:59 -0800
> > > From: bderzhavets@xxxxxxxxx
> > > Subject: Re: FW: [Xen-users] How to: Improve Network Throughput of
> a XEN kernel?
> > > To: jbeulich@xxxxxxxxxx; mike.viau@xxxxxxxxxxxxxxx
> > > CC: xen-devel@xxxxxxxxxxxxxxxxxxx; xen-users@xxxxxxxxxxxxxxxxxxx
> > > 
> > > You cannot load pvops kernel under xen-hypervisor 3.2.1 ( >=3.4.0
> in general)
> > > 
> > > Boris.
> > > 
> > > --- On Fri, 2/26/10, Mike Viau <viaum@xxxxxxxxxxxxxxx> wrote:
> > > 
> > > From: Mike Viau <viaum@xxxxxxxxxxxxxxx>
> > > Subject: FW: [Xen-users] How to: Improve Network Throughput of a
> XEN kernel?
> > > To: bderzhavets@xxxxxxxxx, jbeulich@xxxxxxxxxx
> > > Cc: xen-users@xxxxxxxxxxxxxxxxxxx, xen-devel@xxxxxxxxxxxxxxxxxxx
> > > Date: Friday, February 26, 2010, 3:19 PM
> > > 
> > > Hello again,
> > > 
> > > Just a quick update...
> > > 
> > > I was really hoping it was just going to be an issue with the
> kernel .config, however I copied exact configuration from the working
> Debain 2.6.26-2-xen-amd64 kernel
> > > and used a make oldconfig but the end result was the same.
> > > 
> > > After booting the xenified (2.6.31.12) or pvops kernel (2.6.31.6)
> via both xen-hypervisor 3.2.1 or xen-hypervisor 3.4.2 the maximum data
> transfer to/from the Dom0
> > > (Xen host) is ~1MB/s.
> > > 
> > > Next I plan on re-trying my efforts on a Ubuntu system to see if
> same network bottlenecking is present.
> > > 
> > >
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > > 
> > > Date: Fri, 26 Feb 2010 09:16:59 -0800
> > > From: bderzhavets@xxxxxxxxx
> > > Subject: Re: [Xen-users] How to: Improve Network Throughput of a
> XEN kernel?
> > > To: xen-users@xxxxxxxxxxxxxxxxxxx; viaum@xxxxxxxxxxxxxxx
> > > 
> > > Would escalate to xen-devel. 2.6.31.8 (12) aka Suse xenified was
> Jan Beulich stuff.
> > > He might have an answer.
> > > Boris.
> > > 
> > > --- On Fri, 2/26/10, Mike Viau <viaum@xxxxxxxxxxxxxxx> wrote:
> > > 
> > > From: Mike Viau <viaum@xxxxxxxxxxxxxxx>
> > > Subject: [Xen-users] How to: Improve Network Throughput of a XEN
> kernel?
> > > To: xen-users@xxxxxxxxxxxxxxxxxxx
> > > Cc: waldi@xxxxxxxxxx
> > > Date: Friday, February 26, 2010, 12:09 PM
> > > 
> > > Hi there,
> > > 
> > > I am starting a new thread to try and solve a problem I am having
> will slow network throughput to the XEN host (not to a domU).
> > > 
> > > I have confirmed my problem occurs with PVOPS 'xen/master' kernel
> as well using
> > >
> [6]http://pasik.reaktio.net/xen/pv_ops-dom0-debug/config-2.6.31.6-pvops-dom0-xen-master-x86_64
>  as the .config.
> > > 
> > > It also occurs with a forward ported xenified kernel (2.6.31.12)
> using the attached config-11-xen (.config).
> > > 
> > > However when using the xen kernel found in debain lenny (2.6.26-2)
> for the amd64 architecture I get similiar transfer rates to my Xen
> host as I do on a baremetel
> > > kernel.
> > > 
> > > Bastian Blank perhaps you could point me in the right direction in
> regards to a debian specific customization or patch I should be aware
> of?
> > > Maxium transfer rate (to/from hard disk) using the PVOPS
> 'xen/master' kernel and forward ported xenified kernel (2.6.31.12) is
> bottlenecking at ~1MB/s. On the
> > > debain lenny xen kernel I am getting ~4.5MB/s.
> > > 
> > > I was hoping one could suggest options in the .config that might
> be modified in order to improve network throughput of a XEN kernel and
> therefor increase the data
> > > transfer rates over the netowork to and from the Xen host.
> > > 
> > > Thanks alot for any help! As always I will be willing to accept
> any suggestions :)
> > > 
> > > -M
> > > 
> > > 
> 
> 
> ______________________________________________________________________
> Live connected with Messenger on your phone Learn more.
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users