RE: [Xen-devel] a question about popen() performance on domU

["Petersson, Mats" <mats.petersson@xxxxxxx>]

> -----Original Message-----
> From: xuehai zhang [mailto:hai@xxxxxxxxxxxxxxx] 
> Sent: 24 November 2005 15:41
> To: Petersson, Mats
> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx; Tim Freeman; Kate Keahey
> Subject: Re: [Xen-devel] a question about popen() performance on domU
> 
> > See comments below.  
> 
> Thanks Mats. I have more questions about your comments below.
> 
> Xuehai
> 
> >>-----Original Message-----
> >>From: xuehai zhang [mailto:hai@xxxxxxxxxxxxxxx]
> >>Sent: 24 November 2005 14:02
> >>To: Petersson, Mats
> >>Cc: xen-devel@xxxxxxxxxxxxxxxxxxx; Tim Freeman; Kate Keahey
> >>Subject: Re: [Xen-devel] a question about popen() 
> performance on domU
> >>
> >>Mats,
> >>
> >>Thanks a lot for the response.
> >>
> >>
> >>>I did have a look at popen, and essentially, it does the
> >>
> >>following [
> >>
> >>>the real code is MUCH more complicated, doing lots of
> >>
> >>open/dup/close
> >>
> >>>on pipes and stuff]:
> >>>if (!fork())
> >>> exec("/bin/sh", "sh", "-c", cmd, NULL);
> >>
> >>I took a look at the popen source code too yesterday and the above 
> >>lines are the esstential part. A thread at gnu list
> >>(http://lists.gnu.org/archive/html/bug-global/2005-06/msg00001
> >>.html) suggets
> >>popen() might depend on how fast /bin/sh is executed. On both my VM 
> >>and the physical machine, the kernel version is 2.6.11, 
> glibc version 
> >>is 2.3.2.ds1-21, and /bin/sh is linked to /bin/bash. I also 
> tried to 
> >>see any difference of the shared libraries used by /bin/sh on both 
> >>machines and found /bin/sh on the physical machine uses 
> libraries from 
> >>/lib/tls while for the VM this directory is disabled.
> >>
> >>VM$ ldd /bin/sh
> >>         libncurses.so.5 => /lib/libncurses.so.5 (0xb7fa7000)
> >>         libdl.so.2 => /lib/libdl.so.2 (0xb7fa3000)
> >>         libc.so.6 => /lib/libc.so.6 (0xb7e70000)
> >>         /lib/ld-.so.2 => /lib/ld-.so.2 (0xb7fea000)
> >>
> >>PHYSICAL$  ldd /bin/sh
> >>         libncurses.so.5 => /lib/libncurses.so.5 (0xb7fa6000)
> >>         libdl.so.2 => /lib/tls/libdl.so.2 (0xb7fa2000)
> >>         libc.so.6 => /lib/tls/libc.so.6 (0xb7e6d000)
> >>         /lib/ld-.so.2 => /lib/ld-.so.2 (0xb7fea000)
> > 
> > 
> > In this particular case, I would think that lib/tls is not 
> a factor, 
> > but it may be worth disabling the tls libraries on the 
> pysical machine 
> > too, just to make sure... [just "mv /lib/tls 
> /lib/tls.disabled" should 
> > do it].
> 
> I don't think /lib/tls is the factor too. I did rerun the 
> tests with tls disabled on the physical machine and it gave 
> even worse performance for the tests. So, I switched it back.
> 
> > 
> >>>The fork creates another process, which then executes the /bin/sh, 
> >>>which again causes another fork/exec to take place in the 
> effort of 
> >>>executing the actual command given.
> >>>
> >>>So the major component of popen would be fork() and
> >>
> >>execl(), both of
> >>
> >>>which cause, amongst other things, a lot of page-table work and 
> >>>task-switching.
> >>>
> >>>Note that popen is implemented in glibc [I took the 2.3.6
> >>
> >>source code
> >>
> >>>from www.gnu.org for my look at this], so there's no
> >>
> >>difference in the
> >>
> >>>implementation of popen itself - the difference lies in how
> >>
> >>the Linux
> >>
> >>>kernel handles fork() and exec(), but maybe more importantly, how 
> >>>task-switches and page-tables are handled in Linux native
> >>
> >>and Xen-Linux.
> >>
> >>>Because Xen keeps track of the page-tables on top of
> >>
> >>Linux's handling
> >>
> >>>of page-tables, you get some extra work here. So, it should
> >>
> >>really be
> >>
> >>>slower on Xen than on native Linux.
> >>>[In fact, the question came up not so long ago, why Xen was SLOWER 
> >>>than native Linux on popen (and some others) in a particular 
> >>>benchmark, and the result of that investigation was that
> >>
> >>it's down to,
> >>
> >>>mainly, task-switching takes longer in Xen.]
> >>
> >>I agree with your explanation about Xen was SLOWER than 
> native Linux 
> >>on popen because of the longer task-switching in Xen. The problem I 
> >>met (popen runs faster on Xen VM than the physical machine) looks 
> >>abnormal. I ran several home-made benchmarking programming and used 
> >>the "strace" tool to trace the system call performance. The first 
> >>program is to test the performance of both popen and pclose 
> (a loop of 
> >>popen call with a followup pclose call) and the source of 
> the program 
> >>and the strace results are available at 
> >>http://people.cs.uchicago.edu/~hai/tmp/gt2gram/strace-popen/st
> >>race.txt. The results shows the waitpid syscall costs more time on 
> >>physical machine than on the VM (see the usecs/call valuee in the 
> >>following table).
> >>
> >>            % time     seconds  usecs/call     calls    
> >>errors syscall
> >>            ------ ----------- ----------- ---------
> >>--------- ----------------
> >>VM:         63.43    0.127900        6395        20         
> >>  waitpid
> >>PHYSICAL
> >>MACHINE:    93.87    0.532498       26625        20         
> >>  waitpid
> >>
> >>waitpid is called by pclose as shown in the glibc source 
> code. So, my 
> >>original post questioning the performance of popen should 
> take pclose 
> >>into consideration too. A more accurate question I should post is, 
> >>popen+pclose executes faster on my VM than my physical machine. The 
> >>popen/pclose benchmark I did narrows the problem down to 
> waitpid that 
> >>waitpid somehow is suffering on the physical machine.
> >>So, I did a followup experiment to test the fork and waitpid 
> >>performance on both machines. The program is a loop of fork 
> call with 
> >>a followup waitpid call. The source of the program and the strace 
> >>results are available at 
> >>http://people.cs.uchicago.edu/~hai/tmp/gt2gram/strace-fork/str
> >>ace.txt. The strace results confirm the waitpid costs more 
> time on the 
> >>physical machine (154 usec/call) than the VM (56 usec/call).
> >>However, the program runs faster on the physical machine 
> (not like the 
> >>popen/pclose program) and the results suggest the fork 
> syscall used on 
> >>the VM costs more time than the clone syscall on the 
> physical machine. 
> >>I have a question here, why the physical machine doesn't use fork 
> >>syscall but the clone syscall for the same program?
> > 
> > 
> > Because it's using the same source for glibc! glibc says to use 
> > _IO_fork(), which is calling the fork syscall. Clone would 
> probably do 
> > the same thing, but for whatever good or bad reason, the 
> author(s) of 
> > thise code chose to use fork. There may be good reasons, or 
> no reason 
> > at all to do it this way. I couldn't say. I don't think it makes a 
> > whole lot of difference if the actual command executed by popen is 
> > actually "doing something", rather than just an empty "return".
> 
> Do you have any suggestion why the same code uses different 
> syscalls on two machines which have the same kernel and glibc?

That I can't explain. I guess one possibility is that in some way, the
fork() call gets translated to clone() at some other level. I did a grep
for _IO_fork in the source for glibc, and it comes back as #define
_IO_fork fork . 

> 
> >>>The reason it is not would probably have something to do with the 
> >>>differences in hardware on Linux vs. Xen platforms, perhaps
> >>
> >>the fact
> >>
> >>>that your file-system is a virtual block-device and thus
> >>
> >>lives inside
> >>
> >>>a file that is perhaps better cached or otherwise handled in a 
> >>>different way on the Xen-system.
> >>
> >>Let me describe the hardware context of my VM and physical machine. 
> >>The host of my VM and the physical machine I tested against the VM, 
> >>are two nodes of a physical cluster with the same hardware 
> >>configuration (Dual Intel PIII 498.799 MHz CPU, 512MB 
> memory, a 4GB HD 
> >>with same partitions). The physical machine is rebooted 
> with "nosmp". 
> >>The VM host  is rebooted into Xen with "nosmp" (Xen version 
> >>information is "Latest
> >>ChangeSet: 2005/05/03 17:30:40 1.1846 
> >>4277a730mvnFSFXrxJpVRNk8hjD4Vg"). Xen dom0 is assigned 96MB 
> memory and 
> >>the VM is the only user domain running on the VM host with 395MB 
> >>memory. Both dom0 and the VM are pinned to CPU 0.
> >>
> >>Yes, the backends of the VM's VBDs are loopback files in 
> dom0. Three 
> >>loopback files are used to map to three partitions inside 
> of the VM. I 
> >>acutally thought about the possible caching effect of the VM's VBD 
> >>backends, but not sure how to testify it and compare it with the 
> >>physical machine. Is it possible the Xen has different assurance of 
> >>writing back than the physical machine, that is, the data 
> is kept in 
> >>memory longer before is actually written to disk?
> > 
> > 
> > Xen itself doesn't know ANYTHING about the disk/file where 
> the data for
> > the Dom0 or DomU comes from, so no, Xen would not do that. 
> However, the
> > loopback file-system that is involved in VBD's would potentially do
> > things that are different from the actual hardware.
> 
> So, there is possbility that the loopback file-system can do 
> something tricky like caching and 
> results in better performance for applications running inside 
> of the VM?
> 
> > I think you should be able to mount the virtual disk as a 
> "device" on
> > your system. 
> 
> What does "your system" here refer to? Does it mean dom0 or 
> inside of domU?

Your system here referes to "PHYSICAL". 
> 
> > I don't know of the top of my head how to do that, but
> > essentially something like this:
> > mount myimage.hdd loop/ -t ext3 [additional parameters may 
> be needed]. 
> > 
> > You could then do "chroot loop/", and perform your tests there. This
> > should execute the same thing from the same place on the 
> native linux as
> > you would in DomU. 
> > 
> > Now, this may not run faster on native than your original 
> setup, but I
> > wouldn't be surprised if it does... 
> 
> This is interesting. I will try to run the same tests if I 
> canmount the virtual disk as "device" 
> successfully.

Please share the results... ;-)
> 
> Thanks.
> 
> Xuehai
> > 
> >>>Now, I'm not saying that there isn't a possibility that 
> >>
> >>something is 
> >>
> >>>managed differently in Xen that makes this run faster - I 
> >>
> >>just don't 
> >>
> >>>really see how that would be likely, since everything that 
> >>
> >>happens in 
> >>
> >>>the system is going to be MORE complicated by the extra 
> >>
> >>layer of Xen 
> >>
> >>>involved.
> >>
> >>>If anyone else has some thoughts on this subject, it would be 
> >>>interesting to hear.
> >>
> >>I agree. But given the VM having same hardware/software 
> >>configuration as the physical machine, it runs faster still 
> >>looks abnormal to me. I wonder if there is any other more 
> >>efficient debugging strategies I can use to investigate it. I 
> >>appreciate if any one has any more suggestions.
> >>
> >>Thanks again.
> >>
> >>Xuehai
> >>
> >>
> >>>>-----Original Message-----
> >>>>From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
> >>>>[mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf 
> Of xuehai 
> >>>>zhang
> >>>>Sent: 23 November 2005 20:26
> >>>>To: xen-devel@xxxxxxxxxxxxxxxxxxx
> >>>>Cc: Tim Freeman; Kate Keahey
> >>>>Subject: [Xen-devel] a question about popen() performance on domU
> >>>>
> >>>>Dear all,
> >>>>When I compared the performance of some application on both 
> >>
> >>a Xen domU 
> >>
> >>>>and a standard linux machine (where domU runs on a 
> similar physical 
> >>>>mahine), I notice the application runs faster on the domU 
> >>
> >>than on the 
> >>
> >>>>physical machine.
> >>>>Instrumenting the application code shows the application 
> >>
> >>spends more 
> >>
> >>>>time on popen() calls on domU than on the physical machine. 
> >>
> >>I wonder 
> >>
> >>>>if  xenlinux does some special modification of the popen code to 
> >>>>improve its performance than the original Linux popen code?
> >>>>Thanks in advance for your help.
> >>>>Xuehai
> >>>>
> >>>>_______________________________________________
> >>>>Xen-devel mailing list
> >>>>Xen-devel@xxxxxxxxxxxxxxxxxxx
> >>>>http://lists.xensource.com/xen-devel
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> > 
> > 
> 
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel