[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] 2.6.18.8 xen.org dom0 kernel w/RHEL 5 PV on HVM guests


  • To: Pasi Kärkkäinen <pasik@xxxxxx>
  • From: Andrew Evans <andrewe@xxxxxxxxxxxxx>
  • Date: Thu, 18 Mar 2010 15:20:45 -0700
  • Cc: xen-users@xxxxxxxxxxxxxxxxxxx
  • Delivery-date: Thu, 18 Mar 2010 15:22:32 -0700
  • Domainkey-signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=message-id:date:from:user-agent:mime-version:to:cc:subject: references:in-reply-to:content-type:content-transfer-encoding; b=vn+TVfftcYi9QAEyNcUsM4pnEiwBPzR3nVQoF+l+YFmsx+91cR6wg8I2EfvuUUtf
  • List-id: Xen user discussion <xen-users.lists.xensource.com>

On 03/18/10 14:38, Pasi Kärkkäinen wrote:
> On Thu, Mar 18, 2010 at 12:46:11PM -0700, Andrew Evans wrote:
>   
>> I've been beating my head against the wall trying to get stub domains
>> working with PV on HVM guests running RHEL 5. Up to now I've been using
>> the RHEL 5 dom0 kernel, but that doesn't work with stub domains because
>> gntdev isn't built in (and can't be built as a module either -- I
>> tried). So I built the 2.6.18.8 dom0 kernel from xen-unstable.hg. That
>> got stub domains working with RHEL 5 HVM guests, but PV on HVM guests
>> fail to find the root filesystem when initrd gets ready to pivot to the
>> real root filesystem. PV on HVM without stub domain has the same problem.
>>
>> --- begin domU console output ---
>> Creating block device nodes.
>> Loading xen-platform-pci.ko module
>> GSI 16 sharing vector 0xA9 and IRQ 16
>> ACPI: PCI Interrupt 0000:00:03.0[A] -> GSI 28 (level, low) -> IRQ 169
>> Xen version 3.4.
>> Hypercall area is 1 pages.
>> Grant table initialized
>> Loading xen-vbd.ko module
>> Loading xen-balloon.ko module
>> xen_mem: Initialising balloon driver.
>> Loading xen-vnif.ko module
>> netfront: Initialising virtual ethernet driver.
>> Loading jbd.ko module
>> netfront: device eth0 has copying receive path.
>> Loading ext3.ko module
>> Loading xen-vbd.ko module
>> insmod: error inserting '/lib/xen-vbd.ko': -1 File exists
>> Loading xen-vbd.ko module
>> insmod: error inserting '/lib/xen-vbd.ko': -1 File exists
>> Trying to resume from /dev/xvda2
>> Unable to access resume device (/dev/xvda2)
>>     
>
> Unable to access... hmm.
>
>
>   
>> Creating root device.
>> Registering block device major 202
>>  xvda:Mounting root filesystem.
>>  xvda1 xvda2 xvda3
>> mount: could not find filesystem '/dev/root'
>>     
> I guess it's the same problem here..
>
>
>   
>> Setting up other filesystems.
>> Setting up new root fs
>> setuproot: moving /dev failed: No such file or directory
>> no fstab.sys, mounting internal defaults
>> setuproot: error mounting /proc: No such file or directory
>> setuproot: error mounting /sys: No such file or directory
>> Switching to new root and running init.
>> unmounting old /dev
>> unmounting old /proc
>> unmounting old /sys
>> switchroot: mount failed: No such file or directory
>> Kernel panic - not syncing: Attempted to kill init!
>> --- end domU console output ---
>>
>> Notice that the PV disks and their partitions are detected.
>>
>>     
> Yeah, but they don't work for some reason..
>   

...until the system finishes booting (if I remove 'hda=noprobe
ide0=noprobe' from domU kernel command line, allowing initrd/nash to
mount disks via /dev/hd*)

>> Kernel command line is root=LABEL=ROOT, but root=/dev/xvda3 doesn't work
>> either.
>>
>> This same VM boots fine if I reboot the hypervisor with the RHEL 5 dom0
>> kernel.
>>
>> /var/log/xen/* show nothing out of the ordinary. The only hint of a
>> problem are these lines from xm dmesg:
>>
>> (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 13, error -22
>> (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 13, error -22
>> (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 13, error -22
>>
>>     
> These sound related..
>   

Yup. I can't believe event channels are generally broken though, as the
PV network driver works and the PV disk driver works once the system
comes up.

>> If I boot this same VM without disabling the IDE drivers, it mounts
>> filesystems using /dev/hd* and starts up successfully. Now here's the
>> baffling part: I can then mount / and /home using /dev/xvd*! So the
>> paravirt disk driver *is* working once the domU boots off /dev/hd*, but
>> for some reason it doesn't work when I try to boot from /dev/xvd* directly.
>>
>> So my questions are:
>>
>> 1. Why can't the initrd mount /dev/xvd* to / ?
>>
>>     
> Because the devices don't work for some reason..
>   

Well, yeah, I had worked that much out on my own. :) I can't imagine
what miracle is occurring between the time that initrd/nash tries to
mount /dev/xvda3 to / and when I do it manually from the command line
once I'm able to to log in, but such a miracle is in fact occurring!

>> 2. Why does this work with the RHEL 5 dom0 kernel but not with a
>> similarly-configured xen-unstable 2.6.18.8 dom0 kernel?
>>
>>     
> That's a good question.
>
>   
>> xen-unstable 2.6.18.8 dom0 kernel config attached. I've gotten the
>> Xen-related stuff as close to RHEL 5 as I can, but available CONFIG_*
>> defines vary somewhat between the two kernel source trees.
>>
>> Thanks for any help anyone can provide.
>>
>>     
> So the dom0 kernel was the only thing you changed?
>   

Yes.

> You used the same Xen hypervisor for both dom0 kernels? which hypervisor 
> version? 
>   

Xen 3.4.2, rebuilt from gitco source RPM w/oprofile active domains patch
applied:

$ sudo xm info
host                   : <snip>
release                : 2.6.18.8-xen
version                : #7 SMP Wed Mar 17 18:36:46 UTC 2010
machine                : x86_64
nr_cpus                : 8
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 1
cpu_mhz                : 2500
hw_caps                :
bfebfbff:20100800:00000000:00000140:000ce3bd:00000000:00000001:00000000
virt_caps              : hvm
total_memory           : 8190
free_memory            : 7266
node_to_cpu            : node0:0-7
node_to_memory         : node0:7266
xen_major              : 3
xen_minor              : 4
xen_extra              : .2
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)
cc_compile_by          : root
cc_compile_domain      : <snip>
cc_compile_date        : Wed Jan 27 00:36:30 UTC 2010
xend_config_format     : 4

thanks,

-Andrew


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.