[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] 9.0-RELEASE PV from scratch on XCP v1.1.0

Apologies for the re-post.  Forgot to sign original.  -Trx

Title:  HOWTO-FreeBSD-on-XCP
Author: John D. "Trix" Farrar
Date:   2012-02-09

* The Challenge - ParaVirtualized FreeBSD 9.0-RELEASE under XCP 1.1.0

The idea here is to create a PV FreeBSD VM under Xen Cloud Platform
(XCP) without starting with an HVM first.  The documentation I''ve
been able to find on-line is at least a year old and is mostly written
for Xen (under Linux or NetBSD) rather than XCP.  To make things more
interesting, the hardware on which I have XCP installed doesn''t
support Hardware Virtualization -- so I can''t just install an HVM and
convert it over. A PV under XCP means using pygrub as a bootloader to
load the FreeBSD kernel.  It seems that pygrub doesn''t know enough
about BSD disk slices and partitions to be able to find the kernel on
its own, so it will need a patch

This solution isn''t perfect, but I think it''s a good start.

* The Equipment

Yes, my hardware is old/cheap.

  * Xen Cloud Platform server - XCP v1.1.0

  * Build Server - FreeBSD 9.0-RELEASE (32-bit)

  * NFS Server - anything w/ enough space.  Can be the Build
    Server. (should NOT be the XCP Server)

* The Process

  * Patching pygrub

    Adrian Chadd (http://wiki.freebsd.org/AdrianChadd/XenHackery) did
    a great job laying the groundwork for a Xen (not XCP)
    installation.  I also got a lot of help from Adian''s patch to
    pygrub at http://people.freebsd.org/~adrian/xen/bsd_pygrub/.  The
    problem is that Adrian''s patch "is against the pygrub shipped with
    the xen-3.0.3-80.el5_3.2 CentOS 5.3 package."  Using Adrian''s
    patch, I wsa able to create a patch against the pygrub shipped
    with XCP v1.1.0:

Patch attached.
  * Creating a disk image

    I mounted an exported filesystem from the NFS server so that I
    would have a large, sharable space in which to place the disk
    image.  This space is also mounted on the XCP server.  In both
    cases, the NFS filesystem is mounted under /mnt.

    The following steps are taken on the BUILD host where /usr/src has
    already been compiled with buiidworld and buildkernel targets.

export FSIMAGE="FreeBSD-PV.img"

truncate -s 10G /mnt/${FSIMAGE}              # Create the file where the image 
will reside.
mdconfig -a -t vnode -f /mnt/${FSIMAGE} -u 0 # Create the image itself
fdisk -BI md0                                # Write the DOS partition table

# Create an inital BSD Label 
# For testing, we'll assume one large partition
bsdlabel -w -B md0s1
bsdlabel -e md0s1 # Change partition a from 'unused' to '4.2BSD'
newfs -O1 /dev/md0s1a # UFS, because pygrub doesn't grok UFS2

# Mount up the image and do the installation
mount /dev/md0s1a /media

# Perform the installation to the disk image.
cd /usr/src
make -s DESTDIR=/media KERNCONF=XEN installworld
make -s DESTDIR=/media KERNCONF=XEN installkernel
make -s DESTDIR=/media KERNCONF=XEN distribution

# Set up the ttys(5) file so the VM will use Xen's console.
cat >>/media/etc/ttys <<EOF

# Xen Console
xc0     "/usr/libexec/getty Pc"         vt100   on  secure


# Create fstab(5) so the root (only) filesystem will mount.
cat >>/media/etc/fstab <<EOF
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/xbd0s1a            /               ufs     rw              1       1


# Start and rc.conf(5) file
# NOTE: You won't be able to login via SSH until users are added.
cat >>/media/etc/rc.conf <<EOF

# BUILD Server - Unmount the filesystem, the image is done.
umount /media
mdconfig -d -u 0

The following section is the script I actually used to create a VM
that is more-or-less realistic as a template.


truncate -s 20G /mnt/${FSIMAGE}
mdconfig -a -t vnode -f /mnt/${FSIMAGE} -u 0
fdisk -BI md0
bsdlabel -w -B md0s1
cat >>xcp-optionb-bsdlabel.txt <<EOF
# /dev/md0s1:
8 partitions:
#          size     offset    fstype   [fsize bsize bps/cpg]
  a:      512MB         63    4.2BSD        0     0     0
  b:      512MB          *      swap                    
  c:   41929587          0    unused        0     0     # "raw" part, don't edit
  d:      512MB          *    4.2BSD        0     0     0
  e:    2097152    3145791    4.2BSD        0     0     0
  f:    6291456    5242943    4.2BSD        0     0     0
  g:   12582912   11534399    4.2BSD        0     0     0
  h:   17812276   24117311    4.2BSD        0     0     0


bsdlabel -R md0s1 xcp-optionb-bsdlabel.txt
rm xcp-optionb-bsdlabel.txt

# root partition has to be UFS1.  PyGRUB chokes on UFS2 (fsimage.ufs, actually)
newfs -O1 /dev/md0s1a && mount -v /dev/md0s1a /media
newfs     /dev/md0s1d && mkdir /media/tmp       && mount -v /dev/md0s1d 
newfs     /dev/md0s1e && mkdir /media/var       && mount -v /dev/md0s1e 
newfs     /dev/md0s1f && mkdir /media/usr       && mount -v /dev/md0s1f 
newfs     /dev/md0s1g && mkdir /media/usr/local && mount -v /dev/md0s1g 
newfs     /dev/md0s1h && mkdir /media/home      && mount -v /dev/md0s1h 
pushd /usr/src
make -s DESTDIR=/media KERNCONF=XEN installworld installkernel distribution
cat >>/media/etc/ttys <<EOF

# Xen Console
xc0     "/usr/libexec/getty Pc"         vt100   on  secure


cat >>/media/etc/fstab <<EOF
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/xbd0s1a            /               ufs     rw              1       1
/dev/xbd0s1f            /usr            ufs     rw              1       1
/dev/xbd0s1g            /usr/local      ufs     rw              1       1
/dev/xbd0s1d            /tmp            ufs     rw              1       1
/dev/xbd0s1e            /var            ufs     rw              1       1
/dev/xbd0s1h            /home           ufs     rw              1       1


cat >>/media/etc/rc.conf <<EOF

umount -v /media/home /media/usr/local /media/usr /media/var /media/tmp /media


  * Create the Virtual Host

    I got the bulk of this information from Grant McWilliams'' HOWTO
    for installing CentOS on an XCP host 

    All of the next steps are to be carried on at the Local Command
    Shell prompt on the XCP host.

VMNAME="FreeBSD-9.0"     # Edit this value, each VM must be different

# XCP Server - Create the Virtual Disk Image and copy .img file into it.
export SR_UUID=$(xe sr-list name-label="Local storage" --minimal)
export VDIUUID=$(xe vdi-create \
                    sr-uuid=${SR_UUID} \
                    name-label=${VMNAME}-vdi \
                    type=system \
                    virtual-size=11GiB )

# Pull the built image into a VDI (Virtual Disk Image)
# This operatoin will take a while. (like an hour and a half)
xe vdi-import \
    uuid=${VDIUUID} \

# Create a VM from the "Other install media" template
# Then customize to our taste
export TPLUUID=$(xe template-list \
                    name-label="Other install media" \
                    params=uuid --minimal)

# Create the VM itself
export VM_UUID=$(xe vm-install \
                    new-name-label=${VMNAME} \

# Customize
# Defaults:
# Desc.  Value    VM Parameter(s)
# RAM  = 256 MB = memory-static-max, memory-dynamic-max, memory-dynamic-min
#        128 MB = memory-static-min
# CPUs =   1    = VCPUs-at-startup, VCPUs-max
xe vm-param-set   uuid=${VM_UUID} name-description="New FreeBSD host"
xe vm-param-clear uuid=${VM_UUID} param-name=HVM-boot-policy
xe vm-param-clear uuid=${VM_UUID} param-name=HVM-boot-params
xe vm-param-set   uuid=${VM_UUID} PV-bootloader=pygrub
xe vm-param-set   uuid=${VM_UUID} 
xe vm-param-set   uuid=${VM_UUID} other-config:disable_pv_vnc=1
## xe vm-param-set   uuid=${VM_UUID} other-config:mac_seed=''

# Create the Network Interface

SWITCH="xenbr0"          # 
# The vendor prefix 00:16:3e is assigned to Xen in oid.txt
MAC="00:16:3e:12:34:56"  # Edit this value, each VIF must be different
## MAC="random"          # if you prefer randomly assigned addresses

# Get an ID for the virtual switch.  
# It is assumed that this is on the first bridge.
export NETUUID=$(xe network-list bridge=${SWITCH} --minimal)

export VIFUUID=$(xe vif-create \
                    vm-uuid=${VM_UUID} \
                    network-uuid=${NETUUID} \
                    mac=${MAC} \

# Create the Virtual Block Device that links the
# VDI to the VM
export VBDUUID=$(xe vbd-create \
                    vm-uuid=${VM_UUID} \
                    device=0 \
                    vdi-uuid=${VDIUUID} \
                    type=Disk \
                    mode=RW \
                    bootable=true )


  * Booting it up

# Starting the first time takes a while
xe vm-start uuid=${VM_UUID}
DOMID=$(xe vm-list uuid=${VM_UUID} params=dom-id --minimal)
/usr/lib/xen/bin/xenconsole ${DOMID}

# The main difference on the XCP side is that the disk image is
bigger.  The other steps are the same.  I threw in a condenced line to
start the VM and fork the console just because it's easier to paste
into a terminal window.

VDIUUID=$(xe vdi-create \
             sr-uuid=${SR_UUID} \
             name-label=${VMNAME}-vdi \
             type=system \
             virtual-size=21474836480 )

xe vm-start uuid=${VM_UUID} && \
    /usr/lib/xen/bin/xenconsole $(xe vm-list uuid=${VM_UUID} params=dom-id 

* Open Issues

I'm probably missing something really obvious here, but here are a
couple of issues that keep this from being a complete win.

  * Kernel ignoring/not-getting arguments from the domain builder.
    This means that the kernel can''t find its root device.

  * The pygrub loader won''t execute /boot/loader, so the kernel has to
    be called directly.  No access to loader.conf(5) and the
    customization it affords.

John D. "Trix" Farrar               __\\|//__               Basement.NET
trix@xxxxxxxxxxxx                   (` o-o ')   http://www.basement.net/
   GPG Key Fprint: 525F DBA7 1A62 E4C4 E642  DF95 384B B851 3CEF C10A

Attachment: pygrub.xcp-ufs-debug.patch
Description: Text Data

Attachment: pygrub.xcp-ufs-debug.patch.sig
Description: Text document

Attachment: pgpK6UZXKebPD.pgp
Description: PGP signature

Xen-users mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.