XenServer: Citrix Cobbler, Part One

Ah, blissful zen when eating a citrus cobbler...no, wait, I guess I misread that title, lol. Seriously, though, like many of you out there who have played with Citrix XenServer, you have found dozens of site out there that will teach you how to automate a Citrix XenServer installation (also referred to as unattended or PXE installation, etc). In Part One of this topic, I will explain how I manipulated cobbler, which is predominantly for RedHat-based "kickstart" installations, to install Citrix XenServer 5.6 using the automated answer file. Part Two will discuss "kickstarting" the Xen-distribution of the CentOS via cobbler. Both will assume that you have some rudimentary knowledge of how cobbler works (if not, cobbler documentation is fairly straightforward and easy to fin online).

First, there are plenty of references on how to automate a Xenserver installation scattered throughout the web. From what I can tell, it has not changed much throughout the 4.x to 5.x versions. The version of Citrix XenServer that I started using was 5.6.0, so my knowledge of the Citrix PXE boot install comes from this document.

Second, I will assume you have some rudimentary knowledge of cobbler if you are thinking of using the technique I will be discussing in this blog. By rudimentary, I assume that you've at least tried at least a few "cobbler import" and "cobbler profile add" calls along with the creation of one or two kickstart templates. If you are more advanced, perhaps you've even written some cheetah templates and/or created your own sync-triggers. Regardless, you should understand that cobbler was designed first-and-foremost to handle Redhat-based distributions. Xenserver is compatible with Redhat, but the PXE syntax used in the kickstart is not compliant, nor safe to allow cobbler to manage automatically, which will be explained below.

For the purposes of this exercise, we will assume that all distros, profiles and related files will utilze the following name variable:

    XNAME=citrix560

1. cobbler distro

Citrix XenServer for Linux comes with two ISO files: the XenServer-5.6.0-install-cd.iso and the XenServer-5.6.0-linux-cd.iso. While you can import the installation ISO, I would not recommend importing the Linux ISO because it will be created in a different location in the ks_mirror than you need it to be. The best method for "importing" the distribution into the cobbler ks_mirror is to simply mount the ISOs and copy them appropriately, as follows:

    KSDIR=/var/www/cobbler/ks_mirror/$XNAME
    mount -o loop XenServer-5.6.0-install-cd.iso /mnt/xen
    mount -o loop XenServer-5.6.0-linux-cd.iso /mnt/sub
    mkdir $KSDIR
    rsync -av /mnt/xen/ $KSDIR/
    rsync -av /mnt/sub/packages.linux $KSDIR/

The problem with NOT using the import function, of course, is that it will not create the distro json settings required --- of course, since this is not really Redhat compliant, the assumptions that cobbler makes regarding the kernel and initrd will be invalid, so you will need to follow these steps anyways:

    cobbler distro add --name=$XNAME --initrd=$KSDIR/boot/xen.gz --kernel=$KSDIR/boot/isolinux/mboot.c32

For convenience of the profile kickstart scripts, it is also advisable to create the symlink and kickstart metadata that the cobbler-import step does:

    ln -s $KSDIR ${KSDIR/ks_mirror/links}
    cobbler distro edit --name=$XNAME --ksmeta="tree=http://@@http_server@@/cblr/links/$XNAME"

2. cobbler profile

The automatic import also creates a default profile for each distro, which can be done manually with the following command:

    KSFILE=/var/lib/cobbler/kickstarts/$XNAME.ks
    cobbler profile add --name=$XNAME --distro=$XNAME --kickstart=$KSFILE

If you also have public/custom repos that you have retrieved/created that will be compatible with XenServer, append the following to the profile command:

    --repos='epel5 elff5 yum5'

where the repo names above represent the repos that you arbitrarily created (in the example above, the names represent Extra Packages for Enterprise Linux 5.x, Enterprise Linux Fast Forward 5.x, and Custom Yum Repo for RHEL/CentOS 5.x)


3. cobbler kickstart as answerfile

In normal Redhat kickstarts, the "tree" metadata define above serves as the location where the ISO can be retrieved via HTTP. For our XenServer process, we will utilize this HTTP method to retrieve the answerfile. So, instead of a normal $XNAME.ks "kickstart" file, the simplest "answerfile" would use DHCP:

<installation>
    <primary-disk>sda</primary-disk>
    <keymap>us</keymap>
    <root-password>topsecretword</root-password>
    <source type="url">http://$server/cblr/links/$distro</source>
    <post-install-script type="url">
        http://$server/cblr/aux/citrix/post-install
    </post-install-script>
    <admin-interface name="eth0" proto="dhcp" />
    <timezone>UTC</timezone>
    <hostname>$hostname</hostname>
</installation>

The $variables listed will be resolved by cobbler to the values within the profile report, and the post-install script is simply a file you place in /var/www/cobbler/aux/citrix. Obviously, sda and eth0 can be altered accordingly, depending on your preferences.

If you have detailed static interface info in your cobbler system, you may want to utilize that instead of DHCP, so that cheetah syntax would be:

<installation>
    <primary-disk>sda</primary-disk>
    <keymap>us</keymap>
    <root-password>topsecretword</root-password>
    <source type="url">http://$server/cblr/links/$distro</source>
    <post-install-script type="url">
        http://$server/cblr/aux/citrix/post-install
    </post-install-script>
    <admin-interface name="eth0" proto="static">
        #set $nic     = $interfaces["eth0"]
        #set $ip      = $nic["ip_address"]
        #set $netmask = $nic["subnet"]
        <ip>$ip</ip>
        <subnet-mask>$netmask</subnet-mask>
        <gateway>$gateway</gateway>
    </admin-interface>
    <timezone>UTC</timezone>
    <hostname>$hostname</hostname>
</installation>

4. cobbler synx/triggers

With what you have set up thus far, you would be able to create the PXE configuration just by running:

    cobbler sync

This basically clears out the old data files and regenerates all the dns, dhcp, distro images, and pxelinux configurations. As such, the following entry will appear in /tftpboot/pxelinux.cfg/default:

LABEL citrix560
    kernel /images/citrix560/mboot.c32
    MENU LABEL citrix560
    append initrd=/images/citrix560/xen.gz ksdevice=bootif lang=  kssendmac text  ks=http://1.2.3.4/cblr/svc/op/ks/profile/citrix560
    ipappend 2

As you know, the PXE configuration for Citrix XenServer is not identical to that of a Redhat kickstart, the append line should read:

    append /images/citrix560/xen.gz dom0_mem=752M com1=115200,8n1 console=com1,vga --- /images/citrix560/vmlinuz xencons=hvc console=hvc0 console=tty0 answerfile=http://1.2.3.4/cblr/svc/op/ks/profile/citrix560 install --- /images/citrix560/install.img

Those of you well-versed in standard kickstarts were probably wondering earlier why I did not set kernel=vmlinuz and initrd=install.img above, but now you see why. Those of you who are well-versed in cobbler are probably now considering adding those missing append fields into the kernel-options --- but I have already tried that and the results are not what you want. Basically, the options will be space-delimited, parsed as a list and rearranged alphabetically, which does not work properly with Citrix automated installs (believe me, I tested this extensively). The syntax of that append line is very specific, which is why the xen.gz was configured as the initrd option so that it would appear first. Also, the /images/ directory is missing the remaining files needed for PXE installation. Both of these factors need to be corrected AFTER cobbler syncs up the default PXE file, so we can easily make use of cobbler's post-sync triggers.

Just make a /var/lib/cobbler/triggers/sync/post/citrix.sh script that contains the following:

#!/bin/bash
libdir=/var/lib/cobbler
srcdir=/var/www/cobbler
dstdir=/tftpboot
for profile in $( grep -l citrix560.ks $libdir/config/profiles.d/* ); do
    json=${profile##*/}
    name=${json%.json}
    [[ -d $dstdir/images/$name ]] && \
    rsync -av $srcdir/ks_mirror/$name/{install.img,boot/vmlinuz} \
        $dstdir/images/$name/
done
for file in $( grep -l xen.gz $dstdir/pxelinux.cfg/* ); do
    sed -i'' -e 's!initrd=\(/images/.*/\)\(xen.gz \)ks.*ks=\(.*\)$!\1\2dom0_mem=752M com1=115200,8n1 console=com1,vga --- \1vmlinuz xencons=hvc console=hvc0 console=tty0 answerfile=\3 install --- \1install.img!;' $file
done

Basically, this post-sync trigger finds all occurrence of the specified .ks file in the cobbler profiles.d directory and clones all the necessary XenServer files to /tftpboot/images/$profile (this example assumes that my distro and profile share the same name, which they do). Then it locates all the PXE configurations that reference xen.gz and rewrites the Redhat append line into a Citrix append line. That's pretty much all you need to automate a Citrix XenServer installation. The next part is about the post-install scripts that the Citrix answerfile referenced above if you plan on running things after the XenServer comes up.


5. post-installation

Now the answerfile referenced http://$server/cblr/aux/citrix/post-install, which means that you will need a /var/www/cobbler/aux/citrix/post-install file. Since I use DHCP, I can locate the cobbler server name in the syslogs and use that to disable netboot (to prevent the host from PXE boot upon the next reboot):

#!/bin/bash
server=$( grep DHCPACK /var/log/messages | tail -1 | awk '{ print $NF }' )
system=$( hostname )
# Disable pxe (disable netboot)
wget -O /dev/null "http://$server/cblr/svc/op/nopxe/system/$system"

If you need access to certain system information during the post-install, you can make curl/wget references to cheetah templates within the /var/www/cobbler/aux/citrix directory to retrieve those variables in a script. For instance, let's assume you want to set the default gateway using the cobbler info. You can create /var/www/cobbler/aux/citrix/gateway.template:

#!/bin/bash
# retrieve cobbler variables
#set $gateway = $interfaces['eth1']['static_routes'][0].replace(':','/').split('/')[-1]
route add default gw $gateway

Then add it to the profile or the system using:

cobbler system edit --name=$system --template-files="/var/www/cobbler/aux/citrix/gateway.template=/alias"

Then you can reference it in your post-install script:

curl http://$server/cblr/svc/op/template/system/$system/path/_alias

If you want to add some post-install process that need to take place after the initial reboot, then have the curl destination drop them into /etc/firstboot.d as initrc scripts (of course, if you do this during the Citrix automated installation phase, you need to chroot it as /tmp/root/etc/firstboot.d/99-*). I create a lot of XE scripts in firstboot to handle bonding, default gateway, etc, but that is a topic for another day.