Tuesday, April 12, 2011

XenServer: Citrix Cobbler, Part Two

Originally, I was only going to cover cobbler kickstarts of XenServer. But then, I figured, if I explained how I was automating my ant farm, it would be irresponsible of me not to cover how to automate your ants. Otherwise, you would build one ant manually (aka virtual image) and clone it repeatedly for your additional ants (and, morally, I object to ant cloning). As stated in the previous blog, cobbler is predominantly for RedHat-based installations as it is built around PXE kickstarts (though there are many people who have tried to adapt Debian-based pre-seeds into a kickstart-like infrastructure), so this blog will discuss how I provision CentOS on my XenServer using the xen portion of the CentOS distribution.

The most important script necessary for this to work is my xen virtual machine manager bash script, which is essentially a bunch of Citrix $XE command line options wrapped into a bash getopts script. My suggestion would be to copy it to /usr/local/bin so that it can be executed without having the specify the path --- use whatever configuration management software you want (i.e. cfengine, puppet, chef, etc) or have it installed as part of your cobbler post-install from /var/www/cobbler/aux/citrix, as specified in Part One of this blog. Of course, you may also want to just copy it over manually.

The second most important thing is to configure a privileged user that can ssh to the Citrix XenServer and execute the xen virtual machine manager script as root. Originally, I had configured all the ssh commands to run as root@xenserver, but have since altered everything to utilize sudo and non-root ssh keypairs. We will not be discussing sudoers or authorized_keys in this blog, so if you do not know how to handle that, you should not continue reading. For the sake of the rest of this example, we will refer to this priviledged user as 'uber'.

Now, some people may just create a basic xen virtual machine and clone it as needed for future installations. That will not be covered here, though, the technique is similar and, if you understand all the steps, you should be able to make the cobbler and bash modifications to handle that as well. I prefer to kickstart everything from scratch and then apply configuration management upon post-install that will customize the process accordingly.


1. create Citrix XenServer and install xen virtual machine manager script

I did mention before that this was the MOST important step, so I am listing it again as Step 1. Build your own Citrix server or use the technique I described in Part One. Create your own script, or just use mine.


2. create and configure 'uber' user

On the cobbler host, you need to create and generate ssh keypairs (this example uses RSA) for ~uber/.ssh but on the Citrix host, you need to add the public key to authorized_keys. You will also have to make sure that 'uber' is allowed in sudoers to execute the xen virtual machine manager script.


3. create custom cobbler power template

You can manipulate almost anything in cobbler using cheetah templates. Create /etc/cobbler/power/power_citrix.template as follows:

# By default, cobbler builds the kickstart tree for all systems
# more reliably than the distro being attached to the interface
#set treelist=$mgmt_parameters['tree'].split('/')

#if $power_mode == 'on'
ssh -i ~$power_user/.ssh/id_rsa $power_user@$power_address sudo xenvm_mgr.sh \
    -S "$server" -D "$treelist[-1]" \
    -m "xenbr0=$interfaces['eth0']['mac_address'],xapi1=$interfaces['eth1']['mac_address']" \
    -s "$virt_path" \
    -V "$virt_file_size" \
    -C "$virt_cpus" \
    -M "$virt_ram" \
    -c "$name"
#end if

# xenvm_mgr must exit 0 when VM exists, or poweroff will fail for new VMs
#if $power_mode == 'off'
ssh -i ~$power_user/.ssh/id_rsa $power_user@$power_address sudo xenvm_mgr.sh \
    -d \
    -x "$name"
#end if

All the $variables specified are specific to cobbler, but you may need to adjust "Local storage" according to how you configured the Citrix XenServer or modify xenbr0/xapi1 to different interface numbers, depending on your architecture. My xen virtual machine manager script has wildcard matching so you do not have to know the precise storage name (but you will have to be careful as it will take the first match listed). There are also tests that check that available cpu, memory and disk are available or the new virtual machine is destroyed and the xenvm_mgr will exit non-zero and cause the poweron to retry a few times before failing.


4. configure the cobbler profile

Assuming you imported a Redhat-based distribution, i.e. RHEL 5.5, it should have created two distinct profiles like:

   rhel5-arch
   rhel5-xen-arch

because the distribution contains the kernel and initrd for xen virtual instances. You will "cobbler system add" your new virtual instance using this profile.


5. configure the cobbler system (aka, the xen virtual instance)

Whether you utilize the following options when you do the "cobbler system add" or update them afterwards using "cobbler system edit" is of no consequence to me, I will illustrate using the edit method:

cobbler system edit --name vmname --power-user=uber --power-type=citrix --power-address=citrixdns

What you need to note here is that the power-type corresponds directly to the power_type.template create in Step 3 and the power-address is the dns name or the ip address of your Citrix XenServer. If you need to modify the VM settings, you can override them via cobbler as well:

cobbler system edit --name vmname --virt-cpu=2 --virt-file-size=100 --virt-ram=4096 --virt-path="Local storage"


That's all! The kickstart info is retrieved from the imported profile. The cheetah template now handles the following (so be careful how you use this):

create a new virtual machine using the xen virtual machine manager:

cobbler poweron --name vmname

destroys existing virtual machine using xen virtual machine manager:

cobbler poweroff --name vmname


If you do not want the poweroff to destroy, modify the cheetah template. If you want to add a reboot case in the cheetah template, go right ahead. This just about covers it, I think. Have fun extending the cobbler power templates to handle other virtualizations.

Thursday, April 7, 2011

XenServer: Citrix Cobbler, Part One

Ah, blissful zen when eating a citrus cobbler...no, wait, I guess I misread that title, lol. Seriously, though, like many of you out there who have played with Citrix XenServer, you have found dozens of site out there that will teach you how to automate a Citrix XenServer installation (also referred to as unattended or PXE installation, etc). In Part One of this topic, I will explain how I manipulated cobbler, which is predominantly for RedHat-based "kickstart" installations, to install Citrix XenServer 5.6 using the automated answer file. Part Two will discuss "kickstarting" the Xen-distribution of the CentOS via cobbler. Both will assume that you have some rudimentary knowledge of how cobbler works (if not, cobbler documentation is fairly straightforward and easy to fin online).

First, there are plenty of references on how to automate a Xenserver installation scattered throughout the web. From what I can tell, it has not changed much throughout the 4.x to 5.x versions. The version of Citrix XenServer that I started using was 5.6.0, so my knowledge of the Citrix PXE boot install comes from this document.

Second, I will assume you have some rudimentary knowledge of cobbler if you are thinking of using the technique I will be discussing in this blog. By rudimentary, I assume that you've at least tried at least a few "cobbler import" and "cobbler profile add" calls along with the creation of one or two kickstart templates. If you are more advanced, perhaps you've even written some cheetah templates and/or created your own sync-triggers. Regardless, you should understand that cobbler was designed first-and-foremost to handle Redhat-based distributions. Xenserver is compatible with Redhat, but the PXE syntax used in the kickstart is not compliant, nor safe to allow cobbler to manage automatically, which will be explained below.

For the purposes of this exercise, we will assume that all distros, profiles and related files will utilze the following name variable:

    XNAME=citrix560


1. cobbler distro

Citrix XenServer for Linux comes with two ISO files: the XenServer-5.6.0-install-cd.iso and the XenServer-5.6.0-linux-cd.iso. While you can import the installation ISO, I would not recommend importing the Linux ISO because it will be created in a different location in the ks_mirror than you need it to be. The best method for "importing" the distribution into the cobbler ks_mirror is to simply mount the ISOs and copy them appropriately, as follows:

    KSDIR=/var/www/cobbler/ks_mirror/$XNAME
    mount -o loop XenServer-5.6.0-install-cd.iso /mnt/xen
    mount -o loop XenServer-5.6.0-linux-cd.iso /mnt/sub
    mkdir $KSDIR
    rsync -av /mnt/xen/ $KSDIR/
    rsync -av /mnt/sub/packages.linux $KSDIR/

The problem with NOT using the import function, of course, is that it will not create the distro json settings required --- of course, since this is not really Redhat compliant, the assumptions that cobbler makes regarding the kernel and initrd will be invalid, so you will need to follow these steps anyways:

    cobbler distro add --name=$XNAME --initrd=$KSDIR/boot/xen.gz --kernel=$KSDIR/boot/isolinux/mboot.c32

For convenience of the profile kickstart scripts, it is also advisable to create the symlink and kickstart metadata that the cobbler-import step does:

    ln -s $KSDIR ${KSDIR/ks_mirror/links}
    cobbler distro edit --name=$XNAME --ksmeta="tree=http://@@http_server@@/cblr/links/$XNAME"


2. cobbler profile

The automatic import also creates a default profile for each distro, which can be done manually with the following command:

    KSFILE=/var/lib/cobbler/kickstarts/$XNAME.ks
    cobbler profile add --name=$XNAME --distro=$XNAME --kickstart=$KSFILE

If you also have public/custom repos that you have retrieved/created that will be compatible with XenServer, append the following to the profile command:


    --repos='epel5 elff5 yum5'

where the repo names above represent the repos that you arbitrarily created (in the example above, the names represent Extra Packages for Enterprise Linux 5.x, Enterprise Linux Fast Forward 5.x, and Custom Yum Repo for RHEL/CentOS 5.x)


3. cobbler kickstart as answerfile

In normal Redhat kickstarts, the "tree" metadata define above serves as the location where the ISO can be retrieved via HTTP. For our XenServer process, we will utilize this HTTP method to retrieve the answerfile. So, instead of a normal $XNAME.ks "kickstart" file, the simplest "answerfile" would use DHCP:

<installation>
    <primary-disk>sda</primary-disk>
    <keymap>us</keymap>
    <root-password>topsecretword</root-password>
    <source type="url">http://$server/cblr/links/$distro</source>
    <post-install-script type="url">
        http://$server/cblr/aux/citrix/post-install
    </post-install-script>
    <admin-interface name="eth0" proto="dhcp" />
    <timezone>UTC</timezone>
    <hostname>$hostname</hostname>
</installation>

The $variables listed will be resolved by cobbler to the values within the profile report, and the post-install script is simply a file you place in /var/www/cobbler/aux/citrix. Obviously, sda and eth0 can be altered accordingly, depending on your preferences.

If you have detailed static interface info in your cobbler system, you may want to utilize that instead of DHCP, so that cheetah syntax would be:

<installation>
    <primary-disk>sda</primary-disk>
    <keymap>us</keymap>
    <root-password>topsecretword</root-password>
    <source type="url">http://$server/cblr/links/$distro</source>
    <post-install-script type="url">
        http://$server/cblr/aux/citrix/post-install
    </post-install-script>
    <admin-interface name="eth0" proto="static">
        #set $nic     = $interfaces["eth0"]
        #set $ip      = $nic["ip_address"]
        #set $netmask = $nic["subnet"]
        <ip>$ip</ip>
        <subnet-mask>$netmask</subnet-mask>
        <gateway>$gateway</gateway>
    </admin-interface>
    <timezone>UTC</timezone>
    <hostname>$hostname</hostname>
</installation>

4. cobbler synx/triggers

With what you have set up thus far, you would be able to create the PXE configuration just by running:

    cobbler sync

This basically clears out the old data files and regenerates all the dns, dhcp, distro images, and pxelinux configurations. As such, the following entry will appear in /tftpboot/pxelinux.cfg/default:

LABEL citrix560
    kernel /images/citrix560/mboot.c32
    MENU LABEL citrix560
    append initrd=/images/citrix560/xen.gz ksdevice=bootif lang=  kssendmac text  ks=http://1.2.3.4/cblr/svc/op/ks/profile/citrix560
    ipappend 2

As you know, the PXE configuration for Citrix XenServer is not identical to that of a Redhat kickstart, the append line should read:

    append /images/citrix560/xen.gz dom0_mem=752M com1=115200,8n1 console=com1,vga --- /images/citrix560/vmlinuz xencons=hvc console=hvc0 console=tty0 answerfile=http://1.2.3.4/cblr/svc/op/ks/profile/citrix560 install --- /images/citrix560/install.img

Those of you well-versed in standard kickstarts were probably wondering earlier why I did not set kernel=vmlinuz and initrd=install.img above, but now you see why. Those of you who are well-versed in cobbler are probably now considering adding those missing append fields into the kernel-options --- but I have already tried that and the results are not what you want. Basically, the options will be space-delimited, parsed as a list and rearranged alphabetically, which does not work properly with Citrix automated installs (believe me, I tested this extensively). The syntax of that append line is very specific, which is why the xen.gz was configured as the initrd option so that it would appear first. Also, the /images/ directory is missing the remaining files needed for PXE installation. Both of these factors need to be corrected AFTER cobbler syncs up the default PXE file, so we can easily make use of cobbler's post-sync triggers.

Just make a /var/lib/cobbler/triggers/sync/post/citrix.sh script that contains the following:

#!/bin/bash
libdir=/var/lib/cobbler
srcdir=/var/www/cobbler
dstdir=/tftpboot
for profile in $( grep -l citrix560.ks $libdir/config/profiles.d/* ); do
    json=${profile##*/}
    name=${json%.json}
    [[ -d $dstdir/images/$name ]] && \
    rsync -av $srcdir/ks_mirror/$name/{install.img,boot/vmlinuz} \
        $dstdir/images/$name/
done
for file in $( grep -l xen.gz $dstdir/pxelinux.cfg/* ); do
    sed -i'' -e 's!initrd=\(/images/.*/\)\(xen.gz \)ks.*ks=\(.*\)$!\1\2dom0_mem=752M com1=115200,8n1 console=com1,vga --- \1vmlinuz xencons=hvc console=hvc0 console=tty0 answerfile=\3 install --- \1install.img!;' $file
done

Basically, this post-sync trigger finds all occurrence of the specified .ks file in the cobbler profiles.d directory and clones all the necessary XenServer files to /tftpboot/images/$profile (this example assumes that my distro and profile share the same name, which they do). Then it locates all the PXE configurations that reference xen.gz and rewrites the Redhat append line into a Citrix append line. That's pretty much all you need to automate a Citrix XenServer installation. The next part is about the post-install scripts that the Citrix answerfile referenced above if you plan on running things after the XenServer comes up.


5. post-installation

Now the answerfile referenced http://$server/cblr/aux/citrix/post-install, which means that you will need a /var/www/cobbler/aux/citrix/post-install file. Since I use DHCP, I can locate the cobbler server name in the syslogs and use that to disable netboot (to prevent the host from PXE boot upon the next reboot):

#!/bin/bash
server=$( grep DHCPACK /var/log/messages | tail -1 | awk '{ print $NF }' )
system=$( hostname )
# Disable pxe (disable netboot)
wget -O /dev/null "http://$server/cblr/svc/op/nopxe/system/$system"

If you need access to certain system information during the post-install, you can make curl/wget references to cheetah templates within the /var/www/cobbler/aux/citrix directory to retrieve those variables in a script. For instance, let's assume you want to set the default gateway using the cobbler info. You can create /var/www/cobbler/aux/citrix/gateway.template:

#!/bin/bash
# retrieve cobbler variables
#set $gateway = $interfaces['eth1']['static_routes'][0].replace(':','/').split('/')[-1]
route add default gw $gateway

Then add it to the profile or the system using:

cobbler system edit --name=$system --template-files="/var/www/cobbler/aux/citrix/gateway.template=/alias"

Then you can reference it in your post-install script:

curl http://$server/cblr/svc/op/template/system/$system/path/_alias

If you want to add some post-install process that need to take place after the initial reboot, then have the curl destination drop them into /etc/firstboot.d as initrc scripts (of course, if you do this during the Citrix automated installation phase, you need to chroot it as /tmp/root/etc/firstboot.d/99-*). I create a lot of XE scripts in firstboot to handle bonding, default gateway, etc, but that is a topic for another day.