Using Grub 2 as a bootloader for Xen PV guests

Background: Introduction to Xen PV Bootloaders

In the very early days of Xen it was necessary for the host (domain 0)
administrator to explicitly supply a kernel (and perhaps initial
ramdisk) from the domain 0 filesystem in order to start a new guest.
This mostly worked and for some use cases, i.e. those where the host
admin wants very strict control over what each guest runs, was
desirable and remains so today.
However for other use cases it was rather inflexible since it meant
that the host administrator needed to be involved in what many
considered to be a guest administrator, or even distribution level,
decision i.e. the selection of which kernel to run, with what
parameters etc.
The first solution to this problem to come along was
pygrub. pygrub is an application (written in Python) which
can be used by the Xen toolstack in domain 0 as a kind of
pseudo-bootloader. pygrub will open the guest file system (using a
userspace filesystem access library), extract a configuration file,
parse it and extract the referenced kernel, (optional) initial ramdisk
and kernel command line to be booted almost as if they were provided
from the domain 0 filesystem.
pygrub initially supported configuration files in the GNU
grub (known as “grub-legacy” by upstream today) menu.lst
syntax, which was a common option for distributions at the
time. pygrub even supported an optional curses frontend menu similar
to the native grub legacy interface, allowing boot time selection
between multiple kernels, as well as editing of the command line etc.
This allowed host admins configure a guest to use pygrub and thereby
delegate the management and selection of the guest kernel to the guest
administrator. Guest administrators could use the usual tools which
they expect (i.e. distribution packaging and grub integration) and
configuration file syntax from their non-Xen systems.
Since it was introduced pygrub has gone from supporting the
grub-legacy menu.lst configuration files to supporting a variety of
configuration files including the syntaxes of GNU grub 2,
[syslinux][syslinux] (which includes pxelinux, syslinux, and isolinux)
and LiLo. pygrub remains part of the Xen releases today (and
will be for the foreseeable future) however it has some short comings:

• The configuration file parsers are simplistic and cannot cope with
  all of the constructs which can be used in practice. In particular
  the grub 2 syntax is a particularly expressive shell-like language
  which pygrub can only cope with basic elements of and that support
  is fragile requiring updates as grub-upstream and distributions find
  new ways to make use of the flexibility allowed. This is
  particularly troublesome now that grub 2 is the default in many
  distributions.
• It ultimately boots a potentially untrusted guest kernel as if it
  was a kernel supplied from the domain 0 filesystem (which would
  normally be somewhat implicitly trustworthy). This means that the
  code to build a domain now must take special care to treat the
  kernel image as hostile.

As a result of these shortcomings pvgrub was created (it’s an
unfortunate source of much confusion that pygrub and pvgrub differ
only in the descender on a single letter). pvgrub was a port of GNU
grub (AKA “grub-legacy”) to run as a Xen PV kernel. This had
several advantages:

• Since the pvgrub kernel is supplied by the host administrator it
  is trusted and can be expected not to deliberately attack the domain
  builder. All handling of the untrusted guest inputs now happens
  within guest context where the harm which can be done is contained.
• Since the code is actually real grub-legacy it has the same
  configuration file parser and features as grub running on a native
  system.

One minor downside of pvgrub is that it did not support syslinux
or LiLo configuration files (that would need to be achived via a PV
port of those respective bootloaders), although in practice they are
not so widely used so this was a minor shortcoming.
A second shortcoming was that it is not possible for a PV guest to
switch between 32- and 64-bit operation. This means that the user
needs to know a-priori the type of kernel which they will be booting
and the host administrator needs to provide the ability to select
between 32- and 64-bit builds of pvgrub.
The most serious problem today though is the move of most
distributions from grub-legacy to grub 2, with its radically
different architecture and configuration file syntax. This meant that
admins could no longer simply reuse their existing grub 2 based
workflows and distribution integration. Some workarounds have evolved
such as pv-grub-menu but something better was needed…

PV Grub 2

In November 2013 upstream grub maintainer Vladimir ‘phcoder’
Serbinenko announced that:

pvgrub2 has just became part of upstream grub as ports i386-xen and x86_64-xen.

This meant that it was now possible to compile the upstream grub 2
code base to run as a pvgrub2 Xen PV guest, in much the same way as
the original pvgrub-legacy port of Grub legacy.
This has the same advantages as the pvgrub-legacy port originally
had, except using the more modern grub 2 code base. In addition
since this support was part of upstream grub there was no fork to
maintain and therefore no risk that the Xen PV support would languish.
In the remainder of this blog post I’m going to explain how to install
and use pvgrub2 on your Xen hosts and guests.

Guest Setup

This guide assumes that the PV guest is already setup with a grub2
configuration file, as if it were a native system (i.e. usually
/boot/grub/grub.cfg). Many distribution installers (at least those
for distributions which use grub2) will do this automatically even
within a PV guest. If not then you may need to install manually,
e.g. on Debian by installing the grub-pc package.

Building and installing pvgrub2

Getting the source

The last release of grub was 2.00, released in June 2012, which is
before PV Xen support was added. Since then the grub development team
have released beta versions of grub 2.02. In particular the latest,
2.02~beta2, contains support for Xen. (Don’t be scared off by the beta
tag, in reality several distributions are shipping this version as
their primary bootloader).
Grub 2.02~beta2 can be downloaded from
http://alpha.gnu.org/gnu/grub/grub-2.02~beta2.tar.gz.
Alternatively you can fetch the code from the grub git repository:

git clone git://git.savannah.gnu.org/grub.git

Compiling

The file INSTALL in the code tree contains detailed information on how
to build grub, including a full list of dependencies.
Other than the obvious things, such as make and gcc and slightly
less obvious things such as flex and bison it is worth noting
that:

• compiling 64-bit grub on a 64-bit platform needs to be able to build
  some 32-bit components and therefore requires a biarch capable gcc
  (i.e. one which accepts the -m32 option, pretty much all 64-bit
  distribution gcc packages today include this feature in the default
  compiler) as well as a 32-bit libc (e.g. from the libc6-dev-i386
  package on Debian).
• the Xen headers must be installed, if you’ve installed Xen from
  distribution packages this might require you to install the relevant
  -dev package (e.g. libxen-dev in Debian). If you’ve installed Xen
  from source with make install then you should have the headers
  already.

The process is pretty much the standard configure/make/make install
routine associated with all autoconf based projects.
If you are compiling from git then you will need to start by
generating the configure script. This can be skipped if you are
using the tarball.

$ ./autogen.sh

Next, configure grub2 for use on an x86_64-xen platform:

$ ./configure --target=amd64 --with-platform=xen

To target 32-bit/i386 Xen domains instead use:

$ ./configure --target=i386 --with-platform=xen

The rest of this guide will assume x86_64-xen, but you can substitute
i386-xen throughout if you want to boot a 32-bit guest.
If you want to avoid the possibility of messing with the native
bootloader on the system then add --prefix=/opt/grub2 to the
configure parameters in order to install pvgrub2 in an out of the
way location (if you do this then you may want to add
/opt/grub2/sbin:/opt/grub2/bin to your $PATH or else remember to
give an explicit path to the relevant commands).
Once things are configured then:

$ make

Finally, as root:

# make install

Creating a basic pvgrub2 image

Now that grub is installed on the host the next step is to build the
actual pvgrub2 image which will be used to boot the guest. Grub is
highly modular and allows you to construct images with various
features embedded. It also supports loading additional modules at
runtime.
To create a basic image first create a file named grub.cfg which
contains:

normal (xen/xvda,msdos1)/boot/grub/grub.cfg

This tells grub to read the /boot/grub/grub.cfg configuration file
from the first partition of the xvda device (which must be using
MSDOS style partitioning) and run it using the normal
command interpreter.
Finally we can build the grub image using the grub-mkimage utility:

grub-mkimage -O x86_64-xen -c grub.cfg \
    -o grub-x86_64-xen.bin $prefix/lib/grub/x86_64-xen/*.mod

Where:

• -O x86_64-xen: Is the target platform
• -c grub.cfg: Includes the configuration file which created above
• -o grub-x86_64-xen.bin: Is the output file
• $prefix/lib/grub/x86_64-xen/*.mod: Includes all loadable modules
  in the image. ($prefix is wherever you installed grub to, which
  is /usr/local by default, or /opt/grub2 if you added the
  --prefix=/opt/grub2 option as discussed above).

The reason for the include *.mod is that since this image is going
to running in guest context it is not going to be able to load
additional modules from domain 0 at runtime, since it will only see
the guest filesystem.
Now you can start a guest using grub-x86_64-xen.bin as the
kernel. e.g. by writing in your guest configuration file:

kernel = "grub-x86_64-xen.bin"

This is all that is required. There is no need to give any other
ramdisk, bootloader, cmdline, root, extra etc options. You
will still need to configure the name, disks, network devices etc in
the normal way.
Then, assuming your in-guest grub.cfg is indeed located at
(xen/xvda,msdos1)/boot/grub/grub.cfg, you should be presented with
the usual grub menu when you connect to the guest console.

Loading grub.cfg from any partition

The above simple example is all well and good, but it is rather
inflexible and if the guest uses a separate /boot partition or
otherwise differs from the expectations encoded in the initial
configuration then it won’t find the real configuration file and you
will be dumped to a grub prompt.
Luckily the grub configuration file syntax is far richer than we’ve
used so far, so lets try something more flexible.
As well embedding a bootstrap configuration grub-mkimage is also
able to embed a memdisk (essentially an initramfs for the bootloader)
into the pvgrub2 image, we are going to use this in order to bootstrap
into a more capable grub shell.
First we need to create two configuration files which will be embedded
into the pvgrub2 image:
grub-bootstrap.cfg:

normal (memdisk)/grub.cfg

grub.cfg:

if search -s -f /boot/grub/grub.cfg ; then
        echo "Reading (${root})/boot/grub/grub.cfg"
        configfile /boot/grub/grub.cfg
fi
if search -s -f /grub/grub.cfg ; then
        echo "Reading (${root})/grub/grub.cfg"
        configfile /grub/grub.cfg
fi

The reason for using two configuration files in this manner is that
the inbuilt grub interpreter is rather simple, whereas features such
as if and search are only available from the more
feature complete normal interpreter. Bootstrapping into this more
complex grub.cfg allows us to cope with guests which have a separate
/boot partition by searching for a file named /boot/grub/grub.cfg
or /grub/grub.cfg on any partition. Note that ${root} is set by
the search command and should be entered literally, not substituted
while creating this file.
Next we need to embed the grub.cfg into a memdisk, which is really
just a tarball:

$ tar cf memdisk.tar grub.cfg

Finally we can build the grub image using the grub-mkimage utility:

grub-mkimage -O x86_64-xen \
    -c grub-bootstrap.cfg \
    -m memdisk.tar \
    -o grub-x86_64-xen.bin \
    $prefix/lib/grub/x86_64-xen

Where in addition to the example above we now use:

• -c grub-bootstrap.cfg: Includes the bootstrap configuration
• -m memdisk.tar: Includes our memdisk, which includes our more
  featureful configuration snippet.

On booting this grub image will now try much harder to find a
grub.cfg to run and will work on far more guests without special
tweaking.

Chainloading guest pvgrub1 from domain 0 pvgrub2

The above works well in many situations but has one major drawback
which is that the bootloader is still ultimately under the control of
the host admin. This may present a problem for example if the guest
admin wishes to run a guest which makes use of new features found in a
newer version of grub, or if there is a need to load a module which is
not included in the host grub image from the guest filesystem, since
the module may not be compatible with the running grub.
To address this issue the Xen team has published the Xen x86 PV
Bootloader Protocol specification which describes the
in-guest path where a PV bootloader, such as pvgrub2, should be
installed in order that a host grub can chainload it. Specifically the
bootloader should be installed to either /boot/xen/pvboot-i386.elf
or /boot/xen/pvboot-x86_64.elf depending on the guest bit size.
Support for this protocol can be implemented in the host grub using
the same grub-bootstrap.cfg + memdisk.tar method as above and a
grub.cfg which contains:

if search -s -f /boot/xen/pvboot-x86_64.elf ; then
        echo "Chainloading (${root})/boot/xen/pvboot-x86_64.elf"
        multiboot "/boot/xen/pvboot-x86_64.elf"
        boot
fi
if search -s -f /xen/pvboot-x86_64.elf ; then
        echo "Chainloading (${root})/xen/pvboot-x86_64.elf"
        multiboot "/xen/pvboot-x86_64.elf"
        boot
fi

On the guest side you can build and install a grub within the guest
just like for the host. Then run, as root, within the guest:

# grub-install --target=x86_64-xen
Installing for x86_64-xen platform.
grub-install: warning: no hints available for your platform. Expect reduced performance.
grub-install: warning: WARNING: no platform-specific install was performed.
Installation finished. No error reported.

(the two warnings can safely be ignored, there is no actual
performance impact)
Then the Grub image needs to be moved to the standardised location:

# mkdir /boot/xen/
# cp /boot/grub/x86_64-xen/core.elf /boot/xen/pvboot-x86_64.elf

Alternatively you can apply a patch (submitted
upstream but not yet applied) which causes grub-install to do the
right thing by default and rebuild grub-mkimage on your host.
For maximum flexibility you can combine both the chainloading and
loading the guest grub.cfg directly by putting both options in the
host grub’s memdisk grub.cfg, I recommend trying to chainload first
and only then falling back to loading a grub.cfg from the guest with:

if search -s -f /boot/xen/pvboot-x86_64.elf ; then
        echo "Chainloading (${root})/boot/xen/pvboot-x86_64.elf"
        multiboot "/boot/xen/pvboot-x86_64.elf"
        boot
fi
if search -s -f /xen/pvboot-x86_64.elf ; then
        echo "Chainloading (${root})/xen/pvboot-x86_64.elf"
        multiboot "/xen/pvboot-x86_64.elf"
        boot
fi
if search -s -f /boot/grub/grub.cfg ; then
        echo "Reading (${root})/boot/grub/grub.cfg"
        configfile /boot/grub/grub.cfg
fi
if search -s -f /grub/grub.cfg ; then
        echo "Reading (${root})/grub/grub.cfg"
        configfile /grub/grub.cfg
fi

Chainloading from pvgrub-legacy

What about all those host systems which provide only the legacy PV
grub by default, as is common in some cloud environments? Fortunately
it is possible to chainload pvgrub2 from pvgrub-legacy. Simply
create a grub-legacy configuration file in your guest at
/boot/grub/menu.lst which contains:

default 0
timeout 1
title Chainload grub2
kernel (hd0,0)/boot/xen/pvboot-x86_64.elf

You will need to adjust (hd0,0)/boot/xen/pvboot-x86_64.elf to suit
your actual partition layout (sadly I don’t know of any useful tricks
to find it automatically with grub-legacy).
This will cause the host provided pvgrub-legacy to chainload into a
guest supplied pvgrub2.

Distribution Integration

The above describes how to go about using pvgrub2 manually. Ideally
this would all happen automatically as part of the standard
distribution setup.

Debian

The forthcoming Debian 8.0 (Jessie) release will contain support
for both host and guest pvgrub2. This was added in version
2.02~beta2-17 of the package (bits were present before then, but
-17 ties it all together as described below).
The package grub-xen-host contains grub binaries configured for
the host, these will attempt to chainload an in-guest grub image
(following the specification and falling back to search for
a grub.cfg in the guest filesystems as described
above). grub-xen-host is Recommended by the Xen meta-packages in
Debian or can be installed by hand.
The package grub-xen-bin contains the grub binaries for both the
i386-xen and x86_64-xen platforms, while the grub-xen
package integrates this into the running system by providing the
actual pvgrub2 image (i.e. running grub-install at the appropriate
times to create an image tailored to the system) and integration with
the kernel packages (i.e. running update-grub at the right times),
so it is the grub-xen which should be installed in Debian guests.
At this time the grub-xen package is not installed automatically so
it will need to be done manually (something which perhaps could be
addressed for Stretch).

Others

I’m not aware of any other distributions which have integrated grub2
support for Xen. I’d be glad to be corrected or equally happy to help
advise on any integration efforts.