Improved Xen support in FreeBSD

As most FreeBSD users already know, FreeBSD 10 has just been released, and we expect this to be a very good release regarding Xen support. FreeBSD with Xen support includes many improvements, including several performance and stability enhancements that we expect will greatly please and interest users. With many bug fixes already completed, the following description only focuses on new features.

New vector callback

Previous releases of FreeBSD used an IRQ interrupt as the callback mechanism for Xen event channels. While it’s easier to setup, using a IRQ interrupt doesn’t allow to inject events to specific CPUs, basically limiting the use of event channels in disk and network drivers. Also, all interrupts were delivered to a single CPU (CPU#0), not allowing proper interrupt balancing between CPUs.
With the introduction of the vector callback, events can now be delivered to any CPU, allowing FreeBSD to have specific per-CPU interrupts for PV timers and PV IPIs, and balancing the others across the several CPU usually available on a domain.

PV timers

Thanks to the introduction of the vector callback, now we can make use of the Xen PV timer, which is implemented as a per-CPU singleshot timer. This alone doesn’t seem like a great benefit, but it allows FreeBSD to avoid making use of the emulated timers, greatly reducing the emulation overhead and the cost of unnecessary VMEXITs.

PV IPIs

As with PV timers, the introduction of the vector callback allows FreeBSD to get rid of the bare metal IPI implementation, and instead route IPIs through event channels. Again, this allows us to get rid of the emulation overhead and unnecessary VMEXITS, providing better performance.

PV disk devices

FLUSH/BARRIER support has been recently added, together with a couple of fixes that allow FreeBSD to run with a CDROM driver under XenServer (which was quite of a pain for XenServer users).

Support for migration

With these new features, migration doesn’t break since it has been reworked to handle the fact that timers and IPIs are also paravirtualized now.

Merge of the XENHVM config into GENERIC

One of the most interesting improvements from a user/admin point of view (and something similar to what the pvops Linux kernel is already doing), the GENERIC kernel on i386 and amd64 now includes full Xen PVHVM support, so there’s no need to recompile a Xen-specific kernel. When run as a Xen guest, the kernel will detect the available Xen features and automatically make use of them in order to obtain the best possible performance.
This work has been done in conjunction between Spectra Logic and Citrix.