Xen 4.3 mid-release roadmap update

At XenSummit in August, I talked about the new planning process that Xen.org is experimenting with. This is apparently a pretty hot topic, as that presentation on slideshare.net has received over 72,000 views! We’re about half-way through the planned 4.3 development cycle, and while I’ve been regularly tracking the status of various features on the xen-devel mailing list, and also updating them on the 4.3 roadmap wiki page, I thought it might be useful just to give an update on how things are progressing.

As a reminder, we were aiming overall for a 9-month release cycle. Xen 4.2 was officially released on September 15th 2012, so we’re aiming for a release around June 15th 2013. Estimating based on history that there would be about 6 weeks worth of RCs, and 6 weeks of bug-fixing and stabilization, that puts the feature freeze at March 25th 2013 — a little over a month and a half away.
The main goal of this whole release management process is to help developers (and other interested parties) know where things stand, so that we can make informed decisions about what it’s important to invest time in doing. We had a fairly impressive list of features that we thought might be do-able in the 4.3 timeframe. So how are we doing on them?
The good news is that a lot of the key features are either already finished, or well on track to making it into the release. Features mentioned in the presentation which are on track for the release include the following:

• PVH, the new virtualization mode being implemented by Mukesh Rathor at Oracle, which is fully PV but takes advantage of some of the HVM hardware extensions.
• Event channel scalability Wei Liu has prototype patches for a 3-level event channel.
• NUMA affinity Dario Faggioli has posted the third (and hopefully final) version of these patches.
• Persistent grants for blk devices Roger Pau Monne has already upstreamed the Linux side; qemu patches have already been submitted and are on track for acceptance.
• Using qemu-upstream as the default The main feature requirement for this is Linux stub domains, which Anthony Perard has been working on and should be ready by the feature freeze. There are still a number of other bugs to iron out, but this should be ready by June.
• ARM v7 (32-bit) Stefano Stabellini and Anthony Perard have been working really hard at knocking down bugs and getting functionality working on their various ARM v7 platforms. They’re hoping to have a mostly-working system able to be demonstrated by the Linaro event in Hong Kong in March.
• Arm v8 (64-bit) tech preview There’s no hardware available yet for this; but Ian Campbell has been working steadily towards getting basic “tech preview” functionailty working in the ARM v8 simulator. Both of the ARM projects are going to be really key for Xen going forward, as ARM moves into the server space.
• USB for HVM guests George Dunlap has already posted patches, and just needs to clean them up in response to comments.
• 16 TiB limit Jan Buelich’s patches have already been accepted.
• Backend script rationalization Roger Pau Monne and Ian Jackson have gotten the infrastructure for backend scripts sorted out; this should make it possible for administrators to write custom scripts to be used in setting up or tearing down special back-ends, like iSCSI and so on. It’s also a prerequisite for supporting new features like openvswitch integration.

Along the way, we’ve picked up a few more key features:

• libvirt libxl driver Jim Fehlig’s team at SuSE have been working hard at getting the libxl driver for libxl as functional as the xend driver, and we’ve already found and fixed a number of issues with the interface. There should be no problem getting it finished in time for the 4.3 timeframe. This is a key component in making it easy to integrate Xen into other virtualization-related projects, such as OpenStack.
• Allow XSM to overried IS_PRIV checks Daniel De Graf at the NSA has been working hard to extend the XSM framework to allow it to be used to determine whether a given guest is allowed to do a privileged operation. This lays the groundwork for having a much more flexible level of domain 0 disaggregation, since any VM can be given fine-grained rights to do privileged operations on a specific subset of guests. These patches have already been accepted.
• vtpm enhancements Matthew Fioravante at Johns Hopkins University has been working to improve vtpm support; in particular, in Xen 4.3 we’ll be moving to a “stubdomain-only” model, unifying the vtpm system and making it much more secure.
• osstest Ian Jackson has openened up the code to the “push gate” regression tester that runs on all xen-unstable commits before being pushed into the public tree. This means that developers and other interested parties can now use it to run their own tests, and more importantly, can write tests of their own to be submitted to run on the push gate.
• qxl spice support qxl is a 3d acceleration framework for Spice. This is already implemented in qemu-upstream, and just requries the appropriate plumbing in xl and libxl to get it to work. Community member Zhou Peng posted initial patches, and community member Fabio Fantoni has taken them up and cleaned them up. There’s no reason to think they won’t be accepted by the release.

This is really an impressive amount of work — Xen 4.3 will undoubtedly be an excellent release.
That said, there are a number of features which were on the initial list which are pretty unlikely to make it. One of the most disappointing of these is blktap3. Thanos Makatos has been working diligently on getting it upstream. However, his current role only allows him a few hours per week to work on it. The work is progressing steadily, but at the moment it is very unlikely to make it into Xen 4.3. This means that all of the features that blktap can provide, including snapshotting, vhd support, and so on, will still be more difficult to use for pvops kernel users.
Full VM snapshotting and cloning are blocked on some prerequistite internal functionality (mm queues, for the curious). Storage migration is a possibility, but unfortunately there’s not anyone actively looking at it at the moment. The engineer who was going to be doing pvaudio has had his team go through some changes in strategic focus, and so this is unlikely to be done for 4.3.
There are a number of other features that require a relatively small amount of work, but that the core Xen developers just don’t have the time to do. These include things like adding support for openvswitch, VM import / export, and PVUSB support for xl. If you’ve been looking for a way to get involved in Xen development, these may be bite-sized projects that you can get involved in, and leave your mark on the Xen project.