How the Hypervisor team manages releases

With the Xen Project Hypervisor 4.4 released a bit more than a month ago, the project is planning for version 4.5, so it’s a good time to outline how we manage releases.

New Release Manager: Welcoming Oracle’s Konrad Rzeszutek Wilk

But first I want to thank George Dunlap for successfully managing the 4.3 and 4.4 releases of the Xen Project Hypervisor. The Release Manager role is a volunteer role open to maintainers. George has stepped down from his role recently to find more time for coding and help bootstrap the CentOS virtualization SIG.
Konrad Rzeszutek Wilk has volunteered to take on the role for the 4.5 release. A big welcome and Thank You!
Konrad is Software Development Manager at Oracle. His group’s mission is to make Linux and Xen Project virtualization better and faster. As part of this work, Konrad has been the maintainer of the Xen Project subsystem in Linux, Xen Project maintainer and now also Release Manager for the 4.5 release of the Xen Project Hypervisor. Konrad has been active in the Linux and Xen Project communities for more than 6 years and was instrumental in adding Xen Project support to the Linux Kernel.

How we manage releases

As is the case for many open source projects, the Xen Project community does not maintain a committed roadmap as proprietary software vendors do. However, we do strive to accurately track development for new releases, with a predictable release cadence for major releases and maintenance releases. We aim to release the Xen Project Hypervisor every 6-7 months; historically we had a release cycle that ranged from 9 to 18 months.  The new Release Manager role has already proven instrumental delivering faster turnaround and predictability.

How contributors operate

Our best practice recommends that new features that intend to go into the next release are initially announced and discussed on the xen-devel mailing list. This eliminates the risk of multiple developers starting to implement a feature at the same time. The Release Manager will usually follow up and ask whether the feature should be included into the backlog.
A monthly community call is a forum for raising issues and discussing  bigger development items. We also run a Hackathon once a year (typically in spring) and a 1/2 day, face-2-face meeting with all the core developers before or after our Developer Summit. These in-person meetings are usually used to bootstrap the planning process for a release.

How we create a Roadmap

During planning and development of a release, the Release Manager sends out monthly development update e-mails on xen-devel that are labelled Xen x.y Development Update (check out the 4.4 mails). In these mails, we ask developers:

• whether they have items to be tracked on the roadmap and if so to add them to a backlog;
• for status updates, when they are on the backlog; and
• how likely it is that a feature will be in our codeline (including reviews) by the next update.

Toward the end of the development cycle, we start to treat blocker and critical bugs in the same way. More public announcements on release status are made on our blog and the announce list at critical points in the release cycle.

Stages within a Release Cycle

The following diagram shows the key stages, gates and git branches in our release cycle.

Feature Development / Feature Freeze: Feature development starts as soon as the release branch for the previous release has been created. It ends at the Feature Freeze point, when we typically won’t allow new features into the master branch. Patches that were submitted before Feature Freeze and are being reviewed may still be let in based on a risk analysis. This phase is typically 3-4 months long. The timing is adapted based on major holidays (XMas and New Years) or major conferences, such as Xen Project Developer Summits.
Hardening / Code Freeze: During the hardening phase, which typically lasts 3-4 weeks, we complete any remaining reviews of patches applied during development and focus on bug fixes. This phase ends with the Code Freeze point, after which each bug will be considered on a case-by-case basis and risk to the quality of the release. Major vendors will start running their complete test suites during this phase of development. Hardening ends with the first release candidate (RC1).
Release Candidates / Release: We close the release cycle making a number of release candidates (RCs). Historically, we needed 4-6 RCs and create one every 1-2 weeks. During the Release Candidate phase, we run Xen Project Test Days and major vendors in the community run their test suites against RCs. Typically, only bugs of blocker severity are fixed. This phase typically lasts 6 weeks, but may be extended if blocker bugs are found that need to be addressed. We end this phase by creating a stable release branch.
Release Announcement: Usually, we make an official Release Announcement shortly after the stable release branch has been created. But sometimes, we delay by a few days to coordinate press activities.

Improving the Process

We generally have discussions on how to improve the release process, at face-2-face meetings such as the Hackathons and Developer meetings. At previous developer meetings, we decided to focus on improving our test infrastructure and started using Coverity Scan regularly on our codebase and formed the test framework working group. In the upcoming May Hackathon, we will for example have a discussion about how to handle the ever-increasing traffic on our development list (which regularly hits more than 4,000 messages per month).

Additional Information

• On the 4.5 Roadmap (note that we are only just staring to plan; as such there is not much information available yet)
• On Maintenance Releseas
• On User Feedback