Introducing Xen Project’s New Test Lab
One of Xen Project’s highest priorities is to continually work to improve the quality of our code and code coverage. We’re serious and proactive when it comes to minimising and, whenever possible, eliminating any adverse effects from defects, security vulnerabilities or performance problems. Our users run some of the largest cloud and datacenter operations in the world, so we know that reboots and service interruptions are more than mere glitches in operations.
That’s why the Xen Project Advisory Board has made a significant investment to set up a Xen Project-owned Test Lab based on OSSTEST. We’re excited to announce that the lab is now live and in production.
Our new test lab will allows us to improve upstream quality of the Xen Project Hypervisor, while developers from different organizations now have access to a system that can be used to investigate test failures, schedule test jobs, etc. The test lab is also being used for regression testing of the Xen Project Hypervisor against different upstream and downstream environments. The Xen Project also tests against upstream and downstream open source projects such as the Linux Kernel and OpenStack to ensure that the Xen Project Hypervisor works seamlessly with important key technologies.
To help kick-start the lab, we created the Test Infrastructure Working Group, which helped source a COLO, spec out the hardware requirements for the test system and helped procure the machines needed to run OSSTEST. A special thank you goes to Ian Jackson, who has been instrumental in getting this effort off the ground over the past six months. We also appreciate contributions from our committee comprised of employees from AMD, Amazon Web Services, Citrix, Intel and Oracle, who have worked closely with our maintainers, project leads and wider community to create the new testing facility. As a result, we’ve decommissioned the old system run by Citrix as a service to the Xen Project community.
In this blog, we’ll describe our platform in more detail, identify what’ we’re doing today with OSSTEST and new goals from our Test Infrastructure Working Group.
Xen Project’s New Test Lab
This picture shows a section of our test lab, which is co-located at the Earthlink Boston Data Centre
Setting up a test lab with several different architectures and suppliers as well as a globally dispersed community is simply a challenge. For example, we discovered a number of bugs in the Linux kernel, FreeBSD and Hypervisor, which are currently being addressed. We also tripped over a number of unexpected issues, such as hardware issues and BIOS bugs in some of the machines in the COLO.
How Do We Use OSSTEST Today?
As mentioned earlier, the test lab is being used for regression testing. We’re testing different versions of Xen Project Hypervisor against different versions of the Linux Kernel, Linux distros, FreeBSD, Windows, QEMU, Rump Kernels and other open source projects we interface with. We also do test specific features such as Xen Security Modules, Credit 2 scheduler and other Xen Project features. And more recently we added functionality for performance testing of the Xen Project Hypervisor.
The set-up is essentially following a continuous integration paradigm; changes to Xen Project trees are pushed onto a staging branch. When tests succeed, changes are automatically pushed to the master branch. If there are failures, a bisection algorithm is run, which in many cases will identify the changeset that is causing the issue and developers are then notified that a specific changeset has created a problem.
This figure shows how the contribution workflow and its interaction between the test lab, the staging branch and the master branch.
What’s Next for Xen Project’s Test Lab
Bring the test lab to 100% capacity. As we work through the remaining issues, new machines will be taken into production as issues are resolved. The increased capacity will enable us to add on-demand testing capability, which will enable community members to test on hardware that they may not have themselves.
Broaden Access: The new system is in a community-operated COLO, rather than one hosted inside the Citrix private network. This, and the increased capacity, will mean that we can to grant access to the facility to Xen Project community members regardless of their affiliation, and provide an on-demand testing capability. To do this, we will need to agree upon processes to enable community members to access the test lab, in a similar way as we allow access to Coverity Scan data.
More Test Coverage: This increased capacity will allow us to further increase the breadth and depth of our testing. We have a number of enhancements in the pipeline, including for example testing of FreeBSD hosts, nested virtualisation, and compatibility testing with a variety of OS distributions.
Expand the infrastructure and add another rack. The Xen Project Advisory Board agreed to fund another rack and to procure further test machines in 2015. We expect that some 64-bit ARM machines will be part of the next tranche of machines that the Xen Project will buy.
Xen Project OpenStack CI loop: In addition to the Xen Project Test Lab, the Advisory Board is also funding a 3rd party continuous integration loop to improve Xen Project virtualization and OpenStack integration. A prototype of the CI loop is already running at jenkins.openstack.xenproject.org. Xen Project community members are currently fixing a number of issues in OpenStack Nova, Libvirt and Xen Project, such that the CI loop can become voting on OpenStack Nova commits.
We also continue to run the Coverity Scan static analysis service, which helps us quickly find, review and resolve Xen Project Hypervisor code defects and vulnerabilities. We’ve done this since 2013 and continue to regularly review and fix reported bugs.
If you’re interested in testing Xen, we encourage developers to join the xen-devel mailing list and ask how you can help. We’re always looking for help extending the set of test cases to tune performance for as wide a set of configurations and features as possible.