The Xen Project’s code contributions have grown more than 10% each year. Although growth is extremely healthy to the project as a whole, it has its growing pains. For the Xen Project, it led to issues with its code review process: maintainers believed that their review workload increased and a number of vendors claimed that it took significantly longer for contributions to be upstreamed, compared to the past.
The project developed some basic scripts that correlated development list traffic with git commits, which showed indeed that it took longer for patches to be committed. In order to identify possible root causes, the project initially ran a number of surveys to identify possible causes for the slow down. Unfortunately, many of the observations made by community members contradicted each other, and were thus not actionable. To solve this problem, the Xen Project worked with Bitergia, a company that focuses on analyzing community software development processes, to better understand and address the issues at hand. We worked with Bitergia on an initial statistical analysis of the code review process and later on a Code Review Dashboard for use by the community. The following OSCON presentation lays out the journey, the project went through:
Findings of the Initial Code Review Study
There were three key areas that we found that were causing the slow down:
- Huge growth in comment activity from 2013 to 2015
- We also saw that the time it took to merge patches (=time to merge) increased significantly from 2012 to the first half of 2014. However, from the second half of 2014 time to merge moved back to its long term average. This was a strong indicator that the pre-emptive measures we took, such as a focus on design and architecture reviews and contributor training, actually had an effect.
- Looking at this in more detail, it turned out that complex patches were taking significantly longer to merge than small patches. As it turns out, a significant number of new features were actually rather complex. At the same time, the demands on the project to deliver better quality and security had also raised the bar for what could be accepted, which impacted new contributors more than established ones.
Introducing the Xen Project Code Review Dashboard
To make the tooling that we developed more accessible to the entire community, the Xen Project Advisory Board funded the development of a Code Review Dashboard. We defined a set of use-cases and supporting data that broadly covered three areas:
- Community use cases to encourage desired behaviour: this would be metrics such as real review contributions (not justed ACKed-by and Reviewed-by flags), comparing review activity against contributions.
- Performance use cases that would allow us to spot issues early: these would allow us to filter time related metrics by a number of different criteria such as complexity of a patch series.
- Backlog use cases to optimize process and focus. The intention here was to give contributors and maintainers tools to see what reviews are active, nearly complete, complete or stale.
To find out more check out
- Dashboard: tinyurl.com/xenproject-dashboard
- Documentation: tinyurl.com/xenproject-dashdocs
Value for other projects
Like many FOSS projects, the Xen Project code review process uses a mailing list-based review process, and this could be a good blueprint for projects that are finding themselves in the same predicament. It is already clear, that there are many useful additions that can in future be added to the technology we have developed. In addition, we are currently working on improvements of the Code Review Dashboard as part of an Outreachy Project by Priya V (check out her blog), which is jointly mentored by Lars Kurth (Xen Project) and Jesus M. Gonzalez-Barahona (Bitergia). All the code is open source: if you want to have a look, check out the code and the contribution guide.