Reporting A Bug Against the Xen Hypervisor

With the release process for Xen 4.3 in full swing (we intend to release the third release candidate this week) and with the Xen Test Days initiative (the next one is this Wednesday 5 June, join us on IRC freenode #xentest) I thought it would be useful to offer up some information and guidance on how the Xen project deals with bug reports and how to report bugs against the Xen Hypervisor Project.

Reporting a Bug

Confirm That Your Issue Is a Bug

Experience has shown that oftentimes what appears to be a bug is often just a misconfiguration or misunderstanding of how things are supposed to work. This can be down to a lack of documentation (a perennial problem for Open Source projects and something which we are working to address with our regular Xen Document Days) or the inherent complexity of something of the things which one can achieve (or try to achieve!) using Xen.
With that in mind it is often useful to try seeking help via other means before resorting to submitting a bug report. Useful resources for asking questions are:

• In a Xen system logs the logs can usually be found in /var/log/xen and will sometimes provide useful insight into an issue.
• Search engines. As well as simply searching for key terms relating to your issue you can also search the User and Developer list archives.
• The #xen IRC channel on Freenode.
• The Xen Project Question and Answer System.
• The Xen Users Mailing List.

If you do manage to solve your issue through these channels, especially if your issue was down to missing/bad/wrong documentation, then please do consider updating the documentation, either as part of the next document day or whenever you like, so that the next person can avoid your pain. If you aren’t comfortable updating the docs then please try to at least add the issue to the Document Day TODO list or sending a quick note to the developers — This will help us better target our efforts on future document days.

Provide As Much Relevant Information As Possible

In order to make a bug report as effective as possible it pays to spend some time considering the content of the report. Writing a good bug report can take time and spending that time can make it much more likely that your issue will eventually be solved.
In your bug report please try to describe as accurately and precisely as possible what it is you did, what you saw happen and what you expected to happen. Where possible please cut and paste the exact commands you ran and the output you saw rather than trying to paraphrase them since the details often matter. Please do not omit steps (e.g. because they are obvious), sometimes the simplest seeming things make a difference.
As well as thoroughly describing the issue please include relevant details of your system configuration. In particular:

• Software versions; This should include the full version of Xen as well as any domain 0 or guest kernels and the version of the dom0 and guest OS. Please also tell us where these came from — e.g. did you compile Xen yourself or are you using a distro package? If the former then please tell use which tree and commit you are using, if the latter then please specify the exact package version etc.
• Host configuration; For example if your issue involves networking then include details of the host network configuration.
• Guest configuration; In particular the guest configuration file

Please also include relevant logs:

• Xen dmesg; The Xen console ring can be dumped to a file using the command xl dmesg.
• Domain 0 dmesg; The domain 0 console ring can be dumped to a file using the command dmesg.
• Guest console; The easiest way to obtain these is often to configure xenconsoled to log guest console to a file. See this short guide.
• Output from tools; As discussed above please try to cut and paste the exact commands and the output they produce. Please also turn up the verbosity where possible. In particular run xl with the -vvv option. e.g. xl -vvv create ....

If your issue involves the host (either Xen or domain 0) crashing then you may find that the host reboots before you are able to read the error message. The preferred way of dealing with this is to setup a serial console (See the Xen Serial Console wiki page) and to capture the logs that way. If this isn’t possible for some reason then you can instead add the noreboot option to the hypervisor (not domain 0) command line. This will prevent the host rebooting and give you time to transcribe the error messages or, even better, take a digital photograph.
Additional advice on useful content for bug reports can be found on the Reporting Bugs Against Xen wiki page. This page contains lists of useful logs to include, use commands to show the system configuration as well as links to documents which aim to help you write the most useful bug report you can.
Lastly, If you tried using the documentation and it didn’t help please mention which docs you looked at, this will help us know which documents to focus on improving. If you weren’t able to find any documents dealing with your issue please say where you looked and what search terms you used, again this will help us improve the discoverability of any existing documentation or let us know where there is a hole.

Sending a Bug Report

The Xen developers prefer to receive bug reports via email to the xen-devel@lists.xen.org mailing list. Once you have written the description and collected the logs reporting the bug is as simple as sending an email to the list. There is no need to be subscribed to the list: although non-subscribers are moderated (to help keep the amount of SPAM down) mail is usually accepted promptly and the sender is then whitelisted for future postings. The Xen Project lists also have a policy of copying posters on replies which means you won’t miss out on any followups even if you aren’t on the list.
Please tag your email subject line with a [BUG] prefix. Try to make the remainder of the Subject line a useful (but short) summary of the issue you are seeing.

Writing a Tool to Guide Users

Collecting all the logs to make up a complete bug report can be a bit of a chore. Ideally we would have a self contained tool which could be run on a system to collect up the logs and guide the user through the process of describing the issue and perhaps even send the bug report for them. If you are familiar with a reasonably common scripting language (i.e. likely to be present on most systems running Xen) and would like to help out with a tool like this then please get in touch on xen-devel!

Tracking Bugs

Receiving bug reports via emails is a workflow which works well for the majority of Xen developers however it is prone to useful or important bug reports falling through the cracks. For this reason we are currently experimenting with a new bug tracking system which will help us keep track of the mail threads resulting from bug reports. The system can be found at http://bugs.xenproject.org/xen/. See the announcement for more information.
The tracking system remains email based so please continue to file bugs via email to the list as discussed above.