I want to introduce a process I have been using for the last several months on a web application currently in development. Our application is essentially a templating platform on top of which we can plug different content into an array of template types, organize them into a timeline, and produce a variety of custom workflows. If you're confused, think in terms of an online course which has many chapters, and many topics per chapter for which each has it's own unique workflow and content. For example, say you have 10 chapters, with 10 topics each. Now if each topic is comprised of a 10 page workflow (think test questions) you can quickly see the order of magnitude of the number of possible page + content permutations. In this example, we would be talking about (10 x 10 x 10) 1,000 unique pages that need to be tested. Now multiply this by the number of browsers we support. Managing this amount of data is a daunting task, and the linear QA tool we were using was not allowing us to gain efficiencies by detecting bug trends.
I developed this matrix because I needed to stay in sync with our QA team, without getting lost in the avalanche of data.
The design is very simple. My requirements were to be able to highlight the following bugs and bug trends:
- global bugs
- browser specific bugs effecting all pages
- page specific bugs seen on all browsers
- isolated cases of bugs on one page and one browser
- bugs which don't follow one of the above patterns but are seen more than once
With this matrix, I now have a higher level understanding of which bugs to watch out for when testing a particular flow. It allows me to gain efficiencies by enabling me to report a bug effecting multiple pages or browsers only once, and respectively only having to close the bug once. Whereas using a linear tool such as Trac, Bugzilla, HP QC, etc. would require the bug to be reported and closed multiple times.
I also find that the process of clearing bugs using a linear process looks a lot like this:
- Read bug #123
- Load the proper browser
- Load the proper environment
- Load the application
- Navigate to the specific page
- Try to reproduce the bug
- Write code to fix the bug
- Repeat steps 4, 5, & 6
- If it's fixed, resolve the bug
- Move on to bug #124
- repeat...
i.e., unnecessarily involved.
My opinion is that tackling bugs in such a linear fashion loses the essential component of understanding the scope that this matrix provides visually. Furthermore, using a system of color coding, you are free to prioritize not only a specific bug but an entire trend. You can immediately see where your problem areas are and address them accordingly. Existing bug tracking systems do not provide this type of inherent intelligence and so result in an unnecessary slog through mountains of individual bugs. Another feature you may be looking for is change history. Google Spreadsheets allow a simple interface for multiple team members to insert a comment for each cell. You are free to use whatever system works for your team size. I have found that it works fine on our team of 4 people (2 devs, 2 QA).
I won't go into much more detail so if you have questions, please ask. I have created an example doc (link below) with possible variations of bugs shown so you can get an idea for how to report certain bugs and bug trends. As our team grows, I will likely augment this form with a way to assign bugs to specific members.
Here's the link: Public Google Docs Spreadsheet
If you have a system which you've found to be effective, let me know. What works for you?
