On October 24th, James Shore gave a presentation to the XP PDX group on testing in the agile environment. This was part of a series of events Jim is giving in anticipation of his new book co-authored with Shane Warden, The Art of Agile Development.
By using XP practices as a defect prevention measure, defects are much lower than in traditional development environments and fewer tester resources are required. Jim is seeing in projects that are using this type of agile defect prevention that:
- Defects have been reduced to “a couple a month.”
- The ratio of testers to programmers is 1:4, significantly lower than the 2:1 or 1.5:1 ratio seen in large development organizations.
Why It Works
XP agile engineering practices are a way to produce quality code by preventing defects as part of the software development process. Jim sited two studies to support this position:
- The first raised the somewhat paradoxically issue of “More Testing, Worse Quality.” Essential by increasing the amount of testing resources in an after the fact “find and fix” mode, developers take less responsibility for the quality of their code.
- The second was a straight-up study of average developers using XP practices and the resulting defect rates were ridiculously lower than industry averages.
Jim then showed how specific XP practices fit into specific categories of defects.
Defects of Intent
These are defects where the code does not work as intended by the programmer. For example, I write if(foo.IsBar), when I meant to write if(!foo.IsBar). The XP practices that address these defects are:
- Test-Driven Development (specifically test-first development)
- Pair Programming
- Sustainable Pace
- Continuous Integration
- Shared Coding Standards
Defects of Design
The idea here is that areas of design problems or difficulty are the source of most of the defects, “20% of the modules cause 80% of problems.” These are addressed by:
- Relentless Refactoring
- Collective Code Ownership
- Common System Metaphor
- Simple Design
Defects of Requirement
This is when what was produced is the wrong thing (even though it may have been done it very well!). Code works great, but it’s not what was wanted or needed. To make sure the right thing is done well, practice:
- Sitting with the Customer
- Iterative (Small) Releases
- Example Driven Requirements/Customer Acceptance Tests
Role of the Tester in the Agile Environment
Given that the developers are now preventing defects in the first place, what is the role of the Tester? Jim suggests three primary areas:
- Finding requirement gaps.
- Technical investigation (performance, stability, etc.)
- Exploratory testing.
For the last item, the expectation is that nothing is going to be found. When defects are found, the team meets to do root cause analysis on why the defect occurred. What about the process allowed the defect to happen? Is there a design issue that needs to be addressed? Is the customer accessible and sitting with the team? Exploratory testing is intended provide an overall indicator of the health of the quality in the software development process.
