Agile Performance Testing

Two recent events had me thinking about the subject line. One was a Webinar given by CloudBees titled, “ Adding Performance Testing into the Continuous Integration Process “ (the link is the Webex recording), the other was reading a blogpost shared by a colleague, "Combining Agile with Load and Performance Testing: What am I in for?” In essence both of these presentations ask the same basic question -- with all the focus on Continuous Integration and rapid feedback, "Why hasn’t development embraced Performance Testing as part of their development cycles?" The short answer from both — “It’s Hard”



Too often we give up on hard problems, but one of the development teams that I have worked with put this concept into practice. In addition to the challenges listed in the BlogPost, they struggled with other issues:

• Presenting the results for maximal impact/clarity.   This team worked hard to represent the results graphically (see below).
• The delta between test runs — while the idea is to perform the tests continuously, in reality the practice was simultaneously with development.  (You might also read Tim Hinds, next post "How to Make Your Load & Performance Testing More Continuous")
• Dealing with “White Noise” — allowing for changes in network performance, other activity on the systems under test, etc.  This team chose measure performance test runs as “Passing” based on a plus/minus 15% of previous runs.
• The previous approach created two other corollary problems:
• the tests could “pass” with each run while getting 14.9% worse with each run — hardly a desirable definition of "pass".
• when to re-baseline the performance expectation (and measure the plus/minus 15%) 

All these issues aside, as they say in baseball about having “too much good pitching”, these are good problem(s) to have. The entire process never let the develop team forget about performance, and it helped the team isolate and triage potential performance problems much earlier in the development cycle.  Isn’t that what Continuous Integration is all about?