Getting out of the Testing Game By Bill Matthews Test Architect Manager Technical

What is the Testing Game?  Testing is seen as an end in itself rather than a means to an end  Testing has become Nominalised (verb treated as a noun)  Testing is seen as a barrier that must be passed  Testing driven by process not outcomes  Testing is disconnected from its context  Those outside testing dictate test activities

The challenge  “Testing is taking too long”  “Testing doesn’t find all the bugs”  “It all goes fine until we hit testing…then it just falls apart”  “The PMs struggle to get good testing done”  “We use industry best practices but I’m sure they’ve changed since we last had a consultant in”  “Can you help us improve our testing process practices?”

The Challenge  Company was in a competitive market and so always had various offers in place.  Most offers took about 4 weeks to implement from idea to launch.  Frequently had to roll back changes because of errors.  Releases were slow, very early morning affairs  Rollbacks resulted in system outages => possible loss of revenues  Difficulties in getting a clear picture of what was happening during testing  Phased testing…for small changes

How the management viewed testing

How would you improve the situation?  Traditional approaches to test process improvement take an analytical approach.  Assumes current practices approximate some “best practice”  Measures against an idealised “best practice”  Suggests a prescription to bring current practices closer to the “best practice”  What if your processes are fundamentally broken and don’t fit your current needs? Design Thinking (Exploring What If)

2 minute introduction to the Business Canvas Model

The Value of Testing  The Customers and Value Proposition segment is where the value of our testing activities can be realised. Who are your customers and what problem do we solve for them?  Do we solve the “testing problem” for them?  If you are in the Testing Game…yes.  And our customers also think that’s the problem we solve for them  If not the “testing problem” – what problem do we solve?

How do our customers want to be engaged?  The Customer Relationship and Channels segments is where we understand how our customers want to interact with us.  So how do we deliver the information our customers value?  Those in the Testing Game think it’s all about Test Plans and Metrics  But have never really explored how their customers (really) want to work with them?  If you were a customer how would you want to be engaged?  Think about the user experience/journey when they engage with you – does it feel right and appropriate?

How do deliver something our customers value  Key Activities, Key Partner & Key Resource Segments  This is where the activities associated with Testing sit.  What are the key activities that we need to do (and do well) to deliver value?  Those in the Testing Game have a testing centric view so know all about these segments.  But customers just get what they are given – take it or leave it.  Wouldn’t it be better to have a customer centric view and adapt the How in line with the Need?

What we came up with

The catalyst for change  What would be different if this is how we approached testing?  A flood of energised ideas  Not all of them practical but at least we were Exploring What If  What’s stopping us?  Only the Permission to change

Organic change  Some changes that happened:  The test team started to communicate…and it was infectious!  Collaboration - No longer a long chain of sequential processes  Abandoned the “one size fits all” processes and naturally adopted a more context driven approach  “Does that mean we don’t need to write test scripts?”  Test Plans were replaced with Kanbans  the PMs were wary of this at first but agreed to let us try and manage our own work.  Soon Kanbans were popping up all over the office (outside of the test team)  Focus was on the flow of information  PMs finally stopped telling the testers how to test…and we stopped having test phases based around testing levels.  The team seemed more energised  Overall time for small changes reduced to about 2 week  still not great but an improvement

So the Business Canvas Model is Magic then?  So all the change happened because of a single sheet of paper?  No really, the magic is with the people  It helped the team see a wider context and focus on what adds value  In most cases they already had ideas on how to improve their practices.  Gave them a framework to discuss what they do without really talking about testing  They were given permission to make changes  A little sad that they needed to be given permission