©2006 Systeme Evolutif LtdSlide 1 Test Process Improvement …is a waste of time? Paul Gerrard Technical Director Systeme Evolutif Systeme Evolutif Limited 9 Cavendish Place London W1G 0QD, UK Tel: +44 (0)

©2006 Systeme Evolutif LtdSlide 2 Paul Gerrard Paul is the Technical Director and a principal consultant for Systeme Evolutif. He has conducted consultancy and training assignments in all aspects of project assurance, software process improvement and testing. Previously, he has worked as a developer, designer, project manager and consultant for small and large developments. Paul has engineering degrees from the Universities of Oxford and London, is Web secretary for the BCS SIG in Software Testing, contributed to BS and BS7925-2, host of the Test Management Forum (uktmf.com) and Founder Chair of the ISEB Software Testing Certification Board which has been taken by 20,000+ candidates.uktmf.com He is a regular keynote speaker at both development and testing seminars and conferences in Europe, US and Australia, and won the "Best Presentation" award at several conferences. Paul devised the term Project Intelligence for an innovative approach to results-based management supported by the disciplines of software testing. He is Women’s coach for Maidenhead Rowing Club and webmaster and ISP administrator for several websites including

©2006 Systeme Evolutif LtdSlide 3 Why is Process the Focus of Improvement?

©2006 Systeme Evolutif LtdSlide 4 How to improve… I want to improve my ( insert any activity here ) _______ people improvement _______ organisation improvement _______ process improvement    Changing people (like me) and organisation (like my company) is so hard – let’s not even think about it

©2006 Systeme Evolutif LtdSlide 5 The delusion of ‘best practice’ There are no “practice” Olympics to determine the best There is no consensus about which practices are best, unless consensus means “people I respect also say they like it” There are practices that are more likely to be considered good and useful than others, within a certain community and assuming a certain context Good practice is not a matter of popularity. It’s a matter of skill and context. Derived from “No Best Practices”, James Bach,

©2006 Systeme Evolutif LtdSlide 6 The delusion of quality management (a la ISO 9001) Google search for –“ISO 9001” – 49,700,000 –“ISO 9001 Training” – 15,500 –“ISO 9001improves quality” – 1 ISO 9001 in a nutshell –“Document what you do” –“Do what you document” –“Do what you like” ISO 9001 accreditation without reference to the garbage that is produced by the documented processes.

©2006 Systeme Evolutif LtdSlide 7 Exchange between quality manager and ISO 9001 consultant > 3. The Guidelines suggest the use of a > Life-> Cycle model for software development. Is it > necessary to reference a specific model? No. You do not need to reference a specific lifecycle. But you need to define your development process...which really speaks to the lifecycle of product development. Most development processes are derivatives of some type of model that someone else hatched.

©2006 Systeme Evolutif LtdSlide 8 Exchange between quality manager and ISO 9001 consultant cont… > I would like to learn more about the certification audit > process and how to document reasonable quality > processes which will allow us to provide the highest > quality to our customers and which fall within the 9001 > standards. The key is putting practices in place that hold up over time and that support getting software out the door faster and better. In the simplest sense, an implementation strategy that has worked for all the implementations I've been involved with is....put fundamental, good software development practices in place that are achievable and can act as a baseline for improvement.....check the standard to make sure you've covered your bases....and evolve the practices as the market and business changes. As regards documentation......it is tougher to write a 5 page design review procedure than a 50 page design review procedure because it requires that you communicate in a cogent fashion. But that's the subject of a "Streamlining Documentation" course that I teach.

©2006 Systeme Evolutif LtdSlide 9 The delusion of process models (e.g. CMM) Google search –“CMM” – 12,100,000 –“CMM Training” – 12,200 –“CMM improves quality” – 4 An Evolutif client… –CMM level 3 and proud of it (chaotic, hero culture) –Hired us to assess their overall s/w process and make recommendations (quality, time to deliver is slipping) –47 recommendations, only 5 adopted – they couldn’t change –How on earth did they get through the CMM 3 audit?

©2006 Systeme Evolutif LtdSlide 10 But process models make improvement simple don’t they? People like simple models: –levels of maturity, stepping stones, checklists, roadmaps and outside support for credibility But life is much more complicated, unfortunately “Things should be made as simple as possible, but no simpler” - A. Einstein

©2006 Systeme Evolutif LtdSlide 11 Deming If you can't describe what you are doing as a process, you don't know what you're doing. –W. Edwards Deming It is not enough to do your best; you must know what to do, and THEN do your best. –W. Edwards Deming

©2006 Systeme Evolutif LtdSlide 12 Deming 2 I have to disagree with the first statement The second one states the bleedin’ obvious, doesn't it? –e.g. describe playing football, seducing a woman, winning a deal, convincing someone that a project is risky as a process –Is that “process” the critical factor in success?

©2006 Systeme Evolutif LtdSlide 13 People need process? A big problem with process is it becomes all encompassing Process folk sell process and cast all things in terms of it, forgetting that people who are smart, succeed in spite of process, not because of it It could be argued, that less smart people need process –(By less smart, we're talking about people who need so much structure and enforced discipline they can only operate in the military, or in prison probably) Is our industry really staffed by such people? Do we really want production-line workers?

©2006 Systeme Evolutif LtdSlide 14 Physics quotes… “I believe that a scientist looking at nonscientific problems is just as dumb as the next guy” “It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong” Richard P. Feynman

©2006 Systeme Evolutif LtdSlide 15 Process quotes “I believe that a process consultant looking at non-process problems is just as dumb as the next guy” “It doesn't matter how beautiful your process model is, it doesn't matter how smart you are. If it doesn't agree with reality, it's wrong” me

©2006 Systeme Evolutif LtdSlide 16 Test Process Improvement

©2006 Systeme Evolutif LtdSlide 17 From this… Current Maturity Future Maturity Process Change Current Capability Future Capability Acts of Faith Better Capability = better, faster, cheaper Perceived Results Chain

©2006 Systeme Evolutif LtdSlide 18 To this… Current Constraints/ Problems Changed Aspirations Current Capability Future Capability Acts of Change Better Capability = better, faster, cheaper Actual Results Chain

©2006 Systeme Evolutif LtdSlide 19 Constraints, problems and aspirations Constraints are fixed headcount, budget, timescales, quality of requirements, contracts etc. Problems are “testing takes too long; too expensive; can’t hire testers; bugs get through” etc. etc.… Aspirations: –Personal: personal development, fulfilment, motivation –Organisational: hero culture to team culture, outsourced, higher consistency, predictability Acts of change are…

©2006 Systeme Evolutif LtdSlide 20 Acts of change – focused on constraints, problems and aspirations Changes in behaviour to address specific problems (effectiveness, efficiency etc.) Targeted personal and team development Infrastructure change (process, techniques, tools, environments) to support the changes Managed Transition…

©2006 Systeme Evolutif LtdSlide 21 Managed transition “It isn’t the changes that do you in” –William Bridges Get them to let go of the old ways –Practitioners, managers, everyone Manage the neutral zone –Like a trapeze artist, there’s an uncertain period Launch a new beginning –purpose, picture, plan

©2006 Systeme Evolutif LtdSlide 22 Test improvement model Current behaviour assessed in the context of current problems, goals and constraints Aspirations drive the definition of goals People in the job define and consent to the required changes in behaviour People supported by –Personal/team development plans –Infrastructure investment (process, technology, tools, environment) specific to the change Transitions are managed, not assumed.

©2006 Systeme Evolutif LtdSlide 23 Test Process Improvement …is a waste of time Over to you?