1. Test Management and Contracts in Agile Environments Assurance with IntelligenceSlide 1 Paul Gerrard Gerrard Consulting 1 Old Forge Close Maidenhead Berkshire SL6 2RD UK e: paul@gerrardconsulting.com w: http://gerrardconsulting.com t: 01628 639173

2. Paul is the founder and Principal of Gerrard Consulting, a services company focused on increasing the success rate of IT-based projects for clients. He has conducted assignments in all aspects of Software Testing and Quality Assurance. He has degrees from the Universities of Oxford and London, was on the BCS SIGIST committee for 13 years, Founding Chair of the ISEB Tester Qualification Board and the host/organiser of the quarterly UK Test Management Forum and annual Summit conferences. He is a regular speaker at seminars and conferences in the UK, continental Europe and the USA. Paul has written many papers and articles, most of which are on the gerrardconsulting.com website. With Neil Thompson, Paul wrote 'Risk-Based E- Business Testing' - the standard text for risk-based testing. Paul is currently researching Test Axioms as a basis for test approaches, Critical Thinking and Open Source testing and collaboration tools. In 2008, Paul set up a new company, Aqastra with Susan Windsor. The aim of Aqastra is to provide an assessment, retraining and mentoring service to organisations wishing to transform business users into testers.” Paul Gerrard Assurance with IntelligenceSlide 2

3. Contracts, Test Management and Agile Agile Values, Principles and Behaviour And your problem is…? (brainstorm) How Testing Helps Agile Test Axioms and Agile Discussion of problems identified, with an eye on Axioms “Contracts as a manifestation of management/control” Agenda Assurance with IntelligenceSlide 3

4. Inputs to project plan for delivery - The Contract - The Requirements (business or technical level) The requirements may be good or bad But if the contract is poor - Things get forgotten, underestimated - Testing, risk management, reporting all suffer - Project Management attention on timescales and costs (inputs) not on deliverables, quality, benefits (outputs) - Acceptance process and criteria are loose, so plans are woolly when precision is most important Poor contracts result in poor plans and poor project (and test) management Assurance with IntelligenceSlide 4

5. Some of the main challenges facing testers: - Entry/exit criteria are abused - Testing phases are squeezed - Supplier test coverage is weak, variable If entry/exit criteria, minimum timescales, objective coverage targets are contractual - Yes, the same problems will probably occur - But, most importantly, the supplier may be penalised and may think twice about compromise Requirements can be ambiguous, negotiated But contracts are succinct, checked by lawyers, signed by senior management – so are treated more seriously Why contracts matter Assurance with IntelligenceSlide 5

6. Disregard for fixed requirements - No fixed baseline, no pre-planned tests - Acceptance criteria hard to pin down - ‘Quality gate’ becomes a meaningless term? (hurrah?) If developer testing is good, and users do rapid, iterative, incremental acceptance - Where does system test fit? Self-managing teams – so who do you manage? Do short cycles mean planned, meaningful, thorough, repeatable testing is impossible? Does Agile undermine test management? Assurance with IntelligenceSlide 6

7. Agile approaches are well-established but… Example: “Customer Collaboration over Contract Negotiation” sounds great as a value - None of the Agile principles provide guidance on how contracts are created - Contracts are tools for promoting ‘good behaviour’ in project participants Does Agile undermine the traditional disciplines, approaches, processes? Agile undermines traditional discipline?

8. Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan “There is value in the items on the right, but we value the items on the left more”. Agile values (from agilemanifesto.org) Assurance with IntelligenceSlide 8

9. Agile principles (from agilemanifesto.org) 1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. 2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. 3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. 4. Business people and developers must work together daily throughout the project. 5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. 6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. 7. Working software is the primary measure of progress. 8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. 9. Continuous attention to technical excellence and good design enhances agility. 10. Simplicity--the art of maximizing the amount of work not done--is essential. 11. The best architectures, requirements, and designs emerge from self-organizing teams. 12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly. Assurance with IntelligenceSlide 9

10. The Agile values/principles discourage… - Developer and customer taking ‘positions’ - Negotiating a commercial ‘middle ground’ - Freezing people’s attitudes towards fixed goals And encourage… - Developer and customer are collaborative roles - Goal-setting as ‘part of the process’ - Flexibility to change is the key to success. Agile behaviour Assurance with IntelligenceSlide 10

11. Team are unwilling to take responsibility for the testing Agile teams that don’t do agile Business wants a solution – Agile delivers code Lost the end to end picture – lots of sprints – non-integrated solutions Managing hybrid agile/trad projects In a time box approach, acceptance is ‘unstable’, not meaningful, invalidated by change (bad CM) 1: Agile v Testing problems – Brainstormed comments Assurance with IntelligenceSlide 11

12. Agile dependent on engaged users. Shipped system might not work in full community Performance test in Agile unlikely to be meaningful/scaled NF tests don’t fit Agile easily Imposing Agile on unready people/env/tools Mature Agile needs tools and process Low level tests need higher level process- oriented tests 2: Agile v Testing problems Assurance with IntelligenceSlide 12

13. Agile depends on people talking - but teams are distributed - We hire introverts Agile in credit crunched environment Getting the right users, getting any users Managing the scope of stories – growth without limit (no contracts/Lack of control) 3: Agile v Testing problems Assurance with IntelligenceSlide 13

14. How Testing Helps Agile I use the term INTELLIGENCE to refer to the information and evidence that is provided by testing and testers

15. Agile principles and testing Assurance with IntelligenceSlide 15 PrincipleWhere testing fits in 1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Early and continuous delivery of INTELLIGENCE of valuable software. Testing must link to and demonstrate VALUE. 2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. Changing requirements, however ‘welcome’ will be reduced by providing EARLY test INTELLIGENCE for more reliable decision making. 3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. Need to provide test INTELLIGENCE RAPIDLY with flexibility. 4. Business people and developers must work together daily throughout the project. Better, faster appraisal of the status of delivery through ACCURATE, TIMELY INTELLIGENCE. 5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. See above. 6. The most efficient and effective method of conveying information to and within a development team is face- to-face conversation. Maybe, but developers need INTELLIGENCE to support or justify their proposed approach or position.

16. Agile principles and testing 2 Assurance with IntelligenceSlide 16 PrincipleWhere testing fits in 7. Working software is the primary measure of progress. Delivery of test INTELLIGENCE is the only way to demonstrate progress. 8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. Delivery of INTELLIGENCE must be EFFICIENT. 9. Continuous attention to technical excellence and good design enhances agility. Continuous attention to EVIDENCING progress… 10. Simplicity--the art of maximizing the amount of work not done--is essential. Achievement of test INTELLIGENCE is a SUFFICIENT requirement so should be delivered as a PRIORITY. 11. The best architectures, requirements, and designs emerge from self-organizing teams. Part of the belief system. 12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. Only with evidence of ACHIEVEMENT can effectiveness be assessed.

17. Mary and Tom Poppendieck (http://www.poppendieck.com/) articles/presentationshttp://www.poppendieck.com/ Jens Coldewey, cutter.com, “Outsourcing Agile Contracts”, “Contracting Agile Projects” Kent Beck, Dave Cleal, “Optional Scope Contracts”, http://www.jarn.com/about/OptionalScopeContracts.pdf http://www.jarn.com/about/OptionalScopeContracts.pdf Nora Sleumer et al., “Pay Per Use Contracts” see Poppendieck site Martin Fowler, several bliki posts, www.martinfowler.com/blikiwww.martinfowler.com/bliki Alistair Cockburn, “Agile Contracts”, www.alistair.cockburn.us“Agile Contracts”, www.alistair.cockburn.us DSDM Consortium, “Sample Agile Contract”, http://www.dsdm.org/products/contract.asp http://www.dsdm.org/products/contract.asp Quinary, “Optional Scope Project description”, see Poppendieck site Mountain Goat Software, “Writing Contracts for Agile Development”, http://www.mountaingoatsoftware.com/article/view/5-writing-contracts-for-agile- development”, http://www.mountaingoatsoftware.com/article/view/5-writing-contracts-for-agile- development Lots of references to “Fixed Price Contracts and Agile” on the web. “They do work…”, “They don’t work…” etc. Other references to Agile and Contracts Assurance with IntelligenceSlide 17

18. Test Axioms and Agile (The first application of Axioms) Handout time Assurance with IntelligenceSlide 18

19. Each Axiom has a corollary (i.e. “If you don’t do this or ignore the axiom…”) 5-7 questions (you might add your own) For each question: - What project behaviour am I expecting? - How do I encourage that behaviour? - How do I instill that behaviour? - How do I enforce that behaviour? Answers could… - Give you the text you need to agree in a contract - Indicate where test management time should be spent. Sixteen Axioms Assurance with IntelligenceSlide 19

20. No – ALL testing is Context-Dependent You need to understand your own project, it’s goals, team culture, constraints, the ‘art of the possible’ etc. etc. Every project, goal, culture, constraints etc. are UNIQUE Only YOU can evaluate these to answer the questions So let’s look as an example Axiom and questions. So, you aren’t going to GIVE me the Agile-Test silver-bullet? Assurance with IntelligenceSlide 20

21. Stakeholder Axiom: Testing needs stakeholders Summary Identify and engage the people or organisations that will use and benefit from the test evidence you are to provide Corollary: You won’t have a mandate or any authority for testing. Reports of passes, fails or enquiries have no audience. Questions Who are they? Whose interests do they represent? What intelligence do they want? What do they need it for? When do they want it? In what format? How often? Assurance with IntelligenceSlide 21

22. Go back 11 slides… Let’s look at the problems with Axioms in mind

23. Summary

24. Contracts normally specify obligations and encourage certain behaviours to achieve a goal - Normally fixed costs, fixed timescales and known scope Agile contracts assume scope is variable - Agile ‘works’ because of close collaboration, communication and trust Testing is an information/intelligence provision service Contracts need to specify obligations and behaviours to deliver most valuable intelligence fast Axioms define a minimum and non-negotiable set of behaviours for ANY testing context So can be used as the basis of testing in ANY contract. Agile testing and Axioms Assurance with IntelligenceSlide 24

25. The Axioms specify WHAT should be done It is up to YOU to define HOW it will be done The HOW is a set of agreed processes, procedures, deliverables, schedule, entry/exit criteria These practices are context-dependent and must satisfy the constraints of your context - The development process - The skills, capability and culture of the people - The technology available - The time available - The specific needs of your testing stakeholders - Time and cost - Etc. Using the Axioms Assurance with IntelligenceSlide 25

26. Slide 26 The three axiom groups Stakeholder Value Scope Fallibility Good-Enough Stakeholder Value Scope Fallibility Good-Enough Delivery Repeat-Test Sequence Environment Event Never-Finished Delivery Repeat-Test Sequence Environment Event Never-Finished Design Basis Coverage Prioritisation Oracle Design Basis Coverage Prioritisation Oracle

27. Testing is an information provision service Work out who your testing stakeholders are - You (as a developer), Developers, Users, Project Management, Sponsors What do those stakeholders want? How will you (and/or your supplier) deliver it? Contracts define and enforce behaviours Axioms trigger the right questions to ask. Close Assurance with IntelligenceSlide 27

28. Slide 28 gerrardconsulting.com uktmf.com Test Management and Contracts in Agile Environments Thank You!