Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Ray Arell Sr. Engineering Manager, Intel Corporation

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Key Concepts Personas Scrum Lifecycle Strategy Final Thoughts

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Learning Objectives: Establish a base understanding of Scrum Understanding of what makes up a general test strategy

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Individuals and interactions over processes and tools Working software over contract negotiation Responding to change over following the plan 4 We are uncovering better ways of developing software by doing it and helping others do it. Though this work we have come to value:

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation.

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Automated testing Barely sufficient documentation Bottleneck management Coding standards Collective code ownership Co-location Continuous team integration and CM Customer focus group review Customer onsite Daily standup Deferred Commitment Refactoring Retrospectives Risk management Self-tasking Simple, robust design Small releases Sustainable pace Test-driven development Test first Unit testing Unity statement Use cases User stories Design metaphor Exploratory spikes Feature-based planning Group design Information radiators Inspections “Intentional” design Issue tracking Last Responsible Moment Monitor and adjust Pair programming Project velocity

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. 7 Each sprint is a fixed duration Team works from a prioritized product backlog Short daily team meetings Must deliver working and fully tested code

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Owned by the product owner Formal list of product requirements written as user stories Contains the acceptance criteria Estimated by the development team on the effort to complete Priority and customer value of the item Backlog is reprioritized at the start of each iteration Defects are also tracked on the backlog 8

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Product Owner: Owns the backlog and stakeholder relationship Voice of the stakeholder to the development team Makes decisions and priority adjustments Removes obstacles Scrum Master: Drives the daily process Tracks progress and removes obstacles Facilitates, mediates and protects team The Team: Programmers, test professionals, and other key people Self managed/breaks down user stories down to tasks Reports daily on status in standup meetings (progress, plan, issues) Responsible for demo to stakeholders

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. 10

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. 11 Agenda: 1.What have you done since yesterday's meeting? 2.What are you going to get done today? 3.What obstacles do you need removed? adsfsadfdsafsadfasdf Sasdf Adsfa asdfdasfdsafSdkjfsfsa Asfadf adsfsadfdsafsadfasdf Sasdf Adsfa asdfdasfdsafSdkjfsfsa Asfadf adsfsadfdsafsadfasdf Sasdf Adsfa asdfdasfdsafSdkjfsfsa Asfadf adsfsadfdsafsadfasdf Sasdf Adsfa asdfdasfdsafSdkjfsfsa Asfadf adsfsadfdsafsadfasdf Sasdf Adsfa asdfdasfdsafSdkjfsfsa Asfadf adsfsadfdsafsadfasdf Sasdf Adsfa asdfdasfdsafSdkjfsfsa Asfadf adsfsadfdsafsadfasdf Sasdf Adsfa asdfdasfdsafSdkjfsfsa Asfadf adsfsadfdsafsadfasdf Sasdf Adsfa asdfdasfdsafSdkjfsfsa Asfadf adsfsadfdsafsadfasdf Sasdf Adsfa asdfdasfdsafSdkjfsfsa Asfadf

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Review Release Plans (Developers) (Developers) Distribution, review and adjustment of product requirements (Developers) (Developers) Sprint Develop Features Test Internal Review Adjust CustomerReview Review software Compare backlog with developed software Editing of backlog requirements Add new backlog items to teams Assign backlog items to teams Plan next review (Developers) (Project Stakeholders) All Project Requirements Complete (Quality, Features, Cost) Adapted from : Wikipedia Scrum (development) ; Many Authors

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Sprints are a fixed delivery date Stories that are not done are pushed to product backlog Stories and defects live in the backlog together Software repairs are prioritized with the customer Scrum does not dictate what processes are used within the 2-4 week sprint This is left up to the development team Validation is a part of the development team Validation and test are not visitors--they are a part of the team

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Product backlog: is the repository of all the high- level features for a product Documented user stories Sprint backlog: are the stories being worked on by the team They are decomposed into workable tasks Requirements: are documented in stories Stories are expected to evolve and change Decomposition of stories into tasks will drive clarification Tasks are manageable pieces of work

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. User stories Is a software system requirement that is written in everyday language It is typically one to two sentences It needs to be descriptive and should provide enough information to define the acceptance criteria Acceptance criteria List of measurements that will be used to verify “done” Can define both functional and non-functional test requirements Defined with the customer input Further Reading:

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. User Story: Application Start: As a user I expect that when the application is invoked it will display the software license agreement for a default of 5 seconds. Display time can be updated to a max of 100 seconds Validation Criteria: 1. Visually verify that the license agreement is displayed for the default time period 2. Set display time to n seconds and verify that it displayed for that period of time 3. Verify that default time cannot be set less than 0 and no greater than 100 seconds

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. It is the high-level plan on how testing is going to be done. It defines the type of testing that is going to be used, when it will be used, resources needed and an articulation of key constraints and assumptions.

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Provides a framework for the entire test process Manages expectations throughout the product development cycle Communicates strategic choices with the customer and the development team

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Provide maximum return on investment by Finding the defects that have the highest customer impact and exposure to your company as fast as possible Making efficient use of resources Providing the most important and accurate information to enable informed decision making Enabling faster repair

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Determine the test cost Strategy is translated immediately into time required for testing, and time is converted into cost of testing Adjust your testing effort to available time If testing time is fixed, apply test estimation on test strategy to determine what can be achieved within the given time Communicate early with the customers Test strategy makes it clear which parts cannot be tested, or cannot be tested fully, and what risks will therefore be incurred

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. 1. Test Levels 2. Roles and Responsibilities 3. Environment Requirements 4. Testing Tools 5. Risks and Mitigation 6. Test Schedule 7. Regression Test Approach 8. Test Groups 9. Test Priorities 10. Test Status Collections and Reporting 11. Test Records Maintenance 12. Requirements Traceability Matrix 13. Test Summary 21

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. 22 Annoyance Loss of Revenue Loss of life

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. “Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite.... The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise.”-- Ward Cunningham 23 The current version of the plugin is 0.2 and uses the following formula to calculate the debt :

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Scrum Development periods have a fixed cadence (2-4 weeks) Teams are self-managed Teams work from a product backlog of user stories User stories define the high-level requirements  It is readable  It is testable Test Strategy Defines the high-level plan on how testing is going to be done Serves as a communication vehicle with your customer and the development team It is focused on finding defects as fast as possible Helps to identify and remove the risk of shipping a low-quality product

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. 25

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Learning Objectives: Understanding of customer personas Create several personas

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Amount of Data Amount of Detail User Archetype User Profile- Single User Market Data Actor And Agent User Role User Class/ Category Personas

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Usability and Customer Research is key to Scrum Expert reviews On-line surveys Field research with customers Persona building Usability Research 

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Personas are representations of the group of users of your product It is an amalgam that documents the behavior— goals, skills, attitudes, and environment—of the end-user It is based on research and knowledge of the end- users It is documented and used in both the development and testing of your product

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Edward Application Engineer “There is no ‘One Size Fits All’ with ISVs. Every ISV has a different environment, architecture and customer needs.” Ed has been working in this role for 4 years. He was a SW Engineer before that for 10 years. His work focuses on the implementation of AMT capabilities. Ed’s primary role is assisting ISV engineers in implementing specific features by customizing a solution for their given environment. Much of his day is spent troubleshooting issues and writing new code to test. Usually, Ed travels to the ISV and spends time face to face working with their implementation team. Given the economic climate, he has made changes to the way he interacts with his customers. Most of his interaction with ISVs is done over the phone and via . Goals: Simplify integration for ISV partners Solve issues fast Demonstrate AMT value Values: Good customer relationships Flexible architecture Good documentation Obstacles: Each ISV requires custom solution Troubleshooting issues remotely Translating code to meet ISV needs Design Implications: Design should demonstrate how a feature could be integrated--represent believable user experience Vanilla design = palatable for all potential customers Design should avoid appearing as a competing product

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Example Persona: Sean Senior Information Security Analyst “I am a security nerd.” Sean is a self proclaimed security nerd. This assessment seems justified considering his 15 years in the IT field—9 of which he has specialized in security. Currently Sean manages the patching pipeline for 110,000 systems and 35,000 servers. His priority is the health of these systems and the data they contain. He must ensure the security of the environment while minimizing interference to the workforce’s productivity. Another part of Sean’s job is to research and evaluate new technology—however, carving out time to do this can be challenging. His team is responsible for deciding what technologies will be implemented into the IT environment. All new technology is thoroughly vetted before releasing into the company. Goals: Maintain a secure environment Stay ahead of the next threat Minimal interference to workforce productivity Values: Data: securing user/corporate data; security intelligence; performance metrics Speed & efficiency Control & access Obstacles: Keeping confidential user info protected Differing platforms (& performance) across the company Rogue HDDs with unsecured data Difficult to manage remote/mobile systems due to an infrequent network connection End users introducing unapproved technology into environment. Design Implications: Easy to learn, intuitive interface Instant gratification Eliminate error-prone conditions—stakes are too high (data) Visualize data – represent client system status Reduce/control exposure to confidential info

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Customer satisfaction based testing Using the goals, motivations, and experience of the customer to set your path in session-based testing Helps to bring a higher priority to defects based on possible bad customer experiences Scrubbing the acceptance criteria of your user stories “What would care about?” Focusing the test environment The persona lets you know where it is going to be used 32

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Provides a detailed and consistent understanding of various audience groups. Addresses how teams supports the needs of a market Features and testing can be prioritized based on how well they address the needs of one or more personas. Provide a human "face" to the customer 33

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Although personas have many benefits, they alone will not ensure the success of the product. Business goals and technology constraints also are important to consider. Business UserTechnology Nirvana of SW products

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Agile is about building high customer value with each release…. but what customer? Customer personas create an amalgam of your target customers Utilizing this, you have a greater capability of satisfying your end-user market In theory, if you satisfy the persona, then you satisfy your market 35

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Personas can help you focus your test They require extensive user research Customer satisfaction based testing is a useful addition to your test methodology Remember that they don’t stand alone

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. 37

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Learning Objectives: Discover how the lifecycle and methods integrate together Review example criteria for measuring when a story is complete and when your product is ready to ship

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Scrum project framework puts three major constraints on testing Products are delivered on a fixed cadence and cannot be pushed out All features need to be working and meet the acceptance criteria No features are shipped to the customer if it is not tested, repaired, and retested User stories/features by design are expected to evolve Details and acceptance criteria in the backlog will evolve overtime May be deferred until the maximum amount of information is available Development of the product itself may fill in the gaps Customers may shift priorities They are the customer after all! 39 As nerve-racking as it seems, test planning needs to be distributed within each development sprint.

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. 1.Nightly build testing (Smoke Test) 2.Weekly regression 3.Story validation 1.Customer acceptance testing 1.Full Testing of new features 2.Regression of prior functionality 3.Distribution criteria completed 1.Requirements inspection 2.Establish story validation criteria 3.Customer reviews Project Backlog Product Backlog Product Iteration Sprint Backlog Daily Standup Defects 

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Slide 41 Project Concept Requirements Gathering Adapted from : Agile Project Management, Jim Highsmith Sprint 1 Theme Architecture 1.1 Feature 3 Development Feature 2 Development Customer delivery, review, update requirements Sprint 2 Theme Architecture 1.2 Feature 1 Development Feature 5 Development Sprint n Theme Architecture 1.n Feature(s) n Development Sprint 0Sprint 1Sprint 2Sprint n Customer delivery, review, update requirements Customer delivery, review, done Hard/Complex/Must Haves Less Complex

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. 42

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. 43 Application Scaffolding Application Build-outCleanup Retrospective and Customer Survey Requirement Inspections and Customer Reviews Regression And Formal Customer Testing Core Sprint Testing Methods Customer Engagement and Usability Studies Story Boarding and Prototypes

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. 44 Continuous Integration Release Testing Atomic / Session-Based Testing Inspections Tuesday is a better release day than Friday Important Make sure the story is testable!

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. DoCheck Product Proposal and Description User Story Rev x.x Develop or Update User Story Story Testable? Develop or Update Validation Criteria Review Criteria with Product Owner/Customer Passed Review Product Design Document User Story Rev x.x -1 User Story Rev x.x +1

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. User Story is reviewed for testablity If is not testable, then it is moved to “Blocked”

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Story is implemented per user story description with no blocked issues Code has been reviewed by the author Integrated and works on engineering baseline Author has run unit level testing on story and has high confidence of being done Usage documentation and help files updated Architecture document updated Build files updated

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. 48 The quality of the testing is dependent on the tester's skill of inventing test cases and finding defects. The more the tester knows about the product and different test methods, the better the testing will be.

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. A software test method that combines accountability and exploratory testing Developed by Jonathan and James Bach Test sessions document: Charter Session Session Report Debrief Parsing Results Planning and debrief sessions 49

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Passes peer inspection Passes integrated build testing Code coverage targets have been achieved Meets the validation criteria established/defined with the customer

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. A story is complete when: Story is implemented per user story description Meets the validation criteria established and defined with the customer Passes the code inspection and peer review Passes unit level testing Integrated and checked into the engineering baseline Passes regression testing Usage documentation and help files updated Etc…

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Change-Based Test Management (CBTM): Software validation methodology centered on monitoring change to maximize the effectiveness of the validation cycle Change-Based Test Management is about: Establishing a link between your test cases and regions within your product  Tests built to target a region of code  Test code coverage analysis Monitoring the changes as your product is being developed  Source control delta reports Priority sorting of tests  Critical and changed regions first  Unchanged regions last

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. CBTM Test Suite Test ListPrioritized List Test 1 Test 2 Test 3 Test n Test 4 Test n Test 1 Test 4 Test 2 Test 3 Test Coverage Product Delta Report First Last Remember that the longest test dictates the test process time

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Minimum Criteria: Obtain any required external standards compliance certification Complete smoke test and regression of final build for integration Remove debugging and testing code from software Document customer-visible defects for release notes Install/uninstall of product on clean system; ensure instructions are complete and accurate Perform virus scan of all release files, image and media Verify the presence/accuracy of copyright notices Verify that no unintended disclosure of restricted information is present Verify that all trademarks are correct

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Regression is key to making sure that the prior sprints completed stories stay “done” It is also important that customers don’t see things consistently breaking Pick a regression strategy that fits the complexity of the product

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Master Done List Top Customer Features Defects Repaired Other Weekly: Verify a random pick of 10% Release: Regress all new features Passes global product requirements Verify a random pick of 30% Anything you may have a hunch about Weekly: Verify a random pick of 5% Release: Verify a random pick of 10% Weekly: Verify a random pick of 5% Release: Regress all prior critical defects for 3 sprint cycles Past Sprints

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Prioritizing Criteria Has higher likelihood of having defects If broken, will have a higher impact on the customer Frequency of use by customer If broken, will cause more failures in other requirements Trust your Spidy-sense

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. New Things New features, concepts, technology, tools High Complexity Functions with many interfaces (functions that require interfacing with external systems) or high complexity Frequently changing things Functions that have a lot of updates or bug fixes (frequently changing code) Functions with poor requirements and design Ambiguous specs can lead to incorrect or conflicting implementations Dependencies If broken, will cause more failures in other requirements

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Rushed work Some tasks or projects are chronically under-funded and all aspects of work quality suffer Functions developed under extreme time pressure Late changes: Rushed decisions, rushed or demoralized staff lead to mistakes Tired Programmers Long overtime over several weeks or months yields inefficiencies and errors Learning Curve Functions owned by new developer

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Some of the reasons why a feature may be out of the testing scope Low priority as a result of risk analysis Tested before and is considered stable Will not be included in the current release Will be tested by third party Will be released but not tested or documented as a functional part of the current version

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. It is important to establish clear entry criteria to each phase of the scrum development cycle “Done” needs to measured more than once Regression is key to having continued user satisfaction

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Always review your strategy with your management, customers, subcontractors, and development team members Test strategies for Scrum need to focus on ensuring high customer satisfaction Invest in user research and use tools like personas to focus your testing effort and priority Invest in growing your people. None of this can be done without the talented people who know how to break things

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. 63

Copyright © 2009 Intel Corporation. No part of this presentation may be reproduced in any way without written permission of Intel Corporation. Various Authors, Exploratory Testing, Wikipedia Various Authors, Test Strategy, Wikipedia Various Authors, Scrum (development), Wikipedia Various Authors, Session-based testing, Wikipedia The Scrum Alliance, Ray Arell, Change-Based Test Management, (ISBN: ) James Bach, Heuristic Risk-Based Testing, STQE 11/99 James Bach, Risk and Requirements-Based Testing, Computer, June 1999 Ingrid Ottevanger, A Risk-Based Test Strategy, StarEast 2000 Bret Pettichord, The role of information in Risk Based testing, StarEast 2001 James Bach, Risk-Based Testing Troubleshooter, Paper Draft Erik Petersen, Smarter Testing with the 80:20 Rule, StarWest 2002 Anne Campbell, Using Risk Analysis in Testing, StarEast 2000 Paul Gerrard, Risk-Based E-Business Testing, System Evolutif Gregory T Daich, Defining a Software Testing Strategy Jim Highsmith, Agile Project Management Ruku Tekchandani, Building a Effective Test Strategy John Pruitt and Tamara Adlin, The Persona Lifecycle Pettichord, Kaner, Bach, Lessons Learned in Software Testing, on-line Jonathan Bach, Session-Based Test Management,