2 Confidential Agenda Background Test Planning: Three Issues Test Planning: Three Solutions Practical Tips to Help you in Test Planning Conclusions Q&A
3 Confidential Reasons Why Test Planning is Difficult Planning Expertise Application Expertise Forecasting Expertise Testing Expertise
4 Confidential Test Planning Issue 1: Knowledge (1) Right Mix is Critical! Requirements Requirements: Implicit vs. explicit knowledge (people vs. docs) Expertise Thorough knowledge of applications and flows Breath vs. depth Testing Domain Application Domain
5 Confidential Test Planning Issue 2: Project Management (1) Test Planning is Project Management within Project Management! It is NOT optional Be prepared to articulate WHY is needed Plan – Prepare – Execute motto Take time to do proper planning Vanilla flavour plan is useless
6 Confidential Test Planning Issue 2: Project Management (2) Environment Data Tests
7 Confidential Test Planning Issue 3: The Gruyere paradigm How many holes are in this cheese? What is actually delivered vs. what was planned to be delivered? Are all the features truly complete?
8 Confidential Test Planning Solution 1: The Expert (1) KNOW the WHAT… Industry Knowledge Application/Environment Knowledge Testing Knowledge
9 Confidential Test Planning Solution 1: The Expert (2) Become the testing guru Help your team to become the testing experts Some Ideas: Formal training/certification Attend conferences But there are also cheaper ways: Lunch and Learn sessions Organize workshops with presentations Test Planning Risk-based approaches Test terminology Test case design Metrics Test tools Create a testing library
10 Confidential Test Planning Solution 1: The Expert (3) Build your credibility by becoming the industry/application expert Read ALL documentation Review the requirements, design, architecture Talk to users, developers Dont be afraid of asking questions Ensure to communicate your status properly Assume responsibility when needed
11 Confidential Test Planning Solution 1: The Expert (4) And there is more… Soft Skills! Communications (defect reporting) Detective Negative thinking or What can go wrong Time management Discipline and Tenacity Creativity
12 Confidential Test Planning Solution 1: The Planner (1) KNOW the HOW Testing scope Discuss under/over testing Who will be doing what What needs to be in place before test starts Test case design guidelines Tool usage Metrics to be collected Entry/Exit criteria …
13 Confidential Test Planning Solution 1: The Planner (2) You will reap what you sow Planning is the most essential activity! PLAN, PLAN, PLAN …. You cannot prepare what you have not planned for You cannot execute what you have not prepared for!
14 Confidential Test Planning Solution 1: The Planner (3) Make your Master Test Plan the Bible Document ALL your assumptions Think of ALL risks Use ALL information available to you Establish in/out scope INCLUDING Test boundaries Brainstorm and define what will be tested when! Phased approach Dont forget Environment Test Data Access/Security
15 Confidential Test Planning Solution 3: The Weather-Man (1) Look at signs to forecast quality issues Previous history (application, business function, etc.) Get to know the team Collect and store metrics you will need them! Clearly communicate deviations from plan based on unexpected quality issues
16 Confidential Test Planning Solution 3: The Weather-Man (2) Establish and Communicate Forecast for your Plan: Failure rate SLAs for defect fixes Environment availability
Confidential Practical Tips to Help you in Test Planning
18 Confidential ALWAYS Use Risk-Based Approach! (1) Spend your effort where is worth it,…Unless you have unlimited time/budget/resources Risk-based approach will allow you to test More comprehensibly what really matters First what is more important If needed, know where to cut testing NOT an isolated task, you need to socialize approach and get agreement on risk areas
19 Confidential ALWAYS Use Risk-Based Approach! (2) Technical risks assessment should include: New and immature technology, Brand new system or enhancement New and difficult to understand business requirements, Complexity of business logic, Complexity of technology, Size of the project, Skill mix on the project, Number of users (including customer facing or back office considerations) historical data when appropriated (the risk is higher for modified functions that have frequently failed in the past, i.e., Do you have any historical data of defects by area?)
20 Confidential ALWAYS Use Risk-Based Approach! (3) Business Impact assessment should include: Importance of features from client/customer perspective Criticality of features from the business perspective Frequency of use
21 Confidential ALWAYS Use Risk-Based Approach! (4) As a rule of thumb: High Impact: Calculation Validation Legal Front end (end user) High frequency, High volume of users Medium Impact: Data changes Wrong information displayed Medium frequency and groups of users Low Impact: Display Visual appearance Used by a few users Back office
22 Confidential Risk-Based Approach Process 1. Organize test risk assessment workshop with business and technical stakeholders 2. Identify and list requirements, business functions 3. Evaluate technical and business impact (rate 1-3) 4. Evaluate probability of failure (rate 1-3) You might want to read: http://www.satisfice.com/articles/hrbt.pdf Risk FactorTesting PriorityTesting Extent 7-9HighExtensive 4-6MediumComprehensive 1-3LowSmoke
23 Confidential Risk-Based Approach: Business/Requirements Map Accounting Finance Product Filing Req n-7 Licensing DistributionCommunications Req n-4 Req n-1 Administration Extensive Testing High Degree of Testing Less Testing Required
24 Confidential Cover all aspects of testing When planning… make sure to cover the overall process: Test PreparationTest ExecutionPlanningReporting Test Results Problem / Defect Reports Test Plan (s) Update Test Project Plan Create Regression Test Suite Test Environment set up Test Data set up Test Cases Create Automation Test Suite Test Exit Report Lessons Learned - Recommendations Final Test Report Strategy Test Strategy Test Project Plan (Schedule) Test Coordination
25 Confidential Schedule Templates Detailing All Activities (1) Have an overall schedule template that you can easily fill with specific data Activity Work- Effort (in days) Start DateEnd Date # of Res Resource Details Review Business, Technical Requirements0YY/MM/DD 0Test Manager, Test Leads, Test Analysts Create Test Strategy/Master Test Plan/Traceability Matrix0YY/MM/DD 0 Test Manager, Test Leads, BAs Review Test Strategy/Master Test Plan0YY/MM/DD 0Project Team, Stakeholders Sign-off Strategy0YY/MM/DD 0 Create detailed Test Schedule0YY/MM/DD 0Test Manager Review Test Schedule0YY/MM/DD 0Project Team Sign-off Test Schedule0YY/MM/DD 0 Creation of System Test Cases0YY/MM/DD 0Test Team Review Test Cases0YY/MM/DD 0BAs, BSAs Update Test Cases0YY/MM/DD 0Test Team Create Test Data0YY/MM/DD 0Data Analyst Sign-off System Test Cases0YY/MM/DD 0 Review Regression Test Bed0YY/MM/DD 0Test Team Update Regression Test Bed0YY/MM/DD 0Test Team
26 Confidential Schedule Templates Detailing All Activities (2) Activity Work- Effort (in days) Start DateEnd Date# of ResResource Details Create/Select UAT Test Cases0YY/MM/DD 0Business Reps Review Test Cases0YY/MM/DD 0Test Leads Update Test Cases0YY/MM/DD 0Business Reps Create UAT Data0YY/MM/DD 0Data Analyst Sign-off UAT Test Cases0YY/MM/DD 0 Execute System Test Cases0YY/MM/DD 0Test Team Review exit criteria and produce System Test Report0YY/MM/DD 0Test Manager Sign-off System Test Results0YY/MM/DD 0 Execute UAT Test Cases0YY/MM/DD 0Business Reps Support UAT Execution0YY/MM/DD 0Test Team Produce UAT Test Report0YY/MM/DD 0Business Lead, Test Manager Sign-off UAT Results0YY/MM/DD 0
27 Confidential Systematic Entry/Exit Criteria Entry CriteriaExit Criteria Unit/Integration Testing exit criteria has been fulfilled and approval provided System Test Plan is approved or Approach is documented in the Overall Test Strategy/Plan All system test cases as per plan have been executed System scenarios, conditions and cases are complete and approved All detected fixed defects have been re-tested Test data has been created and is readily available in system test environment No outstanding Severity 1 or 2 defects Test environment is set up and operational System Test Report has been completed, reviewed and approved Plan Test Cases/Data Readiness Environment Readiness Planned Activities Quality Communication & Sign-off Preconditions
28 Confidential Forecasting with Metrics – Failure Rate/Env Document what is the expected failure rate Existing application/delivery team: use previous data Be as granular as possible By Functional Area By Application If you dont have data, rule of thumb is around 30% (i.e., 30% of test cases will fail during test execution) Incorporate retesting activities to schedule
29 Confidential Forecasting with Metrics – Defect Age SLAs Negotiate and document SLAs for defect fixing, then measure How long defects remain in New state (i.e., not assigned?) How long defects remain in Assigned state (Are too many defects assigned to the same resource and therefore there is no progress? How long defects remain in Fixed state (i.e., before being closed by Test Team)? (Is testing team following up on defect fix verification? Are there competitive priorities (execution vs. verification)? A high number of defects in Need Info state could indicate that there are concerns with the requirements, lack of details on the defect (to reproduce).
30 Confidential Conclusions DO PLAN, PLAN, PLAN …. AND THEN PLAN! Get buy-in into risk-based approach Create templates to help you be efficient Be systematic Use historical information/metrics to predict future Share your documents, status and metrics so stakeholders understand challenges Hope for the best, but plan and prepare for the worst! DONT Work in isolation Over complicate your methodology Think you dont need to understand domain Do ad-hoc testing (unless it is planned) Be inconsistent Keep issues, departures from assumptions to yourself Be naïve and plan for the best!
_experience the commitment TM THANKS!! Questions??
32 Confidential Contact Information Elizabeth DAngelo, M.Sc. PhD Director, Quality COE 150 Commerce Valley Drive West Markham, Ontario (905) 695-6513 (416) 459-1257 cell firstname.lastname@example.org