1. COMP8130 and 4130Adrian Marshall 8130 and 4130 Test Management Adrian Marshall

2. COMP8130 and 4130Adrian Marshall Overview Principal Testing Activities Testing Problems Scoping, Scheduling and Estimating Planning Documenting Controlling Test Teams (and independence)

3. COMP8130 and 4130Adrian Marshall Principal Testing Activities Planning scope, approach, resources, budget, documentation set, roles & responsibilities, schedule Preparation test documents, tools, environments, training Execution developers, systems integrators, users, external specialists, Independent V&V agents Reporting informal, formal, automated defects, progress, resource usage, issues & risks….

4. COMP8130 and 4130Adrian Marshall Testing Problems Modern software systems present the following potential problems to testing: Large, complex and expensive distributed processing / multi-tier systems are common Lack of a single end-to-end solution development owner Variability of component maturity and quality Variability of supplier capability and workmanship Significant component interface and integration problems

5. COMP8130 and 4130Adrian Marshall Testing Problems Modern software systems present the following potential problems to testing: Large user populations, and large numbers of concurrent users Unpredictable user loads which are often driven by uncontrollable circumstances Enterprise wide implementation of software systems creates critical business dependency on systems availability ….

6. COMP8130 and 4130Adrian Marshall Testing Problems Modern software systems present the following potential problems to testing: Significant component interface and integration problems Large user populations, and large numbers of concurrent users Unpredictable user loads which are often driven by uncontrollable circumstances Enterprise wide implementation of software systems creates critical business dependency on systems availability ….

7. COMP8130 and 4130Adrian Marshall Testing Problems Problems can occur in: Quality focus; Test scope; Test priorities; Test budget; Test schedule; Test skills and experience; Test documentation; Communications.

8. COMP8130 and 4130Adrian Marshall Scoping, Scheduling and Estimating How do we scope the testing? Understand the system to be tested (requirements) Understand the risks Understand the development process Understand the technology Understand the business objectives

9. COMP8130 and 4130Adrian Marshall Scoping, Scheduling and Estimating Where do we get our requirements from? Operational requirements documents User feedback from prototypes, field trials, site visits Existing systems and their documentation Customers: System specification(s) Statement of work, Contract MIL, ISO or AS Standards Customer Furnished Information

10. COMP8130 and 4130Adrian Marshall Scoping, Scheduling and Estimating Where do we get our requirements from? Specifications: Interface Specifications Product Requirements Specifications, Marketing Specification Drawings Technical Specifications and standards

11. COMP8130 and 4130Adrian Marshall Scoping, Scheduling and Estimating Where do we get our requirements from? Corporate: The bid Marketing Specification Company and project policies, standards and procedures Existing design

12. COMP8130 and 4130Adrian Marshall Scoping, Scheduling and Estimating Analysing the risks Hazard identification Business risks System risks Technology risks Management risks Risk sizing Risk management / mitigation

13. COMP8130 and 4130Adrian Marshall Scoping, Scheduling and Estimating Understand the development process (and influence it?) Incremental Iterative Use Case definitions Integration methods Phasing of builds

14. COMP8130 and 4130Adrian Marshall Scoping, Scheduling and Estimating Understand the technology (examples): PKI Embedded systems Windows / Unix Smart-card EFTPOS jpeg

15. COMP8130 and 4130Adrian Marshall Scoping, Scheduling and Estimating Understand the business objectives (examples): Time to market critical Phased delivery Low price, high volume sales Security, confidentiality, IP protection

16. COMP8130 and 4130Adrian Marshall Scoping, Scheduling and Estimating Scoping. Finally….. …….determine the required tasks to fulfill the scope….. Inspections, unit, integration, system, acceptance, performance, reporting

17. COMP8130 and 4130Adrian Marshall Scoping, Scheduling and Estimating Scoping exercise: Consider requirements, risks, development process, technology and business objectives for the development of: a.R&D program for biometric authentication; b.Game based on the latest movie; c.Fuel management system for the A380

18. COMP8130 and 4130Adrian Marshall Scoping, Scheduling and Estimating Estimating Understand the scope Understand the size of the system (may use COTS products) and the size of the development Understand the complexity of the system Historic data for similar testing scope Gut feel Rule of thumb

19. COMP8130 and 4130Adrian Marshall Scoping, Scheduling and Estimating Estimating - Useful numbers: Overall testing effort (all activities) = 30% of the Test planning = 15% of total test effort Designing and documenting tests cases = 10% of total test effort Test procedures and test execution = 70% of total test effort Test reporting = 5% of total test effort

20. COMP8130 and 4130Adrian Marshall Scoping, Scheduling and Estimating Estimating Evaluate estimates Record actual effort (test metrics) Don’t be too granular (testers aren’t accountants or lawyers) Refine estimates Refine schedule

21. COMP8130 and 4130Adrian Marshall Scoping, Scheduling and Estimating Scheduling Identify the sequence of events Identify the critical path Communicate the draft schedule  Development team  Project manager Refine the schedule (communicate and changes to the risk profile)

22. COMP8130 and 4130Adrian Marshall Scoping, Scheduling and Estimating Scheduling and estimating exercise (small groups): Three month schedule from start of development to release First build due at the end of the second month 12 developers working fulltime Estimate the effort, size of the test team, and present a simple schedule

23. COMP8130 and 4130Adrian Marshall Planning Building the test plan: Scope, Schedule and Estimate Explain the test execution strategy Allocate roles and responsibilities Outline the testing schedule Identify test resource requirements List assumptions, issues and risks Approving the test plan Have appropriate stakeholders authorise / sign the plan….

24. COMP8130 and 4130Adrian Marshall Planning Planning should be done very early - be part of the project bid / project initiation team Obtain clear solution success / acceptance / quality criteria to enable identification of test focus areas Identify & evaluate risks and create a testing strategy which helps to manage them Produce a professional and defendable Test WBS Be appropriately independent Act as a support service not an authority figure Keep testing team and ‘customers’ informed Be involved across the project lifecycle….

25. COMP8130 and 4130Adrian Marshall Documenting Why do we document? What do we document? How do we document? Master Test Plan Entry and exit criteria Unit level testing documentation Documentation for subsequent phases Plans are not static… they must be constantly evaluated Test Cases Test Procedures Test Reports

26. COMP8130 and 4130Adrian Marshall Controlling What is needed to control the activities? Metrics Rate of test design Rate of execution Product quality Product stability Reporting

27. COMP8130 and 4130Adrian Marshall Test resources Test Teams Test Environments Test Tools

28. COMP8130 and 4130Adrian Marshall Test resources Test Teams Independence Identify skills / experience Hiring / contracting Motivating / retaining

29. COMP8130 and 4130Adrian Marshall Test resources Test Environments Procure Develop Control/manage

30. COMP8130 and 4130Adrian Marshall Test resources Test Tools Evaluate Procure Develop Validate/verify Training