1. How to Know That What You Want Has Been Done- 1 Claire Lohr Member, SESC Management Board Chair IEEE 829 Working Group lohrsys@erols.com How to Know That What You Want Has Been Done STC May 1, 2002

2. How to Know That What You Want Has Been Done- 2 Topics Sources of the information presented Inclusion of best practices from industry Definition of the steps for recognition of successful completion of software projects Applicable references NOTE: Best practices marked with

3. How to Know That What You Want Has Been Done- 3 Sources of this Information Highly competitive commercial industries –Semiconductor manufacturing –PC manufacturing –Insurance –Finance –Telecommunications –Etc. Some succeed Some don’t

4. How to Know That What You Want Has Been Done- 4 The Challenge Current formal definitions of quality are … Too broad for software Real need is to attain the customer’s own specific, detailed quality goals

5. How to Know That What You Want Has Been Done- 5 Steps for Recognizing Quality Step #1: define what you don’t want Step #2: define what you do want with sufficient detail Step #3: hire and train good people Step #4: prepare to measure the results (test outputs) in parallel with defining the target (requirements) Step #5: refine with business analysts throughout lifecycle

6. How to Know That What You Want Has Been Done- 6 Step #1 Define what you don’t want (the business case) “ IT buyers are mad and aren’t going to take it anymore … Up to 50% of software cost results from ongoing maintenance and integration activities … We spend so much time and money dealing with software problems, it's ridiculous … Some patches break more things than they fix.” Karyl Scott Information Week January 14, 2002

7. How to Know That What You Want Has Been Done- 7 Step #2: Define What You Do Want Define what you do want (general) –From the Scott article “The Software Bill of Rights, released this week [Jan. 14 2002], stipulates that buyers have the right to expect a quality product, accurate delivery schedules, explicit pricing, development accountability, and effective technical support.” –Universally accepted definitions of quality Meets specifications Meets customer expectations Fitness for use

8. How to Know That What You Want Has Been Done- 8 Define what you do want (specific) –Document requirements and SQA IEEE Std 830-1998 Recommended Practice for Software Requirements Specifications IEEE Std 730-1998 Standard for Software Quality Assurance Plans –Document requirements with Use Cases for more clarity Visual Shows flows Includes everybody –Plan testing in parallel from the beginning IEEE Std 829-1998 Standard for Software Test Documentation Step #2: Define What You Do Want

9. How to Know That What You Want Has Been Done- 9 Use Case Example Step #2: Define What You Do Want Attend IEEE Track Use case Actor SEPG Member Association Attend STC Order SESC standards Visit computer.org

10. How to Know That What You Want Has Been Done- 10 Step #3: Good People Hire and train good people (a good process enhances talent does not replace it) –Thorough employment screening –“Boot camp” training for entry level new hires –Employee friendly work environment ($ to $$$$) –Low turnover –Easy access to training Incorporate standards for process and documentation Use real examples online and in class

11. How to Know That What You Want Has Been Done- 11 Step #4: Prepare Tests and Requirements Prepare to measure the results (test outputs) in parallel with defining the target (requirements) –Helps developers clarify their thinking when answering tester’s questions –Assures testability of the final software –Defines “pass criteria” for the software early in the life cycle –Allows time for acquisition or development of test tools (less use of tools leads to less testing)

12. How to Know That What You Want Has Been Done- 12 Test Documentation (from IEEE Std 829-1998) Test Exe- cution Test Plan Test Design Specs Test Proc Specs Test Case Specs Test Log Test Incident Reports Test Summary Report Always Tailored Real examples available on- line Step #4: Prepare Tests and Requirements

13. How to Know That What You Want Has Been Done- 13 Test Plan Outline 1. Test-plan identifier 2. Introduction 3. Test items 4. Features to be tested 5. Features not to be tested 6. Approach 7. Item pass/fail criteria 8. Suspension criteria and resumption requirements 9. Test deliverables 10. Environmental needs 11. Responsibilities 12. Staffing and training needs 13. Schedule 14. Risks and contingencies 15. Approvals Step #4: Prepare Tests and Requirements

14. How to Know That What You Want Has Been Done- 14 Add to the front-end of the lifecycle –Requirements for testing Choose how many levels of testing for this project Choose documentation for this project Choose metrics –Master Test Plan Over 10 million LOC product line (MANY levels of test to sort out) Identify test levels Hand-off criteria between levels Email lohrsys@erols.com for a template example Test Documentation Tailoring Options

15. How to Know That What You Want Has Been Done- 15 Skip documents –Test Plan Management issues like schedule, and staffing are covered in other documents These issues are not under control of testers Combine documents –Specifications –Procedures –Log Test Documentation Tailoring Options

16. How to Know That What You Want Has Been Done- 16 Example of combination of Std 829 Specification, Procedures, and Log in MS Word on the next slide Email lohrsys@erols.com for an example of this template with an explanation of all of the fields Tailoring Options

17. How to Know That What You Want Has Been Done- 17 Tailoring Options

18. How to Know That What You Want Has Been Done- 18 Design Require- ments Code Unit and System Tests Accept- ance Test Test Plan V1.0 Revise Test Plan and develop the rest of the test documentation Test Doc’s V2.0 Test Doc’s Vx.y Step #4: Prepare Tests and Requirements

19. How to Know That What You Want Has Been Done- 19 Plan coverage measures for tests; examples: –Source Lines of Code (SLOC) –John Musa’s Operational Profile Use detailed checklists for test case development (examples for web site testing follow) Put the checklists in an understandable framework Step #4: Prepare Tests and Requirements

20. How to Know That What You Want Has Been Done- 20 Step #4: Prepare Tests and Requirements Web testing checklist framework

21. How to Know That What You Want Has Been Done- 21 Step #4: Prepare Tests and Requirements

22. How to Know That What You Want Has Been Done- 22 Step #5: Refine with Business Analysts Refine definition of “correct” for software throughout lifecycle –Testers have constant access to “business” analysts (similar to JAD) They use a combination of formal and informal process There is a continuity of personnel (same business analyst from project to project) Business analysts are physically located nearby Authoritative answers for questions (typical testing challenge) Business analysts help deploy to in-house user community

23. How to Know That What You Want Has Been Done- 23 Summary Define what you think you want Keep effective communication open to refine requirements Develop specific tests to prove the goals have been met Retain good people and maximize their potential

24. How to Know That What You Want Has Been Done- 24 References IEEE Std 730-1998 Standard for Software Quality Assurance Plans IEEE Std 829-1998 Standard for Software Test Documentation IEEE Std 830-1998 Recommended Practice for Software Requirements Specifications