High Level: Generic Test Process (from chapter 6 of your text and earlier lesson) Test Planning & Preparation Test Execution Goals met? Analysis & Follow-up YES EXIT NO

Test Planning & Preparation (important but not always fully performed) Two important items in a test plan include: – Specifying Test Goals & Targets in terms of: Reliability Performance Customer satisfaction etc. – Set General Test Strategy which includes: What & how much to test Techniques of testing Tools needed Measurements and Analysis (details of) People & Skills Schedule Setting goals also sets the exit criteria & possibly guides the testing strategy

Testing model Part of Test Strategy is to decide on a testing model via: 1.Collecting information to address & choose a test model Usage –based testing (info on customers’ actual usage pattern) White-box testing (guided by black box testing results) Mutation testing (programming language, past experience, etc.) 2.Analyze and construct the appropriate test model & techniques 3.Validate the model and incrementally improve the model for continuous usage From the Test model then develop test cases: 1.static test cases: whole/partial program; inputs; etc 2.dynamic test runs: whole/partial execution; artificial stop points; etc.

Sample 1: Model from “past” data of your group Percentage Breakdown of Problem Discovery and Fixes 10% 40% 25% 15% 10% ReviewsFunctional test System test Component test customer reported How would you use this for planning? Problems Detected

Sample 2: Model from “past” data of your group Percentage Breakdown of Problem Discovery and Fixes 50% 30% 10% test Technique 1 test Technique 2 test Technique 3 customer reported How would you use this for planning? Problems Detected

Sample 3: Distribution of Error Prone Areas Release 1 customer reported bugs Problems Detected How would you use this for planning Release 2 testing? F1F3 F2 F5 F4

Test Suite and Test Procedure Preparation Test case development is based on, besides test model, a broader functional area such as a usage-scenario which may be composed of several steps --- thus several related test cases needs to be prepared as a test suite Test suites needs to be “managed” (with test mgmt tools) – Categorized – Stored – Reused Test Procedure Preparation (especially important for component & system tests) : – Sequencing of test case execution : based on some relationship (safety-hazard to avoid accidents, usage scenario, etc.) – Problem fix between related test cases : ensure timely fixes for a problem found in the middle of a sequence of test cases or test suite. Developers and testers need to be in synch --- no early release of developers

Don’t Froget: Test Risks and Contingencies Examples of Risks: – Changes to original requirements or design – Lack of available and qualified personnel – Lack of (or late) delivery from the development organization: Requirements Design Code – Late delivery of tools and other resources State the contingencies if the Risk item occurs – Move schedule – Do less testing – Change/relax the goals and exit criteria

Sample: Recording Test Case & Test suites Test Case number (use some scheme as your text book shows ) Test Case author A general description of the test purpose Pre-condition: “includes test results of other pre-req modules” Test inputs Expected outputs (if any) Post-condition Test Case history: – Test execution date – Test execution person – Test execution result (s) : Pass Fail If failed : Failure information : fix status

Test Execution Allocating – Test time – People resources and – Tools (test tools and test systems) Execution – Load the system to be tested – Invoke and run the test cases (input data as needed) – Collecting test result data and measurements Results analysis – Identify failure/success – Collect & Record failure information – Submit failed test cases to development (coder/designer) Actually, of preparation & execution

Test Result Checking & Reporting Basically comparing against expected test result as stated in the test case descriptions. Some specific behavior or sequence may require one to refer back to requirements document or even user. (especially if the test case description is not complete) “Good” problem descriptions can help in: – Clearer understanding of the problem – Aid in problem diagnosis and location of problem – Even possibly in problem fix Non-functional problem such as “consistency or looks” may require experienced tester to analyze

Collecting Test Measurements In-project measurements ---- – helps us decide if we reached the goal and can stop testing – What and how much to test for system regression Post project measurements --- helps classify – Defect types – Defect severity – Impact areas – etc. The measurements and analysis provide us with assessments of : - product reliability, - test coverage - etc.

Test - Human Resources (QA Professional) Test Planning (experienced people) Test scenario and test case design ( experienced people and test discipline educated) Test execution (semi-experienced to inexperienced) Test result analysis (experienced people) Test tool support (experienced people) Test Organization: – Vertical model by each product (small organizations) – Horizontal model (large enterprises) May include external people: – Users – Industry experts (for COTs )

“Clearing a Career Path for Software Testers” by E. J. Weyuker, T.J. Ostrand, J. Brophy and R. Prasad in IEEE Software March/April 2000 Describes a Software Test Professional Development Program in AT&T Basic knowledge includes: computing systems and architecture, software development process, operating system, database & quality principles Tester skills in: Career Path – Test planning - Test Apprenticeship – Test case design - Test Engineer (Mastery of some skills) – Test execution - Test Specialist (expert in some skill) – Test data adequacy analysis - Test Leader – Test automation - Test Manager or Enterprise Tester

Test Automation Test case Generation – hardest area Test plan and management – semi-automatic versions exist Test execution – keystroke capture and replay has been popular for years Test result analysis – some compare analysis to designate simple pass/fail Test coverage – lots of tools Check out some test tools such as HP/Mercury and others --- via on-line search