Most Important Tests (MITs) Most Important Tests is just a method which focuses on the risky and important areas of the system to test –Based on statistical analysis of the “risky” areas of the system that needs testing. Where is the statistic coming from ? –Inspections and reviews? –Personnel history of the developers? –Past projects –The risky areas are then prioritized to identify the “important” areas How would you prioritize them? –Gain agreement on the prioritized list of test areas Make sure resources are received for the test effort

Example from your Assignment 1 Test 1 Test 2 typeReview (formal/inform) material Requirements Specification doc size3 pages major functions4 effort expended 3 x 30minutes = 1.5 pers.-hr total problems found * 6 effectiveness 90 p-m/6 bugs= 15p-m/bug found Code execution “Unit tested” code * Breakdown of the problems found and their resolution Any other items that you may want to track ?

Breakdown of Problems Found Prob. # # 1 Severity* high * There exists a severity description Functional AreaStatus Get statistics by: - Functional area - Severity - Author -Status - etc.

MITs Methodology Provides “tools” for sizing the test effort that allows trade-offs: –Test coverage –Resources –Schedule “tools” for tracking the testing activities’ progress (using S-curve) –Track problems found –Status reporting –Determining end of testing

S- curve Time or Effort Total # of Problems found Return on testing effort starts to Diminish (time to consider stopping)

How do We Use S-Curve? 1.Track a set of tests of a “product” over a period of time. Run a set of tests (covering different areas) Fix the errors Rerun the set of tests What is your guess of the “shape” of S-curve? 2.Track a series of different tests (reviews, unit test, functional test, etc.) of different “stages of a product” over time. Conduct requirements review Conduct design review Conduct unit test Conduct component test Conduct system test What is your guess of the “shape” of S-curve?

Deciding when to Quit Testing Seed the product with errors – call it x Keep account of the number of seeded errors discovered --- call it y Keep track of real errors found – call it r Then tabulate or estimate the total number of errors in the product --- call it total, utilizing the following relationship. – y/x = r/total – total = r * x/y Ask if you can live with an estimated (total – r) errors left in the system.

Set of MITs Activities 1.Publishing test inventory containing all assumptions and requirements of test 2.Count and enumerate all the identified test areas and test cases (not necessary to be executed) 3.Prioritize and rank the enumerated test areas and cases 4.Estimate the effort required to execute each test case 5.Negotiate with management or person who holds resources on the required resources for each of the test coverage/schedule trade-offs.

Set of MITs Activities (cont.) 6.After picking the most important areas for testing perform the following for designing the actual test cases: –path analysis –Data analysis 7.If necessary, renegotiate the resources, schedule, etc. once more is known about the test cases 8.Execute the test cases 9.Track the test progress utilizing the S-curve and determining when to stop 10.Perform a test post mortem to evaluate and plan for improvement. Ideally, these steps should all be utilized, but in reality some are skipped.

Factors Influencing MITs Activities Ease of Implementation –Some steps require a lot pre-training –Some steps require very special skills Cost of Implementation/Return on Investment –Problems found and fixed during pre-release require resources and time. –Problems that are found and fixed during post release require another set of resources and time –(Which is more costly ?) discuss Development Methodology –Some methodology has extensive built in test steps (from planning to tracking); others don’t.

Easier MITs Activities Bug and problem tracking Test inventory & test coverage Planning and performing path and data analysis Ranking the test areas by risk Test effort estimation Test performance metrics Within these activities which ones are easier than others? Why? What do you use as criteria? (resource need, technical skills, etc.)

Harder MITs Activities S-curve analysis –Requires some experience and past data to even have an S- curve Test re-run automation –Needs to capture the conditions of the previous run –Capturing the test cases for automatically re-run pays off if the test cases are re-ran multiple times, especially for software products with multi-releases. Automated test plan generation –Developing test areas from requirements may be easier if Use Cases or other better defined requirements language is used

How Much MITs ? The steps in MITs ought to be selected and modified according to the type of project –May use the full range of MITs if the software project is run: Using traditional “waterfall-like” process Large and complex software Risk averse Highly mission critical –May emphasize certain parts of MITS over others if the software project is run: Using a more Agile approach such as eXtreme programming Small to medium size software Less risk averse not mission critical

How Test Fits in for You? RequirementsDesignCode Test Planning Test Prep Test & Fix RELEASERELEASE

Word of Caution Current fashion of Agile development does not equate to no planning. However –Many are mistakenly doing less or no planning –Emphasis on understanding and documenting requirements is casually allowed to erode User/customer physically close to the developers and are often paired with developers Lot of the requirements are verbal Changes to requirements are not closely controlled under the misnomer of “flexibility” What is your guess of the effect of this attitude to: - testing? (more or less testing? More or less effective testing?)) - quality? (more or less satisfied customer? More or less bugs?)