1. Assessing Requirements Quality through Requirements Coverage Ajitha RajanUniversity of Minnesota Mats HeimdahlUniversity of Minnesota Kurt WoodhamL3 Communications

2. 2 http://www.umsec.umn.edu Model-Based Development Specification Model Visualization Prototyping Testing Code Analysis Properties

3. 3 http://www.umsec.umn.edu Model Validation problem 1.Are the requirements sufficiently defined for the system? 2.How well does the model implement the behaviors specified by the requirements? We propose a testing approach that explores the relationship between requirements-based structural coverage and model-based structural coverage

4. 4 http://www.umsec.umn.edu Are the Requirements Sufficient? Run tests on the model and measure coverage achieved Informal Requirements Model / Design Specification (MUT) Requirements-Based Tests Generate to provide requirements coverage Poor coverage of model implies one or more of the following 1.Missing or implicit requirements 2.Behavior in the model not derived from requirements 3.Requirements-based tests are inadequate Measure

5. 5 http://www.umsec.umn.edu Model Implements Requirements? Run tests on the requirements and measure coverage achieved Model/Design Specification Requirements Model-Based Tests Generate to provide model coverage Measure Poor coverage of requirements implies one or both of the following 1.Model does not adequately implement behaviors specified in the requirements 2.Model is correct and requirements are poorly written

6. 6 http://www.umsec.umn.edu Experimental Setup Requirements Coverage Metric – Unique First Cause (UFC) coverage defined over formal LTL requirements Model Coverage Metric – Modified Condition/Decision Coverage (MC/DC) Four industrial case examples from the avionics domain Two kinds of assessments  Generate test suites to provide requirements UFC coverage and measure MC/DC over the model  Generate test suites to provide MC/DC over the model and measure UFC achieved over formal requirements.

7. 7 http://www.umsec.umn.edu Experiment Results MC/DC Achieved by Requirements-Based Tests

8. 8 http://www.umsec.umn.edu Experiment Results MC/DC Achieved by Requirements-Based Tests Two of the four case examples  UFC test suites provide good MC/DC of the model but 10 – 20% less than achievable Remaining two case examples  UFC test suites achieved poor MC/DC on the model  Many missing requirements on one example  On the other example, structure of the requirements “cheated” the UFC metric

9. 9 http://www.umsec.umn.edu Experiment Results UFC metric “cheated” by the structure of requirements LTLSPEC G( var_a > ( case foo : 0 ; bar: 1 ; esac+ case baz : 2 ; bpr : 3 ; esac )); LTLSPEC G( var_a > ( case foo & baz : 0 + 2 ; foo & bpr : 0 + 3 ; bar & baz : 1 + 2 ; bar & bpr : 1 + 3 ; esac ));

10. 10 http://www.umsec.umn.edu Experiment Results Requirements UFC Coverage Achieved by Model-Based Tests

11. 11 http://www.umsec.umn.edu Experimental Results Requirements UFC Coverage Achieved by Model- Based Tests On all except one case example MC/DC test suites achieved close to maximum possible requirements UFC coverage On one example, the MC/DC test suites did poorly (30% of maximum possible UFC)  Model does not implement all the behaviors in the requirements  MC/DC metric is not rigorous enough

12. 12 http://www.umsec.umn.edu Conclusions Analyzing requirements coverage Vs model coverage provides a promising means of assessing requirements quality. Effectiveness is dependent on the rigor and robustness of the coverage metrics used  UFC sensitive to the structure of the requirements