DAIMIHenrik Bærbak Christensen1 Testing Terminology

DAIMIHenrik Bærbak Christensen2 The terminology jungle Some effort has been invested in defining terms like error, defect, bug, failure, mistake, precisely. The apparent consequence is that everybody comes with their own precise definition...

DAIMIHenrik Bærbak Christensen3 IEEE Std

DAIMIHenrik Bærbak Christensen4 Burnstein Error (Da: fejl) –mistake, misconception, or misunderstanding by developer Fault (defect) (Da: defekt) –anomaly in software that may cause it to behave incorrectly, and not according to specification Failure (Da: svigt) –inability of software unit to perform its required functions within specified performance requirements

DAIMIHenrik Bærbak Christensen5 Bruegge Definitions used in dAPF course: –Failure [Svigt]: any deviation of the observed behavior from the specified behavior. –Error [fejltilstand]: system state where any further processing by the system will lead to an failure. –Fault (defect/bug): mechanical or algorithmic cause of an error.

DAIMIHenrik Bærbak Christensen6 The relations Failure = behavioural malfunction Defect = code anomaly Then –Defect  Failure ? No: –mention a class of defects that will not lead to failure Other way around: –Failure but no defects?

DAIMIHenrik Bærbak Christensen7 Testing = executing based Testing is an execution based process: Finding failures by executing software units –and hopefully identify the defect that caused it This is in contrast to inspection based processes like reviews. Definition: –Test case = set of input expected output execution conditions

DAIMIHenrik Bærbak Christensen8 Definition: Test Case Example: Testing System.abs A necessary part of a test case is a definition of the expected output or result. inputoutput parameter =

DAIMIHenrik Bærbak Christensen9 What about objects? Test cases seem to demand a functional paradigm ??? How would you define the test-case here? acc:Account balance == ? withdraw(150) ? inputoutput parameter = 150???

DAIMIHenrik Bærbak Christensen10 Objects Objects retain state over method calls. Some method calls mutate the object’s state. Thus object state is part of the test case input and output specification! Or: Considered part of the execution conditions inputoutput parameter = 150; acc.balance == 100acc.balance == -50 inputoutput parameter = 150acc.balance == -50 acc.balance == 100

DAIMIHenrik Bærbak Christensen11 Definitions Test –set of test cases + procedures to carry them out Test Oracle –artefact that allows testers to determine whether a test has passed or failed passed = no failure detected Test Bed –Execution environment that provides necessary hardware and software to test a unit

DAIMIHenrik Bærbak Christensen12 Testing Principles Burnstein 1) Testing = process with the intent of a) reveal defects b) evaluate quality 2) Good test case = high probability of revealing an undetected defect –Corollary: write as few tests as possible that covers as much code as possible –This is the basis of the equivalence class testing technique

DAIMIHenrik Bærbak Christensen13 Example Example: System.abs(int x) –If I have already tested that abs(37) == 37 then how much will I gain by also testing abs(39)? All integer values

DAIMIHenrik Bærbak Christensen14 Principles 3) Test results should be inspected meticulously –Corollary: do not use the production code algorithm to calculate the expected output 4) Test case must contain the expected output –it is not something the tester should guess about 5) Test cases should be developed for both valid and invalid input conditions –it is often the error handling code that is tested least...

DAIMIHenrik Bærbak Christensen15 Principles 6) The probability of additional defects in a component is proportional to the number already detected in it –due to high complexity or poor design 7) Testing should be independent of the development group –you often become blind to your code’s faults...

DAIMIHenrik Bærbak Christensen16 Principles 8) Tests must be repeatable and reusable –write them down clearly or even better automate them 9) Testing should be planned –make plans and objectives for each type of testing unit, integration, system 10) Testing activities should be integrated into the software life cycle 11) Testing is creative and challenging