CSC 395 – Software Engineering Lecture 9: Testing -or- How I Stopped Worrying and Learned to Love the Bug

Today’s Lecture Discusses testing in early workflows Starts to answer important questions: When and how should we do testing? Who should be responsible for performing tests? How do we know something is correct? What faith should we place in testing?

What Do We Test For Obviously, need to test for correctness What is correct? Who decides? Important to consider other issues, too Utility - is it easy to use, cost effective, & useful? Reliability - how often will it not work? Robustness - do we rely on specific environment? Performance - is it within space & time constraints?

How Do We Do This Testing? There are two basic types of testing Execution-based testing Non-execution-based testing Testing performs two important actions Verification - Was workflow completed correctly? Validation - Does product satisfies requirements? Terminology Warning: “Verify” also used for non-execution-based testing

Testing Team Very useful to have independent testing team Experts on process of testing Insure developers & analysts deliver high-quality Be involved over entire life-cycle Critical to have managerial independence Bugs, faults, and errors are part of development Testing team must be free to make decisions Arguments with your Boss are hard to win Final decisions should not be made by developer or tester

Walkthroughs Walkthrough team ideally has people Member of team performing current workflow Member of team performing next workflow Member of SQA team Often includes a client Should NOT contain manager, partner, or boss Walkthroughs must be thoroughly prepared Provide items not understood Provide items that appear to be incorrect

Walkthrough Meeting Chaired by SQA member Only interest is in finding faults Practiced in running these meetings Meeting detects any possible faults Should not last longer than 2 hours “Committee is a group that keeps minutes and loses hours” -- Milton Berle Some flagged items may be correct Cost of meeting is high, but good if faults found “A camel is a horse designed by committee” -- Unknown

Running Walkthrough Document-driven & not participant-driven Focus on determining if documents are correct Relies on knowledge brought by participants Eases transfer of responsibilities between teams Faults often found describing documents Often discover problems by talking things through Many times bugs discovered

Inspections Follows five formal steps 1. Current workflow member provides overview 2. Participants prepare by developing understanding of document and look for faults 3. Inspection led by one participant through entire document. Written report completed with 1 day 4. Document goes through rework to fix faults 5. Moderator performs follow-up checking that all faults are fixed and documentation (including reports) are updated

Inspection Team Team consists of members, including: Moderator Member of team performing current workflow Member of team performing next workflow SQA team member One person will serve as reader Leads team through the document One person will serve as recorder Produces report of all possible faults team finds

Metrics Maintained During inspection, detailed metrics kept Is each fault a minor fault or a major fault Is the fault from not meeting the specification or that actual & formal arguments do not match? Where in the project did the fault arise? How many faults were found per document/LOC? How long did it take to find a fault?

Using Metrics Can compare between products Are we getting better at a workflow? Are the value correlated with something Take actions if artifact is overly buggy Suggests design may need rethinking Consider fault statistics from next workflow Some faults may not be detected until later Important to perform quality check on quality team

Testing Results JPL found savings of $25,000 per inspection Fault counts decreased exponentially by phase Important to maintain testing independence Benefits tied to faith in testing team Testing metrics should not be used as performance evaluation Need to consider metrics scientifically One or two results is not a sufficient sample Statistics can always be bent to tell a story

Inspections vs. Walkthroughs Walkthrough Two-step, informal process Inspection Five-step, formal process Both good for finding faults early in process But rely on having reliable, complete process Requires thorough documentation be available Lastly, needs all teams to buy-in to approach

Correctness Proofs Halfway between execution-based & non- execution-based testing Examines if algorithm/code matches specification But does so by developing mathematical proof Relies on creating & understanding proof Also assumes specification is correct

Correctness Proof Begins by converting code to flowchart

Correctness Proof Add I/O specification Loop invariants Assertions Developing these can be as much art as science

Three Myths Software Engineers stink at math Relies on basic algebra; even I passed that class Costs way, WAY, WAY too much Might be worth it in some situation; use sparingly Proofs are too darn hard Nontrivial products have been successfully proven Theorem provers can help automate process Are specification, compiler, & tools correct? Oops, this is real concern

For Next Lecture Wednesday’s lecture discusses execution testing Book provides good overview of how this is done, who should do the testing Additional reading presents JUnit testing tool Before next lecture Spend a few minutes writing a simple class and then playing with JUnit to test it