1 Steve Chenoweth Tuesday, 10/04/11 Week 5, Day 2 Right – Typical tool for reading out error codes logged by your car’s computer, to help analyze its problems. From CSSE 377 – Intro to Testability

2 Today How to do Project 4… –We’ll get your “implementers” a bit more involved this time Software testability – this  –Bass’s Ch 4, pp (Testability scenarios) & Ch 5, pp (Testability tactics) –Lots of discussion also in CSSE 376! Q 1

3 Coming up… Thursday – (Steve’s at Purdue) –Guest speakers – Matt Ellis, Justin Hutchings of Microsoft –Remaining time – if any – Work on first turnin for Testability Friday – More on this & time to work on Project 4 – including… –Getting your “implementers” involved – first talk with them about “what should be more testable” on your system, then have them try to run tests themselves. Monday: Analyzing archs & the ATAM / CBAM

4 We next pick testability from Bass’s QA list… Bass’s list of six, from the inside back cover of his book: –Availability –Modifiability –Performance –Security –Testability –Usability

5 And here’s the project about it: Overall - On the same project, Use an arch tactic to make it more testable: –Determine what is not very testable in your current project system. –Implement a tactic to improve this by a predicted amount! –Have your implementers observe the “before” and “after” testing, and confirm that there’s an improvement We’re also going to continue our study of how to document the arch, so: –An added feature of this project – add one more section to your draft of that, for your sys.

6 And a first step to take: Try to think of a problem area in testing your current system. Run through this area of tests, of your system as it exists now. Time how long this takes, and find a way to measure how “effective” (thorough) it is. See if you can get your “implementers” to repeat the same tests. Make a “prediction” about how fast or effective this testing will be with the testing change you plan to make. Turn in your ideas, those “existing times,” and other measures, with your prediction, in your “team journal” by Friday at 11:55 PM. Next step will then be – Make the changes to testing that will speed this up or make it more effective.

7 What’s Bass say about this QA? Problem – Testing needs to be more efficient & effective –It consumes a high percentage of software development time & effort –The heuristic is “half” Goal – Allow easier testing when an increment of software development is completed. Motivation – The overall arch of the system, and of its testing methods, can help make testing easier Scenarios – What’s in “The Notes” at the end of the supplementary spec template? supplementary spec template What is Testability “about” – Ch 4? What are some good tactics – Ch 5? Q 2

8 Bass’s testability scenarios Source: Unit developer –Increment integrator –System verifier –Client acceptance tester –System user Stimulus: Analysis, architecture, design, class, subsystem integration completed; system delivered Artifact: Piece of design, piece of code, complete application Environment: At design time, at development time, at compile time, at deployment time Response: Provides access to state values; provides computed values; prepares test environment Response Measures: (examples) –Percent executable statements executed –Probability of failure if fault exists –Time to perform tests –Length of longest dependency chain in a test –Length of time to prepare test environment Q 3

9 Example scenario Source: Unit tester Stimulus: Performs unit test Artifact: Component of the system Environment: At the completion of the component Response: Component has interface for controlling behavior, and output of the component is observable Response Measure: Path coverage of 85% is achieved within 3 hours

10 Testability situations It’s a development-time attribute: Probability of fault discovery Need to control components Need to observe component failure Ch 4 Right: Ren & Stimpy tell why people ignore system testability. From web site sctest.cse.ucsc.edu/.sctest.cse.ucsc.edu/

11 Tactics to achieve testability Manage Input/Output –Record/playback –Separate interface from implementation –Specialize access Internal monitoring –Built-in monitors Generally, the goal is to automate as much testing as you can – why? Ch 5 Q 4

12 Examples - 1 Test harness – like the stimulator you might have done for performance –Automates some aspect of testing –But, for finding errors, the test cases need to be more complete (unless performance / reliability are what you’re testing!) –Can be for internal classes, etc., as well as external – See next slide  –Give this itself an “architecture” with options, different modes, etc. Q 5

13 Examples – 1, cntd Typical test harness: In MVC, replace the “View” with a stimulator that fires test cases at the rest of the system: Replace with stimulator, for testing

14 Examples – 1, cntd How about stimulating the GUI? A slightly harder trick, replacing the human with the mouse and keyboard. How do we test the GUI itself? (Or the whole System!?) See list of open source tools at

15 Examples - 2 Built-in monitors – the system itself is “instrumented” to provide test results. –Usually, this can be turned on/off so it doesn’t interfere with normal operations. –May have multiple special modes. –The “heart” of white-box testing!

16 Industrial strength tools Good example – IBM’s Rational PurifyPlus –Shows code coverage –Analyzes memory problems (e.g., buffer overflow, leaks) –Identifies performance issues like bottlenecks –Runs on Linux, Unix, Windows See &cm=k&csr=google&cr=rational_purify&ct=AGRAK605&ck=ration al_purify&mkwid=s28L9ro6h_ _432dt32930&cmp=109HGhttp://www-01.ibm.com/software/awdtools/purify/?cn=agus_rtnlrp &cm=k&csr=google&cr=rational_purify&ct=AGRAK605&ck=ration al_purify&mkwid=s28L9ro6h_ _432dt32930&cmp=109HG

17 Project 4 – Testability (overall) Overall theme (of the testability part of the project) – Pick an area where you want to improve testing. Pick a strategy to improve it, using one of the above tactics, and verify (Friday) with your implementers. The improvement can be in either of these two areas: –Effectiveness (thoroughness) of the testing, or –Speed of the testing, with the same effectiveness. Try it “as-is” to verify how long it takes and the coverage. –Includes letting novice testers, your “implementers,” try it. Guess how much you can improve that with your change. Make the change to improve testability. Run the testing again, with the change. Measure how fast and how effective the new testing is. Let your implementers verify the differences. Report on the results.