Testing Frameworks John Paul Ashenfelter CTO/Transitionpoint

What is a Framework? … a defined support structure in which another software project can be organized and developed. A framework may include support programs, code libraries, a scripting language, or other software to help develop and glue together the different components of a software project. From

What is Testing?  Verifying the software works correctly  Validating that the reliability and stability of the software  Ensuring that the software solves the problem it is supposed to solve  Making sure the user can use the software

Why Test? There is an old saying with software that three years from now, no one will remember if you shipped an awesome software release a few months late. What customers will still remember three years from now is if you shipped a software release that wasn’t ready a few months too soon. It takes multiple product releases to change people’s quality perception about one bad release. -Scott Guthrie

Who Should Do the Testing?  Good Testers Developers QA Staff Help Desk Internal Staff  Bad Testers Users Customers The public

When Should I Test?  When you: write code change code fix a bug integrate code deploy code think of a new edge case take a coffee break…

How Should I Test?  Code: Unit testing  Application: Functional testing  System: Load/performance testing  And if possible, user acceptance testing

Rolling Your Own Framework for Tests  Testing Tools CFCUnit Selenium Grinder  Support Tools SVN Trac Ant CruiseControl

Testing Your Code Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. -Brian Kernighan

Unit Testing In A Nutshell  A developer writes a unit test to verify an extremely specific area (unit) of functionality (code) behaves as they expect  One unit test is not sufficient! Good data Bad data Edge cases

Unit Test Code Example  We’ll use one of Sean’s GGCC CFCs ggcc3 to be specific  We’ll use CFCUnit for the testing

What’s Next for Unit Tests  Test cases become test suites  Test suites can be automated Ant Commit hooks in SCC (eg SVN) Continuous integration (eg CruiseControl)  Quality of code should improve

Testing the Application The trouble with programmers is that you can never tell what a programmer is doing until it's too late. -Seymour Cray

Functional Testing in a Nutshell  Functional tests verify the application works as expected from a user’s point of view  Written by developers and/or QA staff from test cases

Test Cases  Single step (or small series of steps) with single outcome precondition (known input) postcondition (expected output)  Need at least one test per requirement/use case

Functional Test Code Example  We’ll use Selenium  We’ll drive it automatically  Example is basic login process

Scenario testing  Based on a story  Any failure would motivate the fix  Is credible in real-world usage  Involves a non-trivial task  The results are easy to evaluate

Testing the System Premature optimization is the root of all evil in programming. -C. A. R. Hoare

Load Testing in a Nutshell  Measures the performance of the application with multiple concurrent “users”  Flavors of load testing: Load testing expected usage patterns Performance testing known workloads Stress testing to the breaking point

Testing Environment  Load testing is WYTIWYM (what you test is what you measure)  Results rarely (i.e. never) scale as expected to different hardware or software configurations  Can simulate multiple servers with VMWare with same caveats

User Testing Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning. -Rich Cook

Final Thoughts  There has never been an unexpectedly short debugging period in the history of computers. -Steven Levy

More Final Thoughts  Testing IS your job  Testing needs to be integrated into the entire development process  Tests build on one another into comprehensive suite(s)  Automated testing is something computers are really good at, so take advantage of that!

Resources The average software developer, for example, doesn't own a single book on the subject of his or her work, and hasn't ever read one. That fact is horrifying for anyone concerned about the quality of work in the field, for folks like us who write books, it is positively tragic. -Tom DeMarco and Timothy Lister, "Peopleware : Productive Projects and Teams, 2nd Ed."

Booklist  The Pragmatic Programmer, Hunt/Thomas  Code Complete, Steve McConnell  Agile Software Development, Robert Martin  Writing Effective Test Cases, Alistair Cockburn

Software  CFUnit  CFCUnit  Selenium  Grinder  Subversion  Trac  Ant  Eclipse  Find tools at

Questions/Thanks!!!  John Paul Ashenfelter  Transitionpoint Consulting Training Mentoring