1. Improving Testability Dave Catlett Test Architect, Microsoft davidcat@microsoft.com

2. Agenda What is testability? Why is testability important? Background: What is a test? Determining testability: SOCK Improving testability Q&A

3. What is testability? The degree to which components and systems are designed and implemented to make it easier for tests to achieve complete and repeatable code path coverage and simulate all usage situations in a cost efficient manner. In other words: How easy is it to test?

4. Why is testability important? Helps enforce good design principles. Reduces the time to diagnose unexpected behavior. Improves manageability and supportability. Provides the building blocks for self- diagnosing and self-correcting software. Speeds test development while increasing the depth and quality of automated tests.

5. Background: What is a test? Making sure the component works as expected and has the expected effect on the system as a whole in a repeatable and reliable way. Anatomy of a test case: SEARCH – Setup – Execution – Analysis – Reporting – Cleanup – Help

6. Determining testability: SOCK Simplicity – Reducing cost of testing at each phase of SEARCH – Design Observability – Exposing state (aka Visibility) Control – Exercising every nook & cranny as efficiently as possible. Knowledge of expected results – Is observed behavior is correct?

7. Improving Simplicity Make setup and configuration easy – Componentize – Allow multiple installs with varying configs on same machine Use good design principles – Strong cohesion – Loose coupling – Low redundancy

8. Improving Observability Expose component state – Magic numbers (Any value that controls code path execution.) – Resource usage – State transition information – API in/out parameters Tracing – Good tracing library – Observer or Visitor patterns can help maintain good design Be careful of exposing security info!

9. Improving Control Wrap interfaces you have no control over – Forces behavior you need to test your component – Facilitates fault injection tools Windows: Holodeck, AppVerifier Expose Magic Numbers to become configurable at runtime. Be mindful of security

10. Improving Knowledge of expected results Code contracts in source code Document expected behavior in all situations Document end-to-end user scenarios Future: Use modeling (machine readable specifications)

11. Q&A

12. Resources Design Patterns Explained: A new perspective on object-oriented design. Alan Shalloway, James Trott. Addison Wesley, 2002. – Observer pattern (tracing option for new classes) – Visitor pattern (tracing option for existing classes) – Adapter pattern (wrapping for fault injection) Refactoring: Improving the design of existing code. Martin Fowler, et al. Addison-Wesley, 1999. – Refactoring to improve design helps testability at the same time. www.netobjectives.com – Emergent Design, other materials Fault injection tools for Windows – Holodeck: http://www.sisecure.com fault injection tool.http://www.sisecure.com – AppVerifier: http://www.microsoft.com/technet/prodtechnol/winxppro/deploy/appvrfr.mspxhttp://www.microsoft.com/technet/prodtechnol/winxppro/deploy/appvrfr.mspx Google: CCCC cyclomatic complexity tool Brett Pettichord, Industry Testability Guru – Design for Testability: http://www.io.com/~wazmo/papers/design_for_testability_PNSQC.pdfhttp://www.io.com/~wazmo/papers/design_for_testability_PNSQC.pdf – QA-related papers: http://www.io.com/~wazmo/qahttp://www.io.com/~wazmo/qa – Other: http://www.pettichord.comhttp://www.pettichord.com