1. Sonam Chauhan Corporate Express sonam.chauhan@ce.com.au webMethods Testing at Corporate Express

2. Overview Introduction Change Process General Requirements for Testing Old Testing Process New Testing Process Implementation Tips Future Steps and Trends References

3. Introduction Constant pressure for change Changes need to be tested for bugs Testing has business impact –Good testing costs money –Bad testing costs more money! Need to find the Golden Mean

4. Introduction webMethods issues that relate to testing Dev/Test/UAT/Live Builds Much Variation »Due to ISs multiple packages feature Source Code Management Rudimentary »Support planned for interfacing to SCMs Complex Change Deployments Risky »Partially addressed by wMDeployer Testing Support Absent »wM arent adding it

5. Change Process Requirements Definition Requirements Specification Development Review and Testing Change Deployment (Dev Test UAT Live) Changes Validated Changes

6. General Requirements for Testing Like development, testing is an engineering skill –Allocate limited resources for max. benefit. Black-box Integration testing for Regressions Best return for your testing buck –High level tests simulate live use –Tests can be external to code Easy to retrofit to existing codebase –Regression are the costliest bugs All changes must first do no harm –This is not Unit Testing Unit tests cant assure system functionality This is integration/system-level testing

7. General Requirements for Testing Developers can be testers! –Commitment to quality Make test development part of development process Make reviews of the test code part of code review process –Automate testing Automated Regression Testing Harness –Without automation, testing is tedious work Automation increases productivity and efficiency –Permanently add hard-won insight to corporate memory And replay it on demand –Essential for managing change in complex systems –Active testing catches problems early Load testing often very important –Some bugs can only be "teased out" under load.

8. Old Testing Process Implemented Circa 2001 Mostly manual Repository of test cases Test case input submitted to system being tested System output compared to stored test case outputs and Diff generated (image at right) Diff manually inspected and signed off

9. Old Testing Process Advantages –Thorough Inspection of Changes A human eyeballs all differences Disadvantages –Expensive Due to tedium, testing is only done when required –Not Scalable Testing time increases significantly as platform grows more complex –Limited Coverage No load testing Fixed issues are not re-tested –Errors Quality of QA suffers as most testing is manual and tedious

10. New Testing Process The metatest Testing Architecture Implemented Circa 2004 Drivers: –Increasing Complexity –Need for Automation metatest Testing Architecture –Perl command line engine –Open source components –Controls test engines that execute parameterized test scripts –Supports multiple test engines, test parallelization (in addition to threading within tests), test retargeting and test load variation –Test scripts execute sequence of steps, with assertions made at end of each step. –Entire test runtime and test script database in SCM system (CVS) Effort: 1 man-months (excluding test scripts) with one developer (me) Enhanced over a year

11. New Testing Process Sample Test Script – JMeter Test Engine Load testing part of same test script Documents retrieved from TN database to test document transform assertions JMeter [4] test scripts are executed by JMeter instances controlled by metatest

12. New Testing Process Advantages –Cheap Open-source software, no new hardware cost Completely test automation - no tedium –Scalable No time increase with platform complexity - tests run in parallel –Coverage Includes load testing Can run selective tests, or run regression test an entire system in 5 minutes –No Human Errors Quality of QA improves –New Benefits Detailed test results, Test Driven Development, Continual testing Disadvantages –Typically, only specific assertions are automatically proven All differences not examined – this may be mitigated by using full- document assertions regular expressions adjusting for variances

13. Automatic archival of detailed results Data useful for: –Load and stress performance validation (eg: image on the right) [5] –Capacity planning –Trend Detection –Analysing software component failure rates New analytical methods can be applied to stored data in future New Testing Process New Benefit – Detailed Test Results

14. While not Unit Testing, our approach has several of the benefits of classical TDD: –Tests and code concurrently developed –Bugs replicated as tests and fixed –Bugfix tests added to test database –Simple tests –Brief run time for tests –Immediate testing feedback New Testing Process New Benefit –Test Driven Development

15. Continual Testing –Dev/Test/UAT tested nightly –Test & UAT also load tested (Load imposed = 4 x production load) –Results emailed to developer list so they see it first thing in the morning (see image on right) Benefits –All platforms fully regression and load tested each night –Differential testing – identical tests run on different systems each night. When the results differ, bugs are identified by comparing the two systems. In example at right, failures are only on test in load testing… –The load imposed in load testing is updated to reflect growth. Thus, bottlenecks are identified before they hit production. New Testing Process New Benefit – Continual Testing

16. Documentation very important –Documentation crucial to getting developer buy-in –metatest architecture has 35 pages of user documentation (in addition to documentation for the tests themselves) When using open source components, expect bugs –Interact with developers of the component –Contribute if possible –Document problems, link to Bugzilla DBs etc Version control everything to do with testing –Test scripts –Even test engines binaries –Advantages: Branching/ Merging, new developer can be setup in minutes Leverage existing infrastructure –Most software can simulate endpoints. For eg: WM C1 OnRamp can create a CommerceOne envelope from XML Get a domain expert to write your tests Implementation Tips

17. Future Steps and Trends Plans to improve the metatest architecture –Implementing other types of testing –Integration with WmDeployer –Integration with CECM Verification Tool [6] Other Testing Resources in the WM space –Customwares WmUnit [1] –Wmusers.com article on testing using JUnit [2] –Solstice Software [3]

18. Future Steps and Trends webMethods Testing Support –WM response to group request for testing support:Currently no plans exist add unit testing or test service generation services to the developer. –However, WMs IDE is moving to the Eclipse platform. This should open up interesting possibilities with Eclipse plug-ins. –Server support: Ideally, on throwing an particular exception, a WM server would save the pipeline in a testcase format for bug replication on development server. (WM tool does this already?)

19. References [1] Customware website: http://www.customware.net/http://www.customware.net/ [2] wmUsers.com testing article: http://www.wmusers.com/ezine/2003feb_parvanitis_1.shtml http://www.wmusers.com/ezine/2003feb_parvanitis_1.shtml [3] Solstice software website: http://www.solsticesoftware.com/http://www.solsticesoftware.com/ [4] JMeter website: http://jakarta.apache.org/jmeterhttp://jakarta.apache.org/jmeter [5] Article on analysis of JMeter logfiles in stress testing: http://wiki.apache.org/jakarta-jmeter/LogAnalysis [6] CECM change verification tool described in: http://www.wmusers.com/software/uploads/2820B2B_Change_ Management_Presentation.ppt