1. IBM Software Group
SQA Overview

2. SQA / Release / Sysadmin Groups
Ed Barnes (Manager)
SQA
Chris Antonucci
Pan Chan
Victor Chen
Melissa Schoonmaker
Sid Misra
Leigh Ishikawa
Devon Brown
Release
David Teichholz
Sysadmin
Matt Valites

3. SQA Testing Overview Functional Testing Conformance (Autobuilder)
Stability
(Overnight Tests)
Performance
First Article
Tests

4. What are the goals for Conformance (Autobuilder) tests?
Testbase consisting of 20,000+ (XS/XI) and 50,000+ (XA) inputs and reference output files
Tests for conformance of data processing, that output matches expected output for a given input
Coverage grows as development and SQA add testcases over the development and test phases
Passing all cases with exception of known failures is a requirement for the shipment of a release
The test is run on the XA platform and the XS platform (includes XI)
Conformance tests run continuously around the clock

5. What are the goals for overnight tests?
Overnight tests provide our core regression tests
Passing all overnight tests is a requirement for release of a candidate
Most tests fall under the category of “stability” tests, some are functional tests
Most of the tests provide load on the box for an extended period (9-10 hours)
Pass/Fail determination is typically based on a “signature” of:
No backtraces
Clean log messages
Expected pattern of cpu, memory, connections usage, via SNMP
Certain expected levels of errors
Requires human review

6. Overnight Functional Tests
cli-harness
Perl harness that tests execution of cli commands, validation of expected output
snmpwalk
SNMP tests, including walk and get of various management objects, validation of data using soma
webgui harness
TestComplete tool is used to test the configuration of the Datapower system through the webgui, and then verifies the configuration generated against a reference
New customer-use case tests can be added using a Firefox recorder extension
Javascript errors, delays, crashes/backtraces are captured

7. Overnight Stability Tests
ab-all-crypto
Stability under load
Runs apachebench requests across subset of testbase repeatedly
pg-multicrypto
Stability under load with multstep (encrypt, decrypt, sign, verify)
Uses polygraph tool
pg-xalan
Stability under heavy compile load
looping-xa
Stability for proxy and coproc
pg-schema-rewrite
Stability under load for rewrite and document caching

8. Overnight Stability Tests, continued
thrasher
Coproc stability under high concurrency
Data Express/PoW
Stability under load when connections are terminated unexpectedly
mq-dpurl, mq-proxy,mq-gateway,mq-cibc-gw
Stability under load with mq
logging-load
Stability under load with each of the logging target types
pg-wamu
Stability under heavy compile load with Washington Mutual (customer) test suite

9. What are the goals for Performance tests?
Verify functional performance of the system under typical loading conditions
Verifies functional processing performance on selected operations, such as data transformation, SSL traffic, signing, verifying signatures, encrypting and decrypting payloads
Performance should remain essentially the same or improve from release to release
There is an expected improvement in performance for new hardware platforms based on the increase in processing power of the architecture/processor/cache
Performance tests run continuously around-the-clock

10. What are the goals for First Article tests?
Verify the first system produced by the manufacturing process is operational from the customer perspective
Verify that the system can be installed and configured
Verify that the system can be upgraded to new releases, and can rollback to current release
Verify that the system can perform a set of basic processing operations
Verify that the correct version of firmware is installed by the manufacturing process
Verify that shipped systems can be upgraded to the latest available firmware from the IBM/Datapower online support site

11. What about Functional Testing?
New features for releases use the feature-document process
Feature documents cover the use-cases, requirements, implementation details, and testing information
Serves as a foundation of the SQA functional test strategy
Testing status and planning is tracked in the SQA checklist tool online

12. Requirements for a Release

13. What are the requirements for a release?
Conformance test results (Autobuilder/Autotest) must “pass”
Passing means meeting the “high-water-mark” for the release branch (only having the known failures)
Older releases have a number of failing cases (3.3.0, 3.4.0, 3.5.0)
Latest release ( ) has a high-water mark of 0 failures.
Overnight stability tests must all pass
8-10 hour runs for each
No backtraces
Review of logs
Review of tcp connection status
Review of memory, cpu usage throughout cycle
In addition, specific test output unique to each test harness
Performance tests
Equivalent or better performance from release to release, unless specific performance target for a given release

14. What are the online resources for SQA?

15. What are online resources for SQA?
sqa.datapower.com
Test results
Release information
Autobuilder results
Autobuilder/AutoTest/Performance scheduling
wiki.datapower.com
General information for Dev, engineering departments, sysadmin
build0.datapower.com
build1.datapower.com
Build images
perforce.datapower.com
Source code repository/viewer

16. SQA testing lab infrastructure

17. High-level lab infrastructure information
Units under test
9001, 9002, 9003
Broadcom + Cavium + HSM
XG4
Backend servers
Testservers, Appservers
Test harness drivers
Lab-pc
DPAT execution
Workstations
WebGui Testing – standalone workstations (Windows)
Network Infrastructure
GBe flat L2 network across all of lab
Separated performance network