Presentation is loading. Please wait.

Presentation is loading. Please wait.

IBM Software Group SQA Overview.

Similar presentations


Presentation on theme: "IBM Software Group SQA Overview."— Presentation transcript:

1 IBM Software Group SQA Overview

2 SQA / Release / Sysadmin Groups
Ed Barnes (Manager) SQA Chris Antonucci Pan Chan Victor Chen Melissa Schoonmaker Sid Misra Leigh Ishikawa Devon Brown Release David Teichholz Sysadmin Matt Valites

3 SQA Testing Overview Functional Testing Conformance (Autobuilder)
Stability (Overnight Tests) Performance First Article Tests

4 What are the goals for Conformance (Autobuilder) tests?
Testbase consisting of 20,000+ (XS/XI) and 50,000+ (XA) inputs and reference output files Tests for conformance of data processing, that output matches expected output for a given input Coverage grows as development and SQA add testcases over the development and test phases Passing all cases with exception of known failures is a requirement for the shipment of a release The test is run on the XA platform and the XS platform (includes XI) Conformance tests run continuously around the clock

5 What are the goals for overnight tests?
Overnight tests provide our core regression tests Passing all overnight tests is a requirement for release of a candidate Most tests fall under the category of “stability” tests, some are functional tests Most of the tests provide load on the box for an extended period (9-10 hours) Pass/Fail determination is typically based on a “signature” of: No backtraces Clean log messages Expected pattern of cpu, memory, connections usage, via SNMP Certain expected levels of errors Requires human review

6 Overnight Functional Tests
cli-harness Perl harness that tests execution of cli commands, validation of expected output snmpwalk SNMP tests, including walk and get of various management objects, validation of data using soma webgui harness TestComplete tool is used to test the configuration of the Datapower system through the webgui, and then verifies the configuration generated against a reference New customer-use case tests can be added using a Firefox recorder extension Javascript errors, delays, crashes/backtraces are captured

7 Overnight Stability Tests
ab-all-crypto Stability under load Runs apachebench requests across subset of testbase repeatedly pg-multicrypto Stability under load with multstep (encrypt, decrypt, sign, verify) Uses polygraph tool pg-xalan Stability under heavy compile load looping-xa Stability for proxy and coproc pg-schema-rewrite Stability under load for rewrite and document caching

8 Overnight Stability Tests, continued
thrasher Coproc stability under high concurrency Data Express/PoW Stability under load when connections are terminated unexpectedly mq-dpurl, mq-proxy,mq-gateway,mq-cibc-gw Stability under load with mq logging-load Stability under load with each of the logging target types pg-wamu Stability under heavy compile load with Washington Mutual (customer) test suite

9 What are the goals for Performance tests?
Verify functional performance of the system under typical loading conditions Verifies functional processing performance on selected operations, such as data transformation, SSL traffic, signing, verifying signatures, encrypting and decrypting payloads Performance should remain essentially the same or improve from release to release There is an expected improvement in performance for new hardware platforms based on the increase in processing power of the architecture/processor/cache Performance tests run continuously around-the-clock

10 What are the goals for First Article tests?
Verify the first system produced by the manufacturing process is operational from the customer perspective Verify that the system can be installed and configured Verify that the system can be upgraded to new releases, and can rollback to current release Verify that the system can perform a set of basic processing operations Verify that the correct version of firmware is installed by the manufacturing process Verify that shipped systems can be upgraded to the latest available firmware from the IBM/Datapower online support site

11 What about Functional Testing?
New features for releases use the feature-document process Feature documents cover the use-cases, requirements, implementation details, and testing information Serves as a foundation of the SQA functional test strategy Testing status and planning is tracked in the SQA checklist tool online

12 Requirements for a Release

13 What are the requirements for a release?
Conformance test results (Autobuilder/Autotest) must “pass” Passing means meeting the “high-water-mark” for the release branch (only having the known failures) Older releases have a number of failing cases (3.3.0, 3.4.0, 3.5.0) Latest release ( ) has a high-water mark of 0 failures. Overnight stability tests must all pass 8-10 hour runs for each No backtraces Review of logs Review of tcp connection status Review of memory, cpu usage throughout cycle In addition, specific test output unique to each test harness Performance tests Equivalent or better performance from release to release, unless specific performance target for a given release

14 What are the online resources for SQA?

15 What are online resources for SQA?
sqa.datapower.com Test results Release information Autobuilder results Autobuilder/AutoTest/Performance scheduling wiki.datapower.com General information for Dev, engineering departments, sysadmin build0.datapower.com build1.datapower.com Build images perforce.datapower.com Source code repository/viewer

16 SQA testing lab infrastructure

17 High-level lab infrastructure information
Units under test 9001, 9002, 9003 Broadcom + Cavium + HSM XG4 Backend servers Testservers, Appservers Test harness drivers Lab-pc DPAT execution Workstations WebGui Testing – standalone workstations (Windows) Network Infrastructure GBe flat L2 network across all of lab Separated performance network


Download ppt "IBM Software Group SQA Overview."

Similar presentations


Ads by Google