1. Automated Software Testing using Open Source Testing Tools
By Elfriede Dustin

2. Agenda Who is IDT State of Software Testing
What is Automated Test and Re-test (ATRT)
Approach to ATRT
Advantage of open source
ATRT and open source – Our IDE and solution
ATRT Challenges

3. Who is IDT?
IDT specializes in the design, development, and implementation of Automated Software Testing and Re-Test (ATRT) and Quality Assurance (QA) solutions
Deliver turn key automated test suite
Automated test strategy and identification of highest payoff areas to apply automation for your project
Selection of best automation tools for your project
Using your existing test cases or develop test cases for you
Test results documented in requirements traceability matrix
Training and pilot project implementation with companies

4. Who is IDT?
IDT specializes in the design, development, and implementation of Automated Software Testing and Re-Test (ATRT) and Quality Assurance (QA) solutions
Deliver turn key automated test suite
Automated test strategy and identification of highest payoff areas to apply automation for your project
Selection of best automation tools for your project
Using your existing test cases or develop test cases for you
Test results documented in requirements traceability matrix
Training and pilot project implementation with companies
We are currently hiring Java Developers
Plus we’d like to hire 2 or 3 grad students to work for us part time ( i.e. 20 hours week )
must be US citizen 4

5. State of Software Testing - Why ATRT?5

6. State of Software Testing - Why ATRT?
50% or More of Overall Development Cost is Typically Spent on Testing
Utilize Automated Testing Strategies and Technology to Improve Productivity and Quality
Spiral / Release 4
Reduction in Test Days
Spiral / Release 3
Increased Test Coverage
Size & Complexity of SW Baseline
Test Days
Spiral / Release 2
Manual
Test Days
Spiral / Release 1
Automated
Test Days
Manual vs. Automated
Test Days
IDT Provides Automated Software Testing Solutions 6

7. What is ATRT?
Our definition of ATRT is:
Application and implementation of software technology throughout the entire Software Testing Life Cycle (STL) and QA lifecycle; with the goal to improve STL efficiencies and effectiveness

8. ATRT Spans the Software Development Life Cycle – Visual Modeling
Design
Build
Assemble
Round-Trip Engineering
Development Tools
Components
Test
Use cases
Modeling and test case generation

9. ATRT Spans the Software Development Life Cycle – Requirements
Requirements Management and
Process Automation
Visual Modeling
Development Tools
Components
Organizes, tracks, & controls requirements
Requirements Traceability Matrix (RTM)

10. ATRT Spans the Software Development Life Cycle – Automated Test Tools
Requirements Management and
Process Automation
Visual Modeling
Development Tools
Components
Development
Management
Execution
Automated Software Testing
Automated Testing Tools
Automates test cases using vendor-provided,
open-source tools or in-house development

11. Requirements Management and
ATRT Spans the Software Development Life Cycle – Integrated Suite of Tools
Requirements Management and
Process Automation
Additionally:
Middleware
Infrastructure
Defect Tracking
Configuration Management
Memory Leak Detectors
Performance Testing Tools
Documentation Tools
others
Visual Modeling
Infrastructure
Middleware
Automated Testing Tools
Development Tools
Components
Software Configuration Management
Defect Tracking

12. Types of Software Test Life-cycle Support Tools
Which tools are you familiar with?

13. Types of Automated Test Tools
Life-Cycle
Phase
Type of Tool
Tool Description
Business
Analysis
Business Modeling
Records definitions of user needs and automates rapid construction of
flexible, graphical, client-server applications
Configuration
Management
Baselines important data repositories
Defect Tracking
Manages system life-cycle defects
Technical Review
Facilitates communication, while automating the technical review/inspection process
Documentation
Generators
Automate document generation
Requirements
Definition
Manages and organizes requirements; allows for test procedure design and test progress reporting
Verifiers
Verify syntax, semantics, and testability
Use Case
Create use cases

14. Types of Tools (Cont’d)
Life-Cycle
Phase
Type of Tool
Tool Description
Analysis and Design Phase
Database Design
Develops second generation enterprise client-server systems
Structure Charts, Flowcharts, and Sequence Diagrams
Manage processes
Test Procedure Generators
Generate test procedures from requirements, design, or data and object models
Syntax Checkers/ Debuggers
Perform syntax checking and have debugging capability; usually available with built-in programming language compiler
Programming Phase
Memory Leak and Runtime Error Detection
Detects runtime errors and memory leaks
Source Code Testing
Verifies maintainability, portability, complexity, and standards compliance
Static and Dynamic Analyzers
Depict quality and structure of code
Metrics Tools
Code (Test) Coverage Analyzers or Code Instrumentors
Identify untested code and support dynamic testing
Usability Measurements
Provide usability testing as conducted in usability labs

15. Types of Tools (Cont’d)
Life-Cycle
Phase
Type of Tool
Tool Description
Other Testing
Support Tools
Data Extraction Tool
Extract Data from various formats into various formats
Test Data Generators
Generate test data
File Compare Utilities
Find discrepancies between files that should be identical in content
Simulation
Simulates application to measure for scalability, among other tasks
Test Management
Tests management
Network Testing
Monitors, measures, tests, and diagnoses performance across the entire network
GUI Testing (Capture/Playback)
Conducts automated GUI tests; capture/playback tools record user interactions with online systems so they may be replayed automatically
Load/Performance Testing
Conducts load/performance and stress testing
Security Testing
Performs security testing and vulnerability scanning at the application or network level; plus debuggers will allow to check for security coding errors (source code checkers)

16. Deployment Approach – Adapted ATLM
Populate Test Manager with Test Cases and Requirements
Develop Test Scripts from Test Procedures
Test Results:
Pass/Fail by Test Case
Pass/Fail Summary
Test Time Required
Identify or Develop
& Pilot
Test Manager
Test Scripts
Data Comparators
Performance Test Tools
GUI Record/Playback
Storing/Retrieving Results
Collect/Create:
Requirements
Test Cases
Test Procedures
Expected Results
Interface Specifications
System/Component
Configuration Description

17. Why Open Source ?

18. Why Open Source? Advantage of Open Source:
no licensing issues; i.e. license cost; maintenance; etc.
easily modifiable
adaptable
lightweight
flexibility - not tied to one vendor

19. ATRTs IDE - Eclipse Eclipse – this site lists all the plugins
Eclipse/TPTP – Open Source - Developed by IBM
Eclipse/Test and Performance Tools Platform (TPTP) offers a common extensible framework for the following functions:
Requirements Management
Configuration Management - Subversion
Testing Tools (
Profiling: default Java applications, but can be extended to other apps 19

20. ATRTs IDE - Eclipse
Eclipse/Test and Performance Tools Platform (TPTP) offers a common extensible framework for the following functions (cont):
Static Analysis: default for C++ and Java, but can be extended to other apps
Application monitoring and log analysis
Build tool – ant
Xml input and output - using an open source Java package , which is a Java library that has an API for constructing and outputting XML. 
Test Manager – STAF/STAX 20

21. Current Test Tool Implementation – Example and Challenges
Vendor-provided (Capture/Playback) Tool:
Automated test tools mimic actions of the test engineer.
During testing, the engineer uses the keyboard and mouse to perform some type of test or action.
Testing tool captures all keystrokes and subsequent results, which are baselined in an automated test script.
During test playback, scripts compare latest outputs with previous baseline.
Testing tools have built-in, reusable test functions.
Most test tools provide for non-intrusive testing; i.e., they interact with the “application-under-test” as if the test tool was not involved.

22. Current Testing Tools – Challenges (cont)
In-House Software Development required:
Capture/Playback tools generate hard-coded values; test scripts are not reusable, nor do they implement software development best practices right out of the box; scripts need to be modified.
Capture/Playback tools don’t necessarily provide all testing features required; code enhancements are often required to meet testing needs.
Capture/Playback tools are not necessarily compatible with system engineering environment, and software testing scripts need to be developed in-house.
Developed testing scripts are not cross-vendor compatible, even if the same scripting language is used

23. Linux Capture/Playback Tools IDT Findings
Replay Xcessory
QF-Test
Froglogic
Redstone Software
Eggplant
KD Executor
Jameleon
ShUnit2
for shell scripts testing
GNU Linux Desktop Project
Cantana
crontab
Part of Unix
crontab, cruisecontrol,
Dogtail
Expect

24. Automated Test and Re-Test (ATRT)
ATRT includes a common framework comprised of re-usable software services
Open Source Software Test Automation Framework (STAF)
Provides automated test procedures, test inputs, and test results for SW components
Suitable for use by 3rd parties not associated with development of the component

25. What is STAX? STAX is an automation system
With STAX you can automate and monitor your entire test environment
System/product setup
Testcase distribution
Testcase execution
Testcase results collection and analysis
STAX consists of
A programming language designed for automation
Programs written in this language are called jobs
An execution engine (similar to an interpreter for other programming/scripting languages) which manages the execution and runtime behavior of the job
A GUI application, called the STAX Monitor, which provides a dynamically updated view of your executing jobs
A tool (STAXDoc) that is used to generate documentation for your STAX jobs

26. STAF can run on:
    * Windows 95     * Windows 98 (and 98 SE)     * Windows Millenium Edition     * Windows NT Version 4.0     * Windows 2000     * Windows XP     * Windows Server 2003 (IA32, IA64, AMD64)     * Windows Vista (IA32, AMD64)     * Linux (Intel32/64, AMD64, PPC32/64)     * Linux on zSeries (31-bit, 64-bit)     * AIX Version or higher (32-bit, 64-bit)     * Solaris (Sparc) 2.6 and higher     * Solaris AMD Opteron 64-bit     * HP-UX and higher (PA-RISC, IA64 32-bit and 64-bit)     * OS/400 V5R2 or higher     * z/OS UNIX V1.4 and higher

27. STAF Internal Services
DIAG Provides diagnostics services Internal ("DIAG")
DELAY Provides a means to sleep a specified amount of time Internal ("DELAY")
ECHO Echos back a supplied message Internal ("ECHO")
FILE SYSTEM Allows you to get and copy files across the network Internal ("FS")
HANDLE Provides information about existing STAF handles Internal ("HANDLE")
HELP Provides Help on STAF error codes Internal ("HELP")
MISC Handles miscellaneous commands such as displaying the version of STAF
that is currently running Internal ("MISC")
PING Provides a simple is-alive message Internal ("PING")
PROCESS Allows you to start, stop, and query processes Internal ("PROCESS")
QUEUE Provides a network-enabled IPC mechanism for STAF Programs Internal ("QUEUE")
SEMAPHORE Provides network-enabled named event and mutex semaphores Internal ("SEM")
SERVICE Allows you to list services available on a machine and to examine the
Requests that have been submitted on a machineInternal ("SERVICE")
SHUTDOWN Provides a means to shutdown STAF and register for shutdown
Notifications Internal ("SHUTDOWN")
TRACE Provides tracing information for STAF services Internal ("TRACE")
TRUST Interfaces with STAF's security Internal ("TRUST")
VARIABLEProvides a method for maintaining configuration and runtime data
(variables) Internal ("VAR")           

28. STAF External Services
The executable code for external STAF services resides outside of STAFProc, for example in a Java jar file, a C++ DLL file, or a Rexx script file.
CRON Calls into STAF services at a specified time interval External (Java)
Allows you to send messages External (Java)
EVENT Provides a publish/subscribe notification system External (Java)
EVENTMANAGER Allows you to call STAF services when a specified Event occurs External (Java)
HTTP Allows you to make HTTP requests which can be grouped together in a session External (Java)
LOG Provides a full-featured logging facility External (C++)
MONITOR Allows a testcase to publish its current running execution status for others to read External (C++)
RESOURCE POOL Allows you to manage exclusive access to pools of elements, e.g. VM UserIDs or Software Licenses External (C++)
STAX Provides an XML-based execution engine External (Java)
ZIP Provides a means to zip/unzip/list/delete PKZip/WinZip compatible archives External (C++)

29. Technologies used in STAX
STAF
STAF provides the infrastructure on which STAX builds. The full power of STAF and its services is exposed for use within STAX jobs.
XML
The STAX programming language is based on XML. This provides built-in structure to your jobs, as well as providing a set of existing tools for use in constructing your jobs, such as XML (aware) editors and XSLT.
Python
The STAX programming language builds on Python to provide a rich and accessible data model
STAX's Python integration also allows you to access the wealth of existing Python libraries
Java
The STAX programming language allows you to access existing Java classes/libraries, providing another source for reuse

30. Automation Tasks Automation Startup System Setup Testcase Execution
Monitoring
Synchronization
Testcase Output Analysis
Resource Management
Testcase Cleanup
Results Notification
Automation Completion

31. End-to-End Automation with STAF and STAX
Automation Startup S T A X J o b
Event, EventManager, Cron
<function>
<parallel>
<sequence>
<process>
<stafcmd>
<testcase>
<timer>
<block>
<loop>
<message>
<log>
<import>
<job>
<iterate>
<paralleliterate>
<hold>
<release>
<terminate>
System Setup
FS, Process
Testcase Execution
Process, Monitor, Log, Variable,
Queue, ResPool
Testcase Output Analysis
Log, Process
Testcase Cleanup
Process
Results Notification
, HTTP
Automation Completion

32. Key ATRT Architecture Requirements
Support applications running on multiple computers
Support applications developed in different languages
Support applications running on different types of OS’s
Support applications which have GUI and those which do not (for example Interface testing)
Support applications which use different types of network protocols such as TCP/IP, DDS, etc
Support integration of multiple commercial testing tools from different vendors ( allowing as new or better products emerge in the market they can be utilized )
Support testing w/o having to install ATRT on the same computers as the application under test and be able to be distributed across computers

33. Prototype – Proof of Concept
Analysis & Display
Application
Display Application
Display Application
System Under Test
Windows
Based
Simulation of Application
Processing Messages
Eggplant
R/P
VNC Robot
R/P
Simulation of Application
Processing Messages
ATRT
Test Manager
RTM
Bugzilla
Linux Based
STAF/STAX 33

34. ATRT Prototyping Effort
Able to successfully execute ATRT for 48 hours without operator intervention
During the tests:
VNCRobot - Approximately 25,000 operator key strokes were replayed
STAF/STAX - More than 1,000,000 messages processed and verified by ATRT

35. Additional Requirement
Pristine test environment required – automated testing tool cannot be installed on Application-Under-Test PC
Eggplant – allows for remote capture/playback via VNCserver and KVM switch
VNCRobot

36. Automated Software Testing
Software testing paradox
What is it?

37. Effective Automated Software Testing
Current State:
Developing software in order to test software
Automate the test support development: automated IDL to code generation; test data generation (using FireEye from NIST); compare utilities; log file analyzers;
Our Automated Software Testing Goal:
Minimize Human interaction
Possibly get on par with Hardware Automated Testing?

38. Hardware Testing: current state
Many automated test generators for manufacturing/ logic devices, circuit testing, etc., available i.e. SEI/ATG, uMaster, etc.
Automated testing without human interaction is the norm for hardware testing

39. Hardware vs. Software AT
How is hardware testing different?
Software Challenge:
Changing requirements
GUI interfaces
Code access/availability
Defects uncovered earlier in the software development lifecycle are cheaper to fix
Solution flexibility required – problem dependent

40. Proposed solutions Focus on Reusable Components
Test Data / Expected Results
Interfaces
Middleware
Assuming code and models are available:
Model based testing; automated test case generators based on models
Eclipse – effective ATRT IDE
Probes – see Aprobe – Code instrumentation
Building self-testable components
Using standard language MOF to Text to generate automate test cases
GUI Testing tool ideas
New tools - other
Working with NIST to produce additional tools

41. GUI Testing Jemmy - java library to test UI http://jemmy.netbeans.org/
Perl modules
run via VNCServer?

42. Questions ?
with any follow-up questions

43. IDT We are hiring……. Please come see me now if interested or send
with any follow-up 43