1. Test Automation Success: Choosing the Right People & Process
Kiran Pyneni, Automation Manager Aetna, Inc.

2. Agenda Company Background Automation Team Automation Objectives
Our Journey…
Strategy Team
Delivery Team
Benefits of Teams Strategy
Automation Lifecycle
Benefits of Process Integration
Metrics Of Our Success
Automation Coverage, Efforts Automated, Utilization & Cost Avoidance
Questions

3. Aetna Background
Aetna is one of the nation's leading diversified health care benefits companies, serving members with information and resources to help them make better informed decisions about their health care.
Products and management Programs:
Aetna offers a broad range of insurance and employee benefits products.
The first national, full-service health insurer to offer a consumer-directed health plan, Aetna continues to lead the way with its Aetna HealthFund2 line of products, including HSA, HRA and RRA options.
Aetna offers a wide array of programs and services that help control rising employee benefits costs while striving to improve the quality of health care, such as case management; disease and patient safety programs; integrated medical, dental, pharmaceutical, behavioral health and disability information.
Aetna provides members with access to convenient tools and easy-to-understand information that can help them make better-informed decisions about their health and financial wellbeing. 3

4. Automation Team
Improves efficiencies and quality at our organization by automating consistent and repeatable manual tasks using industry standard technologies / tools / frameworks.
Standards &
Best Practices
Automation
Team
Reports & Metrics
Assessments &
Designs
Automation
Development &
Maintenance

5. Automation Objectives
Automating Right Tasks
Utilization of existing
automation
Sustaining and Growing
Automation Suite

6. Decentralized Approach
Our Journey…
Decentralized Approach
Centralized Approach
Benefits
Easy identification of automation opportunities
Higher collaboration between application & automation teams
Challenges
Automation development has become expensive
Usage of different tools & solutions
Standards & Best practices are not shared across applications
Benefits
Standardization of automation tool set at enterprise level
Sharing of coding best practices, templates and standards
Cost of automation development is lowered
Challenges
Slower automation coverage progression
Lower automated scripts utilization
Funding Challenges

7. Our Journey Cont’d. – Challenges & Solutions
Identification of automation opportunities
Prioritization of automation projects based on ROI
Cost of automation
Implementation of optimized designs
Challenges
Early completion of maintenance for automation scripts
Funding for automation projects
Collaboration between automation & business teams
Solution
Structuring automation as strategy & delivery team
Solution
Create automation life cycle process and integrate with company's overall software development life cycle

8. Automation Team Structure
Strategy Team
Automation
Delivery Team

9. Automation Strategy Team
Definition:
To provide automation strategic direction and identify potential new automation opportunities
Characteristics:
Project management experience
Various automation frameworks/tools knowledge
Ability to manage multiple projects in parallel
Very strong written and verbal communication skills
Very strong leadership and analytical skills
Activities:
Process & Best Practices
Assessment & Strategies
Design Solutions & Techniques
Estimations & Planning
ROI Cost/Benefit Analysis
Reports & Metrics
Tool Technical Support & Analysis

10. Automation Delivery Team
Definition:
To develop, deliver, and maintain automation solutions
Characteristics:
Understanding of programming principles and best practices
Knowledge of various automation tools
Knowledge of multiple programming languages
Strong communication skills
Activities:
Scripts Development
Scripts Maintenance
Coding Standards
Scripts Planning & Execution
Day to Day interactions w/End User

11. Benefits of Teams Strategy
Easy identification of automation opportunities
Prioritization of automation projects based on ROI
Improved automation design
Optimized the cost of automation
Clearly defined roles & responsibilities
Better communication process
Optimized automation project planning

12. Automation Lifecycle Phases Activities Deliverables G1 G2 G3
Preliminary Estimations
Automation Test Strategy
Business Case - ROI
Selected automation tool
Detailed Estimations
Go, No Go
Automation Project Plan
Framework documentation – Re-usable components, Portability, test data grid design, etc.
Business Process Test (BPT) components
Automation Scripts
Test Data
Tested automation scripts
Automated test execution results
Lessons Learned
Sign Off G1 G2 G3
Automation Evaluation
Understand functionality of the project / application
Understand testing processes
Understand target environments
Understand test data details
Verify automation tool compatibility
Identify Automation opportunities
Phases
Activities
Deliverables
Automation Preparation
Identify the resources needed.
Understand the requirements with the help application demos.
Create automation framework / design
Identify timelines and milestones
Training required for automation tools
Tool installation if required
Access & Environment Setup
Automation Generation
Create Business Process Test (BPT) components
Parameterize test data in BPT components
Review components for quality and correctness
String the BPT components together to create a test script
Develop automation scripts for BPT components
Identify test data to conduct unit & acceptance testing
Unit testing of automation
Deliver automation to application team
Execute automation by application team
Validation of ROI
Retrospection
Sign off
Execution & Acceptance
**Our Lifecycle reference HP BPT as part of our life cycle process but you can apply your tool set to the process.**

13. Team Collaboration Automation Strategy Team Automation Delivery Team
Evaluation
Automation
Preparation
Automation
Generation
Execution
& Acceptance

14. Automation ROI Calculator

15. Automation ROI Calculator

16. Integration of Automation Lifecycle & Software Development Lifecycle
Evaluation
Feasibility/Analysis
Automation
Preparation
Design
Automation
Generation
Implementation
Execution &
Acceptance
Testing &
Maintenance

17. Benefits of Process Integration
Improved understanding on automation process across the entire organization
Better collaboration between business and automation teams
Funding challenges are removed
Improved on-time delivery of automation scripts
Helped to promote the automation across entire organization
Formal escalation process developed
Initialized the creation of automation reports & metrics

18. Automation Coverage – Regression Testing
35%
Year-2008
42%
Q1-2009
45%
Q2-2009
47%
Q3-2009
52%
Q4-2009

19. Manual Testing Efforts Automated
13,803 hrs
11,750 hrs
10,200 hrs
8,570 hrs
7,047 hrs
Year-2008
Q1-2009
Q2-2009
Q3-2009
Q4-2009
Year
As of 2008
As of 2009
2009 only efforts
Total Efforts Automated
7047 hrs
13803 hrs
6756 hrs

20. Utilization & Cost Avoidance
150%
134%
130%
87%
45%
Year-2008
Q1-2009
Q2-2009
Q3-2009
Q4-2009
Cost Avoidance
10,810 hrs
9,867 hrs
9,618 hrs
7,200 hrs
Q1-2009
Q2-2009
Q3-2009
Q4-2009

21. Savings Vs. Cost Avoidance
Cost Savings
Efforts difference between manual and automation testing
Manual (Not automation) execution frequency
Produced by automation strategy team after assessment
Automation ROI Calculator
Cost Avoidance
Test coverage difference between manual & automation testing
Automation execution frequency
Generated from Quality Center after release implementation
Helped to improve the credibility of automation metrics significantly

22. Automation Metrics – Best Practices
Automation is a way of Testing and process followed for manual testing can be applied
Best Practices followed in Test Management Tool – Quality Center
Define automation work flow
Regression Testing identifier
Automation Assessment status
Automation Development status
Store automation inventory
Manage automated test scripts execution

23. Questions