1. Metrics in Automation testing -
theory and practice
Igor Cernopolc
15th May 2014

2. Metrics, Models, Measurements and Analyses
Why Metrics?
TQM
Customer Focus
Process Improvement
Human Side Of Quality
Metrics, Models, Measurements and Analyses
Testing Metrics help organization to obtain the information it needs to continue improve its
testing process, reduce errors and increase the quality of deliverables, thus achieving the desired Goal.
Easy for management to digest one number and drill down, if required.
Different Metric(s) trend act as monitor when the process is going out-of-control.
Metrics provides improvement for current process.

3. What is a that?
Metric is a standard unit of measurement that quantifies results.
It is obtained using measurement based techniques which are applied to processes, products and services as applicable.
Metrics – few words
Metrics in automation testing
Examples of calculation (few main metrics to be acknowledged)
Handy tools
Metrics from experience (personal and questions to the audience to share) QA

4. Metrics categories Coverage Progress Quality parameters for measuring:
testing scope
success of testing
Coverage
parameters that help identify test progress to be matched against success criteria
metrics are collected iteratively over time
Progress
measures of testing product/process quality
usability, performance, scalability, overall customer satisfaction, and defects reported are a few examples
Quality
Metrics type:
Coverage: meaningful parameters for measuring test scope and success.
Progress: parameters that help identify test progress to be matched against success criteria. Progress metrics are collected iteratively over time. They can be used to graph the process itself (e.g., time to fix defects, time to test, etc.).
Quality: meaningful measures of testing product quality. Usability, performance, scalability, overall customer satisfaction, and defects reported are a few examples.

5. Lifecycle Identify metrics to use Define metrics identified
Analysis
Identify metrics to use
Define metrics identified
Acceptance
Explain the need of metric to stakeholder and testing team
Educate the team to prepare and collect the metrics
Evaluation
Capture the data
Verify the data
Calculate metrics
Reporting
Develop the report with effective conclusion
Distribute report to stakeholders
Obtain feedback

6. What are automated testing metrics?
Automated testing metrics are metrics used to measure the performance (past, present, and future) of the implemented automated testing process and related efforts and artifacts.
Metrics – few words
Metrics in automation testing
Examples of calculation (few main metrics to be acknowledged)
Handy tools
Metrics from experience (personal and questions to the audience to share) QA

7. Is my metric a good one?
A good automated testing metric has the following characteristics:
It is objective
It is measurable
It is meaningful
Data for it is easily gathered
It can help identify areas of improvement
It is simple.

8. Metrics in automation testing
Automatable Percentage
Test Automation coverage
Automation test design productivity
Automation test execution productivity
Effort savings through test automation
ROI of Test Automation

9. Automatable Percentage
𝑃𝐴= 𝐴𝑇𝐶 𝑇𝐶 *100
PA = Percent automatable
ATC = Number of test cases automatable
TC = Total number of test cases

10. Test Automation Coverage
𝑃𝑇𝐶= 𝐴𝐶 𝐶 *100
PTC = Percent of automated test coverage
AC = Automated TC’s
C = Total number of TC’s

11. Automation test design productivity
𝑷𝑻𝑫= 𝑨𝑪/𝑻 𝑷𝑨𝑪 *100
PTD= Percent of automated tests
AC = Automated TC’s
T= Timeframe
PAC = Planned TC’s

12. Automation test execution productivity
𝑷𝑻𝑫= 𝑨𝑪/𝑻 𝑷𝑨𝑪 *100
PTE = Percent of executed tests
AC = Executed TC’s
T= Timeframe
PAC = Planned TC’s

13. Effort savings through test automation

14. ROI of Test Automation

15. Let’s share
How we all automated, with and without metrics

16. Few more things Reliability vs Validity
Ratio, Proportion, Rate, Percentage, Absolute values
Don't measure for the sake of measure
Track and measure on an ongoing basis
Reliability refers to the consistency of a number of measurements taken using the same measurement method on the
same subject.
Validity refers to whether the measurement or metric really measures what we intend it to measure.
Don't measure for the sake of measure
Before measuring anything, set goals. What is it you are trying to accomplish? Goals are important; if you do not have them, what is it that you are measuring?
It is also important to track and measure on an ongoing basis.

17. Q&A
are Welcomed 

18. In one word
Before measuring anything,

19. In one word
Before measuring anything, set goals

20. Thank you
Igor Cernopolc