1. Software Test Automation

2. Test Automation
Test automation: Developing software to test the software is called test automation.
Advantages
It saves time as software can executes test cases father than human do.
Saved time can be used do:
To develop additional test cases
Perform tests like ad hoc testing
Perform some extra manual testing

3. Automation can free the test Engineers from mundane tasks and make them focus on more creative tasks.
Automated tests more reliable.
Manual – executes particular test case many times, there is a chance for human error or a bias – chance for missing defects.
Automation – More reliable results – eliminates boredom and fatigue.

4. Automation helps in immediate testing
Reduces the time gap between development and testing – Need not wait for the availabilty of Test Engineer
Automation can protect an organization against attrition of test engineers – Less person dependent – Documentation may lost when the test engineer’s leaves the job.

5. Automated tests can run round the clock, twenty four hours a day and seven days a week.
Certain types of testing cannot be executed without automation – reliability, stress and load testing.
Automation means end to end, not test execution alone – It consists of picking the right product build, generating the right test data, analyzing the results and filing the defects.

6. Test case - Automation Test case can be represented in many forms
Documented as a simple steps/Assertion statements/set of Assertions.
Example for Assertion:
“Opening a file, which is already opened should fail” – it includes the expected result also – Makes automation Engineer easy to write the code and conclude the correctness of the result.

7. Opportunity to reuse the test cases for different purpose and scenarios.
Test case for testing
Belongs to what types of testing
Click whether log in works
Functionality
Repeat login operation in a loop for 48 hours
Reliability
Perform log in from 1000 clients
Load/stress testing
Measure time taken for login operations in different conditions
Performance
Run login operation from a machine running Japanese language
Internationalization

8. Skills needed for Automation
Different “ generation of automation” – skills required for this depends on what generation of automation the company is in or desires to be in the next future.
First Generation
Avoids repetitive nature of execution tasks.
Facility – to record sequence of actions - Recorded scripts are played back multiple times.

9. Disadvantages
Difficult to perform general tests – Ex: when a report has to use the current date and time.
Frequent changes/No opportunity to reuse or rerun the tests – Record and playback generations test automation tools may not be effective.

10. Second Generation
Helps to develop test scripts that generates the set of input conditions and corresponding expected output.
Repeat the tests for different input and output conditions.
Changes does not require the automated test cases to be changed as long as the input conditions and expected output are still valid.
It focuses on Black Box approach.

11. Third Generation
It enables layman to create automated tests.
No input and expected output conditions required for running the tests – Automatically generated.
Users needs to specify the operations only like login, download
Scenarios for test execution can be dynamically changed using the framework that is available in the approach of automation.

12. Classification of skills for automtion
Different levels of skills are needed based on the generation of automation selected.
Classification of skills for automtion
I generation
II Generation
III generation
III Generation
Skills for test case automation
Sills for framework
Scripting languages
Scripting language
Programming language
Record-playback tools usage
Design and architecture skills for framework creation
Knowledge of data generation technique
Design and architecture of the product under test
Generate test requirements for multiple products
Usage of the product under test
Usage of the framework

13. Scope of Automation
Certain types of tests automatically lend themselves to automation:
Stress, reliability, scalability and performance testing
2. Regression Tests – Repetitive in nature. These test cases are executed multiple times during the development phase. – Saves time – it will be useful for ad hoc/ creative testing

14. 3. Functional Test : Requires complex set up and specialized skill – In automation its possible with less skilled people. 4. Automation should consider – areas those goes less requirement changes/no changes. Normally changes in requirements cause scenario and new features to be impacted, not the basic functionality of the product. Ex: Steering, brake and accelerator can not change over the year. – Automation should done in non user interface portions of the product.

15. Automated test cases need to be maintained till the product reaches obsolescence.

16. Design and Architecture for automation
Design and architecture is an important aspects of automation. The design has to represent all requirements in modules and the interactions between modules.
Architecture for test automation : Test infrastructure – Covers a test case database and a defect repository.
TCDB – Test cases and steps to execute them .( when a particular test case was run and whether it passed or failed)

17. Picture

18. Defect DB – It contains details of all the defects that are found in various products that are tested in a particular organization. ( when the defect was found, to whom it is assigned, what is the current status, the type of defect and its impact and so on)
For automated test case, it will automatically submit the defects to the defect DB during execution

19. Scenario and Configuration File modules
Scenario – how to execute a particular test case
Configuration file : Contains set of variables used in automation. CF is an important for running the test cases for various input and output execution conditions and for running the tests for various execution conditions.

20. Test Framework – it does two works:
What to execute
How they have to be executed?
It picks up test case from TCDB and picks up the scenarios and executes them. The variables and their defined values are picked up by the test framework and the test cases are executed for those values.

21. For Example : if there is a scenario that requires a particular test case be executed for 48 hours in a loop, then the test framework executes those test cases in the loop and times out when the duration is met. Note: Test frame work can be developed by the organization/bought from vendor.

22. Tools and Results Modules
Set of tools are required to pick up test case and run.
Build tools : to extract TC from TCDB and compile the test case
Runtime tool : To run the compiled code.

23. When a test framework executes a set of test cases with a set of scenarios and the different values will be stored for future use.
The results that come out of the tests run by the test framework should not overwrite the results from the previous test runs.
For example : If test engineer can request the test framework to “ execute all test cases that are failed in previous test run” – Should not overwrite the results of previous.

24. Result Generator and Reports/Metrics Modules
Preparing repots is complex task – it should be part of the automation design.
Technical report – Debug reports for developers

25. Generic requirements for test tool/framework
No hard coding in the test suite
Adding test case/suite expandability
Adding new test case should not affect the other TC.
should not result in retesting the complete test suite.
3. Reuse of code for different types of testing, test cases

26. 4. Automatic setup and clean Each test program should have a setup program that will create the necessary setup before executing the test cases. A set up for one test case may work negatively for another test case. So it is important not only to create the setup but also “undo” the setup soon after the test execution for the test case.

27. 5. Independent test case To execute a particular TC , it should not expect any other TC to have been executed before nor should it implicitly assume that certain other TC will be run after it. EX: TC 2 to be executed after TC-1

28. 6. Dependent TC
Particular TC should be executed before or after a dependent TC
Insulating TC during execution
Interruption of test case execution by some events or signals – Ex automatic pop up screens on web browsers. – It affects TC execution- Framework provides an option for users to block some of the events.

29. 8. Coding standard s and directory structure CS and DS for a test suite may help the new Er in understanding he test suite fast and help in maintaining the test suite. 9. Selective execution of test cases Framework may have multiple test suites. Test suite may have multiple test programs. Test program may have multiple test cases. The test tool or a framework should have a facility for the test engineer to select a particular test cases or set of test cases. EX: test-program-name 2,4,7-10.

30. 10. Random execution of test cases Ex:1 random test-program-name 2,1,5 Test engineer wants the test tool to select one out of test cases 2,1,5 and executed. Ex: 2 test-program 1(2,1,5) test-program2 test-program5 Test Engineer wants one out of test programs 1,2, and 5. If program 1 is selected then one out of test cases 2,1 and 5 to be randomly executed. if program 2 or 3 are selected, then all the test cases in those programs are executed.

31. 11. Parallel execution of test cases It simulates the behavior of several machines running the same test and hence is very useful for performance and load testing. 12. Looping the test cases Reliability testing requires the test cases to be executed in a loop. Iteration loop Repeat_loop, 50 Test –program 1(3) – test case 3 form test program1 is repeated 50 times. Timed loop Time_loop, 5 hours test- program1 (2,1,5) test_program 2 Test cases 2,1 and 5 from test program1 and all the test cases of program 2 are executed in order, in a loop for five hours.

32. 13. Grouping of test cases The grouping of scenarios allows several tests to be executed in a predetermined combination of scenarios. it allows the selected test cases to be executed in order, random, in a loop all at the same time. Ex: Group scenario 1 Parallel, 2 AND repeat, Scen1 test_program1(2,1,5) test_program2 test_program3 All test programs are executed by two instances simultaneously in an iteration loop for 10 times.

33. 14. Test case execution based on previous result Regression test methodology requires that test cases be selected based on previous result and history. Ex: Rerun all the test cases which were executed previously Rerun only failed/not run test cases Execute all test cases that were executed on “ Jan 26, 2015”.

34. Challenges in Automation
Automation should panacea ( Cure all / solution ) for all problems nor should it be perceived as quick – fix solution for all the quality problems in a product
Automation requires significant initial outlay of money as well as steep learning curve for the test engineer before it can start paying off.
Management should have patience and persist with automation. The main challenge is because of the heavy front-loading of costs of test automation, management starts to look for an early payback.

35. Successful test automation endeavors are characterized by unflinching management commitment, a clear vision of the goals, and the ability to set realistic short term goals that track progress with respect to the long term vision.
If any of these attributes are found lacking in the management team, it is likely that the automation initiatives will fail. And it may even wrongly start a negative mindset about automation.

36. TEST - Measurement & Metrics
Against:
Collecting metrics is too hard ... it’s too time consuming ... it’s too political ... they can be used against individuals ... it won’t prove anything
For:
In order to characterize, evaluate, predict and improve the process and product a metric baseline is essential.

37. Terminology
Measure: Quantitative indication of the extent, amount, dimension, or size of some attribute of a product or process.
Metrics: The degree to which a system, component, or process possesses a given attribute. Relates several measures (e.g. average number of errors found per person hour)
Indicators: A combination of metrics that provides insight into the software process, project or product
Direct Metrics: Immediately measurable attributes (e.g. line of code, execution speed, defects reported)
Indirect Metrics: Aspects that are not immediately quantifiable (e.g. functionality, reliability)
Faults:
Errors: Faults found by the practitioners during software development
Defects: Faults found by the customers after release

38. Test Metrics and Measurement
Metrics & Measures
Why?
Collecting metrics about processes and products which are meters for organization’s performance and continuous improvement.
Measures
Quantitative indication of the extent, amount, dimension, or size of some attribute of a product or process. A single data point.
a certain quantity or degree of something .
A standard unit used to express the size, amount or degree of something.
Examples: product size in lines of code
Distance between two places
No. of developers and testers

39. Metrics:
Metrics is a relative measurement of status of process or product in terms of two or more entities taken together for comparison.
(e.g. average number of errors found per person hour)
Note: Distance and Time is not a metrics, as it is, but to calculate the speed, the ratio between D and T – It indicates a relative measurement of two entities and makes a better sense for taking decisions.

40. Metrics can be used for measurement of performance and improvement/deterioration of performance, and actions can be initiated when we compare different entities
Test team gives lot of inputs under testing –
Identified defects
Requirement Coverage
Functional coverage
Levels of compliances ( fulfillment)
These are helpful for organization – to understand the level of project management, process implementation, strong and weak areas of projects and actions needed to improve the weak area.

41. Testing related data
Test results data
– specifies the outcome of every test case in terms of whether it has passed or failed.
– Accepted or Rejected
- Deliver a product or not
- Need any action to fix the defects found
Test reports may contain the recommendations on the basis of test results.

42. Test suites and Test events
- Gives input like various coverage
- which will be useful to measure test scenario and No. of test cases defined in test plan.
Defects log
- Useful for measuring software product quality
- More defects found – Major problems in software project management and development processes.

43. Efficiency of development and test team
- Evaluated data like defect fixing efficiency and test case execution efficiency.
Data may be related to the following:
Project plan defines and priorities the test factors and quality factors – Important for product. ( Must/should be and could be )

44. Expected Business objectives validations: Data related to acceptance testing which can tell whether the customer requirements are fulfilled or not and whether the defects found can be taken for process improvement or not.
Expected Interface objectives validations: Test result data which can tell whether the data/objects can be correctly passed among software components of the same system or between different systems.
Interfaces – internal – defined by low level design
Interfaces – External – defined by high level design
External interfaces must work in different configurations such as operating systems, browsers and databases.

45. Expected Vs Actual Functions/sub functions coverage:
- Test data related to requirements and the outcomes of tests conducted - Gives information about the coverage of requirements and coverage of functionalities.
Units related data: Data related to design, coding and testing associated to the smallest identifiable software component called unit is required to ensure that every small part of the application works correctly as defined.
- Individual unit work correctly – Higher probability that entire software will also work correctly.

46. Platform/Browser related data : Test results related to hardware/software environment, operating systems, browsers and databases in which system will operate represent a compatibility data of software.

47. Defects data
Defects data is useful to define the status of software application.
Useful to evaluate the processes used for building software.
Defects data useful to evaluate the process capability of an organization, in terms of deployment and ability of processes to produce desired result.

48. Defects data may include the following:
Data about defects uncovered during review and testing
Date of defect uncovered and Date of defect is closed
Defect ID, type of defects uncovered, module where they originate - Gives information about weak areas in software development process.
Severity of defects
How defects was uncovered ( test plan/test script)

49. Some metrics are used by software organizations in general to capture the development and test processes and products .
1.Process Metrics
- it deals with processes used for software development and testing. They are considered to be independent of the type of project/product, technology and customer.
2. Project / Product Metrics
- It is related with individual project/product. They are completely dependent on the type of project, product , technology and customer.

50. Process Metrics
- Process metrics mainly refer to the measurement of the processes used for software development and testing.
- It is very difficult to measure the process directly, but we can measure the attribute of the product produced by the processes, to measure the process indirectly.

51. - Generally, process metrics refer to culture of an organization and patterns of working. - It cannot be changed very easily for a project and one may need to take some deliberate efforts like implementation of quality processes or updating existing processes to improve them.

52. Example: Requirement elicitation efficiency Test team efficiency Testing efficiency Review efficiency 2. Project/Product Metrics - Project/Product metrics may differ from one instance to another and one level to another level, depending upon type of projects and customer. - So metrics may differ

53. Example
Software size
Testing coverage
Software Complexity

54. Efficiency
There are many aspects of efficiency definitions included in software metrics.
Efficiency is measured as size expressed in number of code lines/ function points developed by a project team per unit time.
Development Efficiency = Size of software/ Time required
Time may be expressed in person days, person hours , etc.

55. Efficiency in fixing the defects
It measures the efficiency of development team, in fixing the defects found by test team.
Defects identified by testing team may be fixed by development team as per priority of the defects defined by development team/customer.

56. The defects are closed only by testing team after retesting them and if these are found to be closed.
Defect fixing efficiency = (Defect closed by testing *100)/defects declared as resolved by development team
This is process metrics which talks about the ability of development team to fix the defects.

57. Defect Removal Efficiency
It talks about ability of project or organization to find and fix defects, so that they do not go to the customer.
DRE= (Defects found by project team in house*100)/ Total number of defects found by project team and customer.
This is process metrics

58. Expected level of defect removal efficiency may be very close to 100%.
As it drifts away from 100%, it shows lower ability of development and test team in finding and fixing defects.

59. Defect finding Efficiency of test team
The number of defects found by testing team per unit time or test case is a measure of defect finding efficiency of test team or test case.
This is product metrics
Defect finding efficiency ( time based) = Number of defects / Total time taken for verification or validation activities

60. TESTING EFFICIENCY
Time required by a test team to execute one test case may be measured as testing efficiency.
Testing efficiency = Number of test cases executed / Time spent in testing

61. Let ‘T’ be the test suite under evaluation, ‘m’ be the number of faults contained in the program under test ‘P’, ‘n’ be the total number of test cases and Pos (Fk) the position of the first test in ‘T’ that exposes fault ‘i'. Then the average percentage of fault detection APFD can be calculated as:

62. Fault/Test Case t1 t2 t3 t4 t5 t6 t7 t8 t9
t10 f1 ∗ f2 f3 f4 f5 f6 f7 f8 f9
f10
No. of faults 2 3 1
Time 5 7 11 4 10 12 6 15 8 9

63. Assumed that the test suite ‘T’ consist of ten test cases {t1,t2,t3,t4,t5,t6,t7,t8 ,t9,t10} and then it is assured that the tests detect the ten faults f1, f2,f3,f4,f5,f6,f7,f8,f9 and f10 as shown in Table.
Consider the two test suites TS1 (prioritized) with test sequence {t4,t1,t2,t7,t6,t9,t10,t5,t8,t3} and TS2 (Non prioritized) with test sequence {t1,t2,t3,t4,t5,t6,t7,t8 ,t9,t10}.
Rate of fault detection of test case Rti is the ratio of number of fault detected per second.

64. Rate of fault detection
Test case
Rate of fault detection t1
0.40 t2
0.42 t3
0.09 t4
0.75 t5
0.20 t6
0.25 t7
0.33 t8
0.133 t9
t10
0.22

65. Priority is set according to decreasing order of value of Rti, since the more the rate of fault detection the more will be priority.
APFD for Prioritized test case