1. The Role of Experience in Software Testing Practice Zahra Molaei Soheil Hedayatitezengi Comp 587 Prof. Lingard 1 of 21

2. Introduction 1 1. We will look at three projects that overseen by Siemens Austria to find out how experience used in software testing is important. 2. The goal of paper is to “better understand the role of experience for effective testing in order to develop successful testing strategies and tool support.” 2 of 21

3. Structure of paper 1. What is the Motivatio n of the paper. 2. A brief overview of the 3 investiga ted projects 3. We Look at the presented result by author. 4. We look at the author’s Conclus ion. 3 of 21

4. Paper Motivation 1. Few empirical studies exist that focus on how software testing is conducted in practice 2. We need more demonstration of how software testing is done rather than reports of it. 3.Experience is an important factor for developing test cases 4 of 21

5. Research Question All three case that studied in the paper analyzed based on following questions: What testing activities are based on experience? What is the perceived value of experience for testing? What defined experience- based testing techniques are used? How has the experience relevant for testing been established? What measures are taken for managing and evolving experience in testing? 5 of 21

6. Data Collection How they have collected the data in the paper? Testers, developers, test managers and project managers were interviewed based on the presented research question. Project documents such as test plans, spes doc were also analyzed. The participants of the study reviewed the data that was collected to verify that it was accurate. 6 of 21

7. Description of three projects This source for this table was the article. 7 of 21

8. Mission of project: Develop a high-reliable and high-performance software platform including IP-based services for telecommunication networks. Size of project: mid-sized, 35 People worked on this project, 12 in testing. Characteristics: Development and testing were distributed in Austria, Germany, other areas of Europe. 8 of 21 Description of three projects Case 1. Telecommunication

9. More than 60 percent of the tests addressed non-functional requirements such as availability, performance. The system had to support two different hardware platforms, each platform with different configurations. Testing activities spanned over different levels : Component testing, integration testing and system testing were done. 9 of 21

10. Description of three projects Case 1. Telecommunication Component Tests: Conducted by developers, test cases were derived from interface specifications, tests test were simulating various hardware device. Based on personal experience of developers additional tools applied. integration Tests: mainly concerned with performance issues and done at the same time as development. Some tests were devised from the developers' experience due to the lack of detail in the complex requirements. System Tests: Test cases were derived from the requirements. There were based on tester’s interpretation of requirements. Regression Testing also became important as test cases increased. The selection of tests for regression testing was made based on the developers' experience. Developers worked with testers in order to let them know what parts of the software were changed. 10 of 21

11. Description of three projects Case 2. Banking and Insurance Mission of project: Replace the existing host application in the domain of banking and insurance with a Web application written in Java.(1 year) Size of the project: Project team size: 15-20 people. The project team were consist of : employees of the banking and insurance company, Siemens personnel and people from other software engineering companies. 11 of 21

12. Description of three projects Case 2. Banking and Insurance

13. Characteristics: The development process was incremental and consisted of several releases. The requirements consisted of use cases which were input for the development and implementation and system testing. Testing consisted of unit, integration, system and acceptance testing. 12 of 21

14. Description of three projects Case 2. Banking and Insurance Unit and integration testing were done under the guidance of senior developers with experience in the technology. System tests: a team of 5 was drawn from important users and domain experts. Test cases were from use cases in a formal way. Because many functional and non-functional requirements were incomplete or ambiguous, some tests in the second release of application had to be made based on tester’s experience. In second release of application parts of the system were tested much more thoroughly based on their knowledge of risk of potential defects in critical part of the problem. 13 of 21

15. Description of three projects Case 3. Railways System. Mission of project: Investigating testing of an electronic interlocking system of a railways control system which is a safety-critical project. Size of project: Large scale project and about 100 people were involved in this project. Testing and development of the project was based on the CENELEC standard. 14 of 21

16. Description of three projects Case 3. Railways System. The standard that used in the system promoted the V-model for development and testing. There had to be a test case which corresponded with every single requirement. Test cases at the system and software level were designed by tester based on their experience with interlocking system.

17. Description of three projects Case 3. Railways System. Test cases for lower-level parts of the system were developed by testers familiar with railway control systems. Also, the requirements were not all precise, so exploratory testing, which was experience- based, had to be done. 15 of 21

18. Summary of Results What activities of testing are based on experience? Test case development Regression testing Test automation 16 of 21

19. Summary of Results What is the perceived value of experience for testing? More defects are found Missing parts of the requirements specifications can still be tested for. Experience-based testing is expensive 17 of 21

20. Summary of Results In Telecommunications, the test cases had to be automated before the execution which made experience-based testing difficult. This technique was infeasible for a large extent of test cases. In Banking and Insurance, deriving test cases from experience had been planned for the second iteration. Error guessing was used in this case to discover defects, and these test cases were based on where failures had occurred before. In Railway Systems, error guessing had to satisfy the IEC 61506 standard. What defined experience- based testing techniques are used? 18 of 21

21. Summary of Results In telecommunications, gained experience was based on previous testing of the same systems. In Banking and insurance, intimate knowledge of organization due to working for the company for several years. In railway, several years of experience in testing different types of interlocking systems. Own knowledge of railway system. How has the experience relevant for testing been established? 19 of 21

22. In all three projects, management considered experience as essential for effective testing. All interviewed partners emphasized training on the job. Experience was gained from just participating in testing itself. Iterative development also gave testers the opportunity to learn what to test for in each version of the software based on past experience. What measures are taken for managing and evolving experience in testing? 20 of 21 Summary of Results

23. conclusions Finding the optimal mix of testing knowledge and domain knowledge is thus a vital issue for successful projects reviewing. improving requirements specifications for testing is an easy yet effective measure also to improve testing. Existing testing tools failed to leverage experience for effective testing and future work needs to be done to enhance testing tool usage. 21 of 21

24. Questions