1. Supervisor: Prof. Jyri Hämäläinen Instructor: Jari Simolin (M.Sc), Nokia Siemens Networks Espoo, 15.12.2009 Jyri Ilama

2. Testing terminology Test automation process & Benefits of TA UMTS architecture IPA2800 platform Call Management domain Continuous Integration The CI project of Call Management CI FRT pilot project Lessons learned What went wrong? Results and analysis 2

3. (our) Functional testing Using the system Actually, a script uses a sub system of an individual entity (one, separate network element) The actual, acceptance / system testing is done later in a System Verification phase Regression testing Testing that the software still works after changes are made in some unit 3

4. 4 How can this be achieved? Whats inside the black boxes? Pitfalls to avoid Good practices

5. The required effort for selecting a set of test cases and executing them takes a few minutes More tests can be executed more often Some cases are even impossible to perform by manual testing Use of resources Repeatability Reusability Simply: Saving resources and time! 5

6. 6 We are interested in RNCs and MGWS

7. Network elements are complex networks of interconnected computer units, communicating via message connections Network elements have very strict requirements on fault tolerance the five nines availability performance 3G network element platform based on DX200, running in RNCs and MGWs But much more efficient and clearer of its architecture 7

8. Responsible of all call related operations Setup, release, and connecting of calls Capacity and performance Remaining calls in case of recovery actions E.g. Unit failures 8

9. A practice of software development, where the members of a team integrate their work very frequently, usually at least daily Whenever an integration is made, it is verified by an automated build to detect possible integration errors as quickly as possible A complex system that involves lots of people, servers, tools and connections working tightly together 9

10. 10

11. A computer that runs some CI software and executes integration builds whenever software changes are made Tools CruiseControl & Hudson Java-based free software, e.g. Ant, Maven and Rake builds supported Modularity: plug-in support 11

12. Started in May 2007: only PRB compilation builds in the beginning, unit tests added later No decent FRT software available Hipla: an output report of even hundreds of thousands of lines of plain text, not so nice Summer of 2008: HiBot was designed for running old (Hipla) test cases in a new way HIT2Python translator, all old Hipla code -> HiBot Telnet functions implemented manually, in a hurry, by a person who left the company This was the point where this tool was left in the hands of a summer trainee; me 12

13. I got HiBot up and running quickly, against all estimates Started implementing and/or debugging all the functionalities Lots of occasional problems with the Telnet connections 13

14. MGW test cases of Call Resource Handling Other pilots by other teams in Call Management Even more work with correcting all the commands Still random crashes and annoying connection and prompt detection problems External problems IPA Reporting Server Other tools, such as CruiseControl and SVN 14

15. Test automation problems leading to jams and crashes The old, manual test cases had to be automated properly June 2009: The "Telnet guy" came back The problems were found and defects corrected Time for me to focus on the essential: to find and fix defects in IPA2800 platform I implemented a very useful script 15

16. A very useful script for investigating results 16

17. Old manual cases are the most difficult ones Changed outputs of functions: scanning strings If the printout changes a little, does your macro still work? The principle of modularity No hard-coded values Reusability of test cases a Forever-loop is definitely not your friend 17

18. Consecutive execution A test case should be possible to run in any kind of situation, no matter if there are e.g. other calls already, hanging, or whatever Proper teardowns At the end of test suites, so the execution of a new suite can be started from scratch Execution order Problematic suites: Which first? Possible system breakers as last ones 18

19. New software: Focus on the essential Critical features/functionality R&D&T should be done instead of R&D And regression testing should be as important as the testing of new features Scrum mode: were too deep inside it. Tasks that are not backlog items wont get enough priority, at least easily Needed: RT sprints or a team dedicated to RT! Close all external problems out of sight Write totally new test cases if possible 19

20. A stabile, working CI environment for our FRT Real defects are found all the time Also Backlog Item Testing can be done efficiently The "output2report" script was found very useful Good lessons about test automation As well as writing modular, intelligent test cases A very good quality level of the product release! 20

21. 21