1. Test Case Management and Results Tracking System October 2008 D E L I V E R I N G Q U A L I T Y (Short Version) www.megalith-solutions.com

2. What is TestTrace ?  System for Managing Test Cases and Test Results  Centralized  Web-based  Open source code  Can be used during testing of Software and Firmware  Import and Export to/from Excel, TXT, XML, HTML, Anite and R&S log files, etc.  For the engineers working in the field the cycle “TestTrace->Excel->TestTrace” is supported  Developed according to the requirements of R&D engineers, test engineers and managers D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com

3. Results in MS Excel Text log files Description of Software, Firmware, etc Unifying different types of information in centralized storage Results in HTML Data Base D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com

4. Working with Tests Results (Overview) D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com

5. Workflow overview Test Plan Voting defines final verdict Regression analysis Automated grouping & sorting of test results Automated Test Run Manual Run / Rerun Progress against test plan Cross Build analysis Release Note Import results into Data Base. Grouping and sorting of the test cases is done automatically Final verdict is calculated according to the selected voting scheme. By default if there was at least one PASS status among all runs then final verdict is PASS. Regression analysis is performed automatically on user request. Generated “Release Note” shows all regressions in comparison to previous test results. You can see the progress of testing against a test plan. Results of testing are shown for selected builds on one page. Helps to see test failures across many builds. Final “Release Note” is submitted to management Define a list or test cases to be run. By default the status of all test cases is set to “Not run” Run test cases (on target and simulation) in automated mode Run\rerun selected test cases manually. D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com

6. Test 1 Test 2 Test 3 Test 4 Test 5 … Test Plan A Repository developers testers managers accessible by different users Test Plans are created by manager Step 1. Prepare Test Plans Manager D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com Test 1 Test 2 Test 3 Test 4 Test 5 … Test Plan B

7. test 1 test 2 test 5 test 7 Test Plan C Manager build_12Nov06_Fix095 test 1 test 2 test 3 test 4 test 5 test 6 Test Plan B Creates new build record Selects appropriate Test Plan assigned Test Engineers Run 1Run 2Verdict Perform tests and store results Product ManagerDevelopers Statistical DataRegression Reports test_01 (log-file) test_02 (log-file) test_03 (log-file) Regression Reports test_01 test_02 test_03 Repository Step 2. Testing a new Software/Firmware build Regression Reports test_01 (log-file) test_02 (log-file) test_03 (log-file) Comparison Reports b1 b2 b3 test_01 test_02 test_03 D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com test 1 test 2 test 5 test 7 Test Plan A

8. Levels of data presentation to the user D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com

9. Custom Report “A” Custom Report “B” Builds Level 1 Levels of data presentation to the user Overview... D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com

10. Builds Custom Report “A” Custom Report “B” Level 1 Level 2 Tests grouped by “series” Overview 44-series26-series D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com Levels of data presentation to the user

11. Builds Custom Report “A” Custom Report “B” Level 1 Level 2 Level 3 Tests description & tests history Tests grouped by “series” Overview D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com Levels of data presentation to the user

12. Builds Custom Report “A” Custom Report “B” Level 1 Level 2 Level 3 Tests description & tests history Tests grouped by “series” Overview D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com Levels of data presentation to the user

13. D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com What does TestTrace provide for you

14. You can use TestTrace to D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com store the test results in a structured and organized way; track the history of the test case status; write notes for each execution of the test case; store description of test case; define and reuse Test Plans; check the completeness of testing against Test Plan; view reports about test execution; compare test results with previous results; quickly identify regressions; and more …

15. Data stored in TestTrace D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com Test Cases (with any requested attributes) Test Plans Test Results Groups (with any requested attributes) Test Runs (with test equipment reference) Test Results Statuses (with text notes and reference to log-file) Test Equipment (with any requested attributes) User Equipment (with any requested attributes) guest; developer; manager; administrator. User groups with different access rights:

16. D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com Data presentation in details

17. D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com Slides included in the full version of this presentation Getting to the tests results Tests results page in details Detailed information about Filters Using filters on the “overview” pages Using filters on the “test results” pages Test Results analysis in TestTrace: Release Note and Cross-Label Analysis Test Results History You can request a full version of this presentation by sending email to info@megalith-solutions.cominfo@megalith-solutions.com

18. Test case descriptions (general information) Every test case can have attributes to store additional information about a test case. These attributes can be added on user request. D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com Example of attributes: Requires – text describing what is required for the test; Tests – text describing what is tested by the test; Sequence – text describing the performed steps; Expectations – text describing what is expected at the end of the test.

19. Tested Releases Status Pass / Fail Test case descriptions (3GPP) For telecommunication tests the attributes can reflect 3GPP data. The green table on the picture below, for example, shows the 3GPP attributes. D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com 3GPP attributes Note: these fields were automatically imported from excel file downloaded from 3GPP.

20. Filters D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com Viewing information in TestTrace

21. Available Filters D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com Filters applied to the Test Results: Filter Tests and Results by TestPlan; Filter Tests and Results by predefined filter; Filter Results by Time Interval. Filters applied to Test Results Groups: Filter Groups by Type; Filter Groups by Time Interval; Filter Groups by Attribute (for example, Customer, SW Release, etc) ; Example of filtering Test Results Groups by Customer “Toshiba” Example of filtering tests

22. D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com Test Equipment management

23. User Equipment List D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com This is your test platform : PCB boards, finished products with your hardware, etc. New attributes can be added

24. Test Equipment List D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com This is usually 3 rd party test equipment. New attributes can be added

25. D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com ( Automated loading of test results, Import of historical data ) Data Import / Export

26. Import of test results from log files TestTrace architecture allows an easy and quick way for development of parsers to handle practically any test results log format. ASCII text XML HTML Data Base … other formats Anite *.vdc R&S *.html Excel *.xls D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com

27. Export of test results into other formats ASCII text XML Data Base … other formats TestTrace architecture allows easy and quick development and adding of new “plug-ins” to handle export into ant file format. D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com

28. Using Excel files and TestTrace Historical Excel file Contains already defined: -List of test cases; -Test Plans; -Test Results. If you have a lot of historical data stored in Excel files you can import them into TestTrace. Once you have all the data in one database you can perform statistical analysis. For the engineers working “in the field” you can have TestTrace and Excel files coexisting: (a) an Excel file is generated containing the list of tests and test descriptions; (b) engineer performs tests in a remote location, putting test results into excel file; (c) results from the excel file are imported into TestTrace TestTrace imports: List of test cases; Test Plans; Test Results, etc. Excel file for test results Contains: -List of test cases based on a TestPLan; -Test Results. Import only the results D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com

29. D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com Extending the functionality and Customization

30. “Plug-in” approach for adding new functionality Plug-ins can written and used to: - parse/import new formats of data; - export data in a new format; - implement new reports to display information in the way you want; - connect to other systems to control them; - to connect from other systems and to control TestTrace from them. Easy to integrate new functionality TestTrace core Interface New functionality implemented as plug-in D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com

31. Customizing user interface Each window is implemented as a separate module and the information displayed in the window is defined by a template. Templates are implemented as “plug-ins”. The template defining a window can be modified to suite your needs. You can also copy a template and from the copied template create your own window to display the information you need. A template implemented as plug-in defines what is displayed in a window D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com

32. D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com How do we work with our Customers

33. We provide full support during all stages of evaluation D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com Evaluation You receive from us evaluation version of our software. We help you to import your test results data Pilot Project Additional Customizations We ask you about your requirements, the way you manage test results now and Your Vision of test results management. If needed then based on your requirements we customize TestTrace to fit your Vision. Later we can help you with additional customizations.

34. D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com Additional Slides

35. D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com [ additional slides ] Advantages of using Data Base

36. Three main advantages: Centralized data management Data independence Systems integration Advantages of using Data Base (1/2) Redundancies and inconsistencies can be reduced Better service to the Users Flexibility of the system is improved Cost of developing and maintaining systems is lower Standards can be enforced Security can be improved Integrity can be improved Enterprise requirements can be identified Data model is developed, enforcing discipline and standardization Bring the following benefits: Data Base D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com

37. Test results data will not disappear somewhere Web-base interface provides one point of access, easily accessible from any computer Unified user interface simplifies usage of the tool by different users groups, and shortens time and effort spent on deployment of the new tool. Unified data access technology : one SQL statement to extract any data and perform any comparison instead of dedicated parser for each type of test results. Open interface for integration with other systems. Easier management: backup, coping, restoring. Alerting users on some events (email notification) Graphs can be used for data analysis (to show trends in test results) Statistics and metrics can be gathered and calculated Advantages of using Data Base (2/2) Data Base Practical benefits D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com

38. Main functions of the system. Stakeholders. Managers Developers Regression Analysis Everyday management of test campaigns and tests results Statistical Data Reports Test Engineers D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com

39. Questions.. ? Ask us by email: info@megalith-solutions.com info@megalith-solutions.com Visit our web-site: www.megalith-solutions.com www.megalith-solutions.com D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com

40. Demo Request by email: info@megalith-solutions.com info@megalith-solutions.com D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com

41. Thank you D E L I V E R I N G Q U A L I T Y www.megalith-solutions.com You can request a full version of this presentation by sending email to info@megalith-solutions.cominfo@megalith-solutions.com