1. Keynote SIGOS GmbH Klingenhofstrasse 50d 90411 Nuremberg Germany www.keynote-sigos.com Advanced Reporting & Diagnosis Training 2009-07-31

2. ©2008 Keynote SIGOS GmbH Testing is our Competence Repetition 2 Brief Repetition BASE Reporting

3. ©2008 Keynote SIGOS GmbH Testing is our Competence 3 SITE Report Environment real time reporting alarming Measurement strategy Test executions Test execution on different locations KPIs Base measurements (input and output)‏

4. ©2008 Keynote SIGOS GmbH Testing is our Competence 4 How to create a Report? ■ Reports are created within the Report Editor. ■ You can open the Report Editor with the Report Administration button in the toolbar. ■ The Toolbox for Reports will be opened together with the Report Editor in the navigation area Report Administration

5. ©2008 Keynote SIGOS GmbH Testing is our Competence 5 Toolbox for Reports ■ The Toolbox for Reports contains all items you need to define your reports: Folder to build up a hierarchy in your report definitions Report Definition defines a new report which consists of one or more charts Chart is a diagram which shows one or more measurements Dashboard defines the path, where a new Dashboard can be edited (via the additional Dashboard Application) Alarm defines a threshold on a measurement. If the threshold is reached, a defined action will be raised. Measurement Databases contain the measurements of the executed test cases. The layout is defined by the Measurement Strategy. Measurements are the results of the executed test cases. The calculation of the measurements is defined in the Measurement Strategy

6. ©2008 Keynote SIGOS GmbH Testing is our Competence 6 Report Definition via Drag & Drop ■ You can easily create a report by using the Drag & Drop mechanism: Toolbox for Reports Report Editor

7. ©2008 Keynote SIGOS GmbH Testing is our Competence 7 Report Editor & Working Area The created Report will automatically appear at the bottom of the working area:

8. ©2008 Keynote SIGOS GmbH Testing is our Competence 8 Report Configuration In order to configure the report, you have to set the corresponding parameters on each level: KPI Definition Level Chart Definition Level Report Definition Level

9. ©2008 Keynote SIGOS GmbH Testing is our Competence 9 Chart Definition Level - Chart Types Matrix Multi: One fixed column header (x-value-string) and multiple KPIs (y1-value) Matrix Single: One fixed column header, one fixed row header and a single KPI (y1-value)

10. ©2008 Keynote SIGOS GmbH Testing is our Competence 10 Matrix Reporting

11. ©2008 Keynote SIGOS GmbH Testing is our Competence Matrix Multi Select Matrix_Multi Choose column header (x-value-string) Choose KPIs (y1-value) 11

12. ©2008 Keynote SIGOS GmbH Testing is our Competence Matrix Multi Matrix Multi with additional alarm definition 12

13. ©2008 Keynote SIGOS GmbH Testing is our Competence Matrix Single Select Matrix_Single Choose KPIs for column header and row header Choose KPI to be displayed within the cells 13

14. ©2008 Keynote SIGOS GmbH Testing is our Competence Matrix Single Matrix Single Chart Success Rate per test case AND per location 14

15. ©2008 Keynote SIGOS GmbH Testing is our Competence 15 Derive To Feature

16. ©2008 Keynote SIGOS GmbH Testing is our Competence Derive To Functionality Report displays MeanDownloadRate for 2 operators in 2G / 3G How is the MeanDownloadRate for operator?  Create new Report OR derive report to operator 1 16

17. ©2008 Keynote SIGOS GmbH Testing is our Competence Derive To Functionality Open context menu and select Derive To 17 Right Mouse Click

18. ©2008 Keynote SIGOS GmbH Testing is our Competence Derive To Functionality Select KPI to derive from (e.g. MeanDownloadRate) Select KPI to derive from Select a_plmn (operator) 18

19. ©2008 Keynote SIGOS GmbH Testing is our Competence Derive To Functionality Derived Chart  MeanDownloadRate for operator 1 19

20. ©2008 Keynote SIGOS GmbH Testing is our Competence 20 Dashboard

21. ©2008 Keynote SIGOS GmbH Testing is our Competence Dashboard Feature Monitoring of all important KPIs on one screen No knowledgment of SITE/GlobalRoamer necessary (end user) Easy accessable via HTTP Different Dashboards for different needs 21

22. ©2008 Keynote SIGOS GmbH Testing is our Competence Dashboard Feature Drag & Drop new Dashboard definition into Report Editor Rename Dashboard definition 22 Drag & Drop new Dashboard Defiinition

23. ©2008 Keynote SIGOS GmbH Testing is our Competence Dashboard Feature Collect (create or copy) the report definitions for the Dashboard 23 Collect Reports for Dashboard

24. ©2008 Keynote SIGOS GmbH Testing is our Competence Dashboard Feature Open link SITE Dashboard 24 Access CU/UI via HTTP

25. ©2008 Keynote SIGOS GmbH Testing is our Competence Dashboard Feature Open link SITE Dashboard Choose created Dashboard definition 25

26. ©2008 Keynote SIGOS GmbH Testing is our Competence Dashboard Feature Open link SITE Dashboard Choose created Dashboard definition Open Dashboard properties 26

27. ©2008 Keynote SIGOS GmbH Testing is our Competence Dashboard Feature Select collected Report(s) to be displayed within Dashboard 27

28. ©2008 Keynote SIGOS GmbH Testing is our Competence Dashboard Feature Define coordinates of Reports within the Dashboard (if necessary) Repeat this for all Reports 28

29. ©2008 Keynote SIGOS GmbH Testing is our Competence Dashboard Feature A first Dashboard draft is being created Dashboard is displaying the SuccessRate for ALL servives on ALL used locations What about filtering? 29 Initial Dashboard

30. ©2008 Keynote SIGOS GmbH Testing is our Competence Dashboard Filter Define new filter(s) for Dashboard Navigation and assign filter icon(s) Define required MySQL-Statement 30 Drag & Drop new filter nodes Select Font Size & Icon Create MySQL filter expression Select Database & Table

31. ©2008 Keynote SIGOS GmbH Testing is our Competence Dashboard Filter If you want to know, how the services are performing in the UK  select UK 1/2 and press Update Now 31 All Services for UK

32. ©2008 Keynote SIGOS GmbH Testing is our Competence Dashboard Filter - Settings What about the view on individual services?  Define Dashboard settings REMARK: Dashboard filter will overwrite Report Definition filter! 32 1. Add New Selection List 2. Add New Filter for Selection List 3. Define MySQL Expression for filter

33. ©2008 Keynote SIGOS GmbH Testing is our Competence Dashboard Filter - Settings  Filters for individual services and locations 33 All needed Filters

34. ©2008 Keynote SIGOS GmbH Testing is our Competence Dashboard Filter - Timing Define Timing for Dashboard 34

35. ©2008 Keynote SIGOS GmbH Testing is our Competence 35 Table Reporting

36. ©2008 Keynote SIGOS GmbH Testing is our Competence 36 Table Report Drag and drop the a table symbol from the Toolbox for Reports:

37. ©2008 Keynote SIGOS GmbH Testing is our Competence 37 Table Report The report will look exactly as the main table from within the Database View

38. ©2008 Keynote SIGOS GmbH Testing is our Competence 38 Table Report Configuration In order to configure the report, one have to set the corresponding parameters similar as with the chart reports. Parameter Tree

39. ©2008 Keynote SIGOS GmbH Testing is our Competence Table Report Configuration - Column List 39 ■ You can customize the Column List in order to show only certain KPIs

40. ©2008 Keynote SIGOS GmbH Testing is our Competence 40 Table Report Configuration - Column Filter ■ Also here an individual column filter for each KPI can be defined ■ NOTE: The column filter of the KPI has to match the column filter of the report definition (if defined)

41. ©2008 Keynote SIGOS GmbH Testing is our Competence Table Report Configuration - Options 41 ■ The entries can be sorted by any of the shown columns: ascending (ASC) or descending (DSC) ■ A limit on the number of entries to be shown in the table report can be set

42. ©2008 Keynote SIGOS GmbH Testing is our Competence Table Report Configuration - MySQL 42 ■ MySQL commands can be used in the Column Filter to perform additional manipulation of the KPI data and the table default settings ■ To open the Column List Dialog Window right click on it and chose Show Details Right-click and chose Show Details

43. ©2008 Keynote SIGOS GmbH Testing is our Competence Table Report Configuration - MySQL 43 ■ Additional grouping of the entries by different KPI values can be performed by using the grouping macro in the Options field (grouping= ) Show maximum 5000 entries Group the shown data according to the a_usedplmnname Group the shown data according to the a_usedplmnname Sort the entries by SBRate descendingly and Count ascendingly Sort the entries by SBRate descendingly and Count ascendingly

44. ©2008 Keynote SIGOS GmbH Testing is our Competence Table Report Configuration - MySQL 44 Example of using the MySQL commands in the Column Field and the Options to show: the number of tests executed at all tested operators during a SIM Box Detection Campaign (Count), the number of tests where the CLI was properly transmitted (OK), the number of tests with Quality Problem (QP), number of tests where SIM Box was detected (SB), and the testing efficiency (SB Rate) Example of using the MySQL commands in the Column Field and the Options to show: the number of tests executed at all tested operators during a SIM Box Detection Campaign (Count), the number of tests where the CLI was properly transmitted (OK), the number of tests with Quality Problem (QP), number of tests where SIM Box was detected (SB), and the testing efficiency (SB Rate)

45. ©2008 Keynote SIGOS GmbH Testing is our Competence 45 Advanced Diagnosis

46. ©2008 Keynote SIGOS GmbH Testing is our Competence Reporting & Analysis 46 ■ If the displayed KPIs are aggregated, you can open them in a tabular view in order to start the drill down

47. ©2008 Keynote SIGOS GmbH Testing is our Competence Reporting Lets have a closer look to the diagnostic possibilities... 47

48. ©2008 Keynote SIGOS GmbH Testing is our Competence 48 SITE 2.3 Diagnostic possibilities I.Test case diagnosis based on the Reports a)Analyze the PASS / FAIL ratio b)Analyze KPIs c)Come to the right conclusions II.Test case diagnosis based on error-messages III.Test case diagnosis based on the SITE Trace a)Overview about SITE Trace information b)SITE Layer3 Tracing: Non-Access Stratum (NAS) c)Example 1 – reject causes d)Example 2 – request retries e)SITE Layer3 Tracing: Radio Resource Control (RRC) f)Overview about SITE Result files g)Example 1 – Screenshot of Test case HTTP Browser h)Example 2 – Tcpdump of Test case FTP_dl_ps

49. ©2008 Keynote SIGOS GmbH Testing is our Competence 49 SITE 2.3 Diagnostic possibilities Overview about SITE Trace information  By the SITE Trace View the user can look at the detailed trace produced by a test case. This is possible while the test is running and can also be opened for already finished test cases. A detailed interpretation of the send and received messages can be shown. See also Graphical User Interface (GUI) / Trace view in Online Help

50. ©2008 Keynote SIGOS GmbH Testing is our Competence 50 SITE 2.3 Diagnostic possibilities Overview about SITE Trace information  Two important columns are:  PTC Test component where the event occurred.  Event Mnemonic abbreviation describing the event which caused the trace record.  Examples of Test case events (T??): The TSC (Start Test Case) and the TEC (End Test Case)  Examples for Message events (M??): These trace lines contain messages which are either sent (MSH) or received (MRH).  Examples for Parameter events (P??): The parameters are divided into three groups: Test Case Parameter (PTS), Sequence Parameter (PSS) and Resource Parameter, i.e. the values which are determined by the resource management (PRS).  Examples for Result events (R??): The type of the result is indicated by the event tag: Measurement Integer (RMI), Measurement Float (RMF), Measurement String (RMS), Measurement URL (RMU) and Measurement Timestamp (RMT).

51. ©2008 Keynote SIGOS GmbH Testing is our Competence 51 SITE 2.3 Diagnostic possibilities SITE Layer3 Tracing: NAS  On most Qualcomm-based 3G probes, SITE is able to record traces of the Layer 3 Non-Access Stratum (NAS) signaling messages exchanged between the mobile station (MT, TE, USIM) and the core network (CN) over the UMTS Uu air interface.  The primary goal of this tracing facility is to get a deeper insight into the message flow and the protocol data exchanged between mobile station and network. SITE basically interacts with the 3G system under test via AT commands and responses at the R reference point.

52. ©2008 Keynote SIGOS GmbH Testing is our Competence 52 SITE 2.3 Diagnostic possibilities  An important example of diagnostic information which cannot be retrieved from the AT interface is reject causes of various kinds (see Figure 1).  The trace which would only base on AT commands gives the trace information Error=PDP_ContextActivation Could not start pppd…. You cannot get deeper information.  Now the trace record of the Layer 3 NAS signaling messages gives the deeper cause Requested service option not subscribed.  All trace information which starts with ↔ TRACE refers to Layer 3 messages.  Lets have a look at Test case ID 1164226 of Presales-System SITE Layer3 Tracing: Example 1 – reject causes

53. ©2008 Keynote SIGOS GmbH Testing is our Competence 53 SITE 2.3 Diagnostic possibilities Figure 1:SITE Layer 3 Trace – reject cause SITE Layer3 Tracing: Example 1 – reject causes

54. ©2008 Keynote SIGOS GmbH Testing is our Competence 54 SITE 2.3 Diagnostic possibilities ■ A second important example is the occurrence of request retries influencing certain KPIs depending on durations (e.g., for PDP context activation). Alone a second context retry request took about 30 seconds. SITE Layer3 Tracing: Example 2 – request retries

55. ©2008 Keynote SIGOS GmbH Testing is our Competence 55 SITE 2.3 Diagnostic possibilities The remaining layer 3 protocol on the Uu interface is Radio Resource Control (RRC) which belongs to the access stratum (AS) as it ends in the UTRAN. In the protocol stack, RRC lies below the NAS protocols. Many aspects of this protocol are mainly of local, and transient, relevance as opposed to giving insight into the network behaviour, while certain pieces of information are indeed valuable in the latter respect. Example: SITE Layer3 Tracing: Radio Resource Control (RRC)

56. ©2008 Keynote SIGOS GmbH Testing is our Competence 56 SITE 2.3 Diagnostic possibilities See also White Papers (PDF) / Result files in Online Help ■ As the functionalities for multi-media testing in SITE grow sheer verdicts and measurements are not as sufficient as test results. During tests for MMS, video streaming, fax, IVR etc. result files are created within SITE which are of interest for the user. ■ The result files are stored on the Local Units, but will be shown as URL sensitive text widgets in the SITE Trace. For the user a clickable URL is the activation method of viewing a result file. Overview about SITE Result files

57. ©2008 Keynote SIGOS GmbH Testing is our Competence 57 SITE 2.3 Diagnostic possibilities Example 1 – Screenshot of Test case HTTP Browser  It might be important to check the Screenshot of Test case HTTP Browser, if it is running “FAIL” with error message: Content of downloaded page for given URL does not match expected content pattern: URL='http://www.keynote-sigos.com' pattern='Trainer' at PTC_A in status Mainhttp://www.keynote-sigos.com  Lets have a look at Test case ID 1178016 of Presales-System  The parameterized keyword “Trainer” doesn’t appear in the Screenshot, but the Testcase runs “PASS”, if the keyword is e.g. “Newsletter”. SITE Result Files

58. ©2008 Keynote SIGOS GmbH Testing is our Competence 58 SITE 2.3 Diagnostic possibilities Example 2 – Tcpdump of Test case FTP_dl_ps  It might be important to check the Tcpdump-file of Test case FTP_dl_ps, if e.g. the mean download rate is very low or if other KPIs are out of the normal range.  The most common software to have a look at Tcpdump files is Wireshark. Please download and install at your client-PC, if you like to work with Tcpdump- files: http://www.wireshark.org/  After wireshark-installation and mouseclick on the corresponding URL sensitive text widget in the SITE Trace (Result files), select Wireshark to open the file. SITE Result Files

59. ©2008 Keynote SIGOS GmbH Testing is our Competence 59 SITE 2.3 Diagnostic possibilities Example 2 – Tcpdump of Test case FTP_dl_ps  Lets have a look at Test case ID 1166343 of Presales-System  One reason of the low mean download rate are the Retransmission like shown in the Wireshark Capture File SITE Result Files

60. ©2008 Keynote SIGOS GmbH Testing is our Competence 60 Final Case Study / Exercise

61. ©2008 Keynote SIGOS GmbH Testing is our Competence Final Case Study Usage of all SITE / GlobalRoamer functionalities in order to realize a smart and clever test and report setup Easy to maintain and to modify No changes within Reporting needed after modifying test setup Example: GRQ test setup Sophisticated usage of SET structure 61

62. ©2008 Keynote SIGOS GmbH Testing is our Competence Outbound MO calls Outbound MT calls Outbound roaming calls Framework Definition – Test Specification Call term specification Call term specification

63. ©2008 Keynote SIGOS GmbH Testing is our Competence Results Define Report Setup only once Changes within test setup  No changes within report setup 63 Test Definition(s) Report Definition(s)

64. ©2008 Keynote SIGOS GmbH Testing is our Competence Test Setup 64 Pre- configured test cases are GPRS attach CS network registration Speech service FTP download (IP service) SMS service CLI verification Pre- configured test cases are GPRS attach CS network registration Speech service FTP download (IP service) SMS service CLI verification path/OPERATOR_B_SIDE_MCCMNC_vs _OPERATOR_A_SIDE_MCCMNC” Every operator gets his own folder -You can execute (schedule) all folders by using the main folder Or -You can execute one individual pair of operators by selecting the relevant folder In principle – the configuration of the tests are identical – the difference of the tests are done via the SET framework – ( the green subfolder for the tests) Every operator gets his own folder -You can execute (schedule) all folders by using the main folder Or -You can execute one individual pair of operators by selecting the relevant folder In principle – the configuration of the tests are identical – the difference of the tests are done via the SET framework – ( the green subfolder for the tests)

65. ©2008 Keynote SIGOS GmbH Testing is our Competence SET Setup Define SET structure 65

66. ©2008 Keynote SIGOS GmbH Testing is our Competence SET Setup Define SET structure 66

67. ©2008 Keynote SIGOS GmbH Testing is our Competence SET Setup Usage of VARIABLES within test definitions 67

68. ©2008 Keynote SIGOS GmbH Testing is our Competence SET Setup Usage of clever naming conventions 68 Test Definition Names Chart Filter according to test definition name

69. ©2008 Keynote SIGOS GmbH Testing is our Competence Report Setup Pre-Configuration of all scenarios 69 Success Ratio per Day / Week Test Results - MO Side Roaming Test Results - MT Side Roaming Test Results – MO & MT Side Roaming Performance Matrix (Detailed View)

70. ©2008 Keynote SIGOS GmbH Testing is our Competence Report Setup Set filter expressions for individual scenarios 70

71. ©2008 Keynote SIGOS GmbH Testing is our Competence Report Setup KPI Details for indiviadual charts 71 KPI Details Success Ratio KPI Details Performance Details

72. ©2008 Keynote SIGOS GmbH Testing is our Competence Reports 72

73. ©2008 Keynote SIGOS GmbH Testing is our Competence Adding New Test Scenarios Adding of new test scenarios easy and flexible No changes within report settings necessary 73 Copy, Paste & Rename new Scenario Adapt SET VARIABLES

74. ©2008 Keynote SIGOS GmbH Testing is our Competence Automatic Reporting Report charts are being adapted automatically 74 Automatic Report Adaptation

75. ©2008 Keynote SIGOS GmbH Testing is our Competence 75 Thank you