1. Quality Service Assurance: Monitoring, Testing, and Metrics Eugene G. Yu (gyu@gmu.edu)gyu@gmu.edu Lingjun Kang (lkang3@masonlive.gmu.edu)lkang3@masonlive.gmu.edu Archie Warnock (warnock@awcubed.com)warnock@awcubed.com

2. Outline Monitoring – Service availability Testing – Integration – Status testing – Routine testing Metrics – Integrated Troubleshooting – Internal view and query for details (to track down problems)

3. Monitoring – current status Purposes – Service availability Current status – Uptime robot What: connector endpoints, CWIC service endpoints Method: uptime robot Frequency: Hourly – GetCapability What: GetCapability, OSDD, Xml validity (content-level) Method: python script (Archie) Frequency: every 4 hours

4. Monitoring – requirements & future Requirements – To visualize, manage, and subscribe to the service availability status online Proposals – Online service availability portal/page Functions – Monitor and archive the service availability status and statistics – Allow browsing and reviewing current status and archives online – Allow subscription for notification through email/sms Method – Manage the status in database for every check – Determine the frequency of status check – Web services for notification subscription – Visual integration as a separate page in the Metrics portal

5. Testing – current status Purposes – Assure integrity of software at different levels (unit, integration, system, regression, content) Current status – Unit testing – Integration testing – System testing – Regressional testing – Routine functional testing

6. Testing – unit testing What – Testing the functionality of modules. How – JUNIT GetRecordByID (35) GetRecords(80) Dataset (number of datasets) Wrapper (3) Who – Connector developer When – During development

7. Testing – integration testing What – Testing the functionality of integration into mediator. How – JUNIT – Scripting Who – Mediator integrator/developer When – Integration of connector into mediator

8. Testing – system testing What – Testing the functionality of the system as a whole. How – Script – Manual access (test pages, test clients) Who – Developer team – Testers? – producers, users, clients (CWICSmart Test instance, CWICStart test instance) When – After system deployed into DEV and/or TEST servers

9. Testing – regression testing What – Testing the proper functionality of the system after a change is applied How – Script – Manual access (test pages, test clients) – Implicit monitoring: check validity of xml (in monitoring) Who – Developers When – After a change is applied and the program is deployed into TEST server before being incorporated into the operational service

10. Testing – routine functional monitoring What – Testing the proper functionality of the system in operation (part of the routine monitoring) How – Script Who – Operational support When – Repetitively testing in a pre-determined frequency

11. Testing – requirements & future Requirements – Clear specifications for each type of testing – Automate the testing Proposals – Specifications Provide guidelines for each type of specifications to clarify what, how, who, and when to run the testing Record and archive the testing records to keep track on all the info – what, how, who, when – Automation Automated tools/cases/scripts Frequency to run the automated test Record the testing – what, how, who, and when in a database for diagnosis Password-protected Web-based archive of tests and status to help tracing the problem

12. Metrics – current status Purposes – Summary of usages Current status – Updates Integrated metrics Remote search time – Outlier screening longer than usual: >500 seconds mostly likely caused by power outage or anything beyond control of computing service

13. Metrics – requirements & future Requirements – Detecting and filtering outliers beyond computing service – Visualizing and browsing archives of status Proposals – Service status page Browsing archives of status Displaying current status Subscribing to alerts (abnormal detection and alerting) – Outlier detection and filtering Outlier detection Criteria to filter out

14. Troubleshooting – current status Purposes – Tracking down details Current status – Debugging Searching the logging database manually Examining logging files individually Ad-hoc approach – Document for troubleshooting guide (?) – Email notification from uptime robot and python monitoring scripts

15. Troubleshooting – requirements & future Requirements – Browsing the logging in details Proposals – Secured service usage detail query Password-protected page for querying the logging database Visualize and report IP-specific usages Visualize and report date-specific usages Generic SQL-based query and reporting Integrated it as one protected page in the metrics portal