1. Testing, Monitoring, and Control of Internet Services Aditya P. Mathur Purdue University Friday, April 15, 2000 @ Washington State University, Pullman, WA. Last updated: April 12, 2000

2. Testing and Management of Internet Services 2 Summary [1] 4 Belief –There is a growing need for testing and management of Internet Services. 4 Observations –Use of the coverage principle and a knowledge of the saturation effect allows us to design a controlled and an effective process for testing Internet Services. –A component based distributed architecture is needed for the test and management tool.

3. Testing and Management of Internet Services 3 Summary [2] 4 Conclusions –A uniform test and management process is feasible despite the variety in services and the underlying infrastructure. –Interface Testing is a practical option when testing individual components or complete applications. However…. –More research is needed in Interface Mutation and Dynamic Testing.

4. Testing and Management of Internet Services 4 Issues 4 Testing –Methodology and techniques 4 Management –Uniform interface to a heterogeneous environment –Generic event and control specification and processing mechanisms –Variable overhead mechanisms

5. Testing and Management of Internet Services 5 Client/Server Stub/Skeleton Request/data Client/Server Stub/Skeleton Request/data Component ORB Communication. Structure of an Internet Service Component ORB: Object Request Broker

6. Testing and Management of Internet Services 6 Testing: Objectives 4 Adequacy assessment and test enhancement 4 In-use test and debug (dynamic testing) 4 Load test 4 Evaluation of fault tolerance

7. Testing and Management of Internet Services 7 The Coverage Principle Measuring test adequacy and improving a test set against a sequence of well defined, increasingly strong, coverage domains leads to improved reliability of the system under test.

8. Testing and Management of Internet Services 8 Coverage Domains and Elements Requirements Classes Functions Mutations Exceptions Data-flows Coverage Domains Coverage Elements

9. Testing and Management of Internet Services 9 Sample Hierarchy Requirements coverage Function/method coverage Statement coverage Decision coverage Data-flow coverage Mutation coverage Strength Low High Cost Low High

10. Testing and Management of Internet Services 10 Saturation Effect: Reliability View FUNCTIONAL, DECISION, DATAFLOW AND MUTATION TESTING PROVIDE TEST ADEQUACY CRITERIA. Reliability Testing Effort True reliability (R) Estimated reliability (R’) Saturation region Mutation Dataflow Decision Functional RmRm R df RdRd RfRf R’ f R’ d R’ df R’ m tfstfs tfetfe tdstds tdetde t df s t df e tmstms tfetfe

11. Testing and Management of Internet Services 11 Proposed Test Methodology 4 Test individual components using traditional black-box and white-box techniques 4 For COTS, use interface testing 4 Prior to or during deployment, test applications using Interface Testing

12. Testing and Management of Internet Services 12 Component Interface Methods: m 1, m 2, …,m k Exceptions: e 1, e 2, …,e k 100% Method Coverage # methods executed # methods defined 100% Exception Coverage # exceptions raised # exceptions defined 100% iMutation Score # distinguished mutants total # imutants - #equivalent imutants Interface Testing

13. Testing and Management of Internet Services 13 What is Interface Mutation ? Test Suite T contains Request-A. Client Server Interface Request-A Response-A Mutated Interface Client Server Request-A Response-B

14. Testing and Management of Internet Services 14 Experimental Evaluation 4 Select components –Number of components= 3 –Number of interfaces= 6 –Number of methods= 82 –Number of mutants= 220 4 Seed errors one by one in the components 4 Errors related to: –Portability, data, algorithm, language-specific

15. Testing and Management of Internet Services 15 Error Revealing Capability Comp 1Comp 2Comp 3Total 100% 0 20% 80% 60% 40% Using Mutation Adequacy Using Statement/Decision coverage

16. Testing and Management of Internet Services 16 Code Coverage Comp 1Comp 2Comp 3Total 100% 0 20% 80% 60% 40% Block coverage with Mutation Adequate Tests Decision coverage with Mutation Adequate tests

17. Testing and Management of Internet Services 17 Tests Required Comp 1Comp 2Comp 3 35 0 5 30 15 10 For Mutation Adequacy For statement/decision coverage 20 25

18. Testing and Management of Internet Services 18 Results [1] 4 Interface mutation leads to fewer tests that reveal almost as many errors as revealed by statement and decision coverage. 4 Interface mutation is a scalable alternative to using code coverage.

19. Testing and Management of Internet Services 19 Results [2] 4 Reveals –programming errors in components. –errors in the use of component interfaces 4 Reveals certain types of deadlocks 4 1 2 5 3 Server callback Client Server Client Request

20. Testing and Management of Internet Services 20 Monitoring 4 Why? –Resource planning –Control –Testing and debugging 4 What? –State of individual or a group of components at different levels of abstraction.

21. Testing and Management of Internet Services 21 Control 4 Why? –Resource allocation –Prevention of unauthorized use –Testing and debugging 4 What? –Component behavior and location.

22. Testing and Management of Internet Services 22 Monitoring: Strategy UserEvent(s) specifies State MonitorApplication State monitors Event detectorUser specified events detects Event notification Waiting user notifies

23. Testing and Management of Internet Services 23 Control: Strategy UserControl action(s), trigger event(s) specifies Event notifierController signals ControllerControl action(s) executes

24. Testing and Management of Internet Services 24 Internet Service Zones

25. Testing and Management of Internet Services 25 LLI MCAR db Zonal Manager LLLL CS LOG Host 1 C Architecture of Wabash 2.0 [1] Wabash GUI Request Client sends a request to a managed object LL determines whether request can be passed or not If the request is allowed, LL forwards it to the CORBA Server after time-stamping it Response LL stores information about the request, records it in a log, sends the response back to client LL gets the response from the CORBA Server If the request is not allowed, LL throws exception and does not forward request to the CORBA Server GUI requests data from the Zonal Manager Zonal Manager requests data from corresponding LL LL returns data to Zonal Manager Zonal Manager returns data collected from LL to GUI

26. Testing and Management of Internet Services 26 LLI MCAR db Zonal Manager LL CS LOG Host 1 C Wabash GUI Request Architecture of Wabash 2.0 [2] Manager sends command (‘ANY’, DENY, OBJECT_A) Command is forwarded to corresponding LL for OBJECT_A Client sends request to OBJECT_A LL determines that request has to be denied LL does not forward request to CS Throws exception back to client

27. Testing and Management of Internet Services 27 Architecture of Wabash 3.0

28. Testing and Management of Internet Services 28 Dynamic Testing 4 Question: –How to test an Internet Service while it is in use? –Answer: Use the dynamic testing procedure. 4 What is the dynamic testing procedure?

29. Testing and Management of Internet Services 29 Faulty Server group Dynamic Testing Faulty server Test Client Ra 2 Client 1 Isolated server 3

30. Testing and Management of Internet Services 30 Limitations of Dynamic Testing 4 Test client might generate undesirable actions: –Persistent data modification. –Irreversible actions. 4 Application limited to: –Closed and well understood domains. –Simulated or isolated service environments.

31. Testing and Management of Internet Services 31 The Wabash Project: Goals and History 4 Goal: –Develop a methodology and a tool to test and manage large brokered distributed systems. 4 Sponsors: –NSF, SERC, British Telecom, Tivoli, and Telcordia. 4 Staff: –PI, 3 doctoral, 1 MS, and 3 UG students.

32. Testing and Management of Internet Services 32 The Wabash Project: Goals and History 4 Progress: August 1999 TDS 1.1 available to SERC affiliates. August 2000 Wabash 2.0 available to SERC affiliates. Experiments to assess goodness of proposed interface testing criteria completed. Inter-university collaboration established. December 2000 Uniform interface for Jini/JMX/CORBA objects.

33. Testing and Management of Internet Services 33 Ongoing Research 4 Non-intrusive procedures for dynamic testing. 4 Generalized event-control model and its implementation. 4 Implementation of the unified architecture to assist with the management of JMX, JINI, and CORBA objects. 4 Light-version of Wabash for SmartHome management.