1. Testing Internet Services Sudipto Ghosh Sambhrama Mundkur Aditya P. Mathur: PI Ramkumar Natarajan Baskar Sridharan Department of Computer Sciences Purdue University @ Design2Deploy Monday July 3, 2000 Last updated: June 28, 2000

2. Test and Management of Internet Services2 Areas of Concern [1] Test Methodology Adequacy assessment and test enhancement Load testing/Performance measurement Testing for fault tolerance In-use testing and debugging (dynamic testing)

3. Test and Management of Internet Services3 Areas of Concern [2] Management Uniform interface to a heterogeneous environment Generic event and control specification and processing mechanisms Variable overhead mechanisms

4. Test and Management of Internet Services4 Client/Server Stub/Skeleton Request/data Client/Server Stub/Skeleton Request/data Component ORB Communication. Structure of an Internet Service Component ORB: Object Request Broker

5. Test and Management of Internet Services5 Service Domain Client An Internet Service A component

6. Test and Management of Internet Services6 Areas of Concern Test Methodology Adequacy assessment and test enhancement Testing for fault tolerance Load testing/Performance measurement In-use testing and debugging (dynamic testing)

7. Test and Management of Internet Services7 Elements of a Proposed Test Methodology Test individual components using traditional black- box and white-box techniques. For COTS, use interface testing. Prior to deployment, test applications using Interface Testing. Test for compliance with requirements for FT. Load test an application prior to or during use. While in-use, test and debug an application using dynamic testing.

8. Test and Management of Internet Services8 Areas of Concern Test Methodology Adequacy assessment and test enhancement Testing for fault tolerance Load testing/Performance measurement In-use testing and debugging (dynamic testing)

9. Test and Management of Internet Services9 The Coverage Principle Measuring test adequacy and improving a test set against a sequence of well defined, increasingly strong, coverage domains leads to improved reliability of the system under test.

10. Test and Management of Internet Services10 Coverage Domains and Elements Requirements Classes Functions Mutations Exceptions Data-flows Coverage Domains Coverage Elements

11. Test and Management of Internet Services11 Sample Hierarchy Requirements coverage Function/method coverage Statement coverage Decision coverage Data-flow coverage Mutation coverage Strength Low High Cost Low High

12. Test and Management of Internet Services12 Saturation Effect: Reliability View FUNCTIONAL, DECISION, DATAFLOW AND MUTATION TESTING PROVIDE TEST ADEQUACY CRITERIA. Reliability Testing Effort True reliability (R) Estimated reliability (R’) Saturation region Mutation Dataflow Decision Functional RmRm R df RdRd RfRf R’ f R’ d R’ df R’ m tfstfs tfetfe tdstds tdetde t df s t df e tmstms tfetfe

13. Test and Management of Internet Services13 Component Interface Methods: m 1, m 2, …,m k Exceptions: e 1, e 2, …,e k 100% Method Coverage # methods executed # methods defined 100% Exception Coverage # exceptions raised # exceptions defined 100% iMutation Score # distinguished mutants total # imutants - #equivalent imutants Interface Testing

14. Test and Management of Internet Services14 What is Interface Mutation ? Test Suite T contains Request-A. Client Server Interface Request-A Response-A Mutated Interface Client Server Request-A Response-B

15. Test and Management of Internet Services15 Interface Mutation: Experimental Evaluation Select components Number of components= 3 Number of interfaces= 6 Number of methods= 82 Number of mutants= 220 Seed errors one by one in the components Errors related to: Portability, data, algorithm, language-specific

16. Test and Management of Internet Services16 Characteristics of applications APP 1 Component-3Component-2Component-1 LOC # Interfaces # Methods # Exceptions # Mutants 148412682950 222 271045 1 69 11 24127 0 APP 2 LOC # Interfaces # Methods # Exceptions # Mutants 189327995559 122 1179 15 34 613 Object Group Server Account Database Manager Activity Logger Client Component-1Component-3 Component-2 APP 1 Customer Group Service Manager Account Server Activity Logger Account Database Manager Administrator APP 2

17. Test and Management of Internet Services17 Control-flow coverage obtained 100% 0 20% 80% 60% 40% Component-1Component-2Component-3 APP 1 100% 0 20% 80% 60% 40% ActivityLoggerAccountDBManagerGroupServiceManager APP 2 Block Coverage with T ME Block Coverage with T IM Decision Cov- erage with T ME Decision Cov- erage with T IM Coverage

18. Test and Management of Internet Services18 100% 0 20% 80% 60% 40% 100% 0 20% 80% 60% 40% Number of errors revealed Component-1Component-2Component-3 APP 1 ActivityLoggerAccountDBManagerGroupServiceManager APP 2 Errors Using T ME Using T IM Using T CF

19. Test and Management of Internet Services19 Number of tests required 0 30 0 Component-1Component-2Component-3 APP 1 ActivityLoggerAccountDBManagerGroupServiceManager APP 2 Tests | T ME | | T IM | | T CF | 20 10 5

20. Test and Management of Internet Services20 Observations [1] Interface mutation leads to fewer tests that reveal almost as many errors as revealed by statement and decision coverage. Interface mutation is a scalable alternative to using code coverage.

21. Test and Management of Internet Services21 Observations [2] Reveals programming errors in components. errors in the use of component interfaces Reveals certain types of deadlocks 4 1 2 5 3 Server callback Client Server Client Request

22. Test and Management of Internet Services22 Areas of Concern Test Methodology Adequacy assessment and test enhancement Testing for fault tolerance Load testing/Performance measurement In-use testing and debugging (dynamic testing)

23. Test and Management of Internet Services23 Testing for fault tolerance Problem: Often error recovery code is not executed by test inputs How do we know if the fault recovery code adequately meets the requirements? Solution: Simulate the occurrence of faults by injecting them Fault injection testing at the interface Increases coverage of fault recovery code Reveals inadequacies in fault recovery code

24. Test and Management of Internet Services24 Interface fault injection m Client StubSkeleton Server ORB Fault Injected Simulate effects of problems in server, network and ORB

25. Test and Management of Internet Services25 Faults suitable for injection Time delays caused due to Network delays Server crashes or malfunctions Overloaded server Overflowing request buffer in ORB Unexpected exceptions received by client Server crashes Unexpected value User-defined exceptions raised by the server Implementation object invalid

26. Test and Management of Internet Services26 Areas of Concern Test Methodology Adequacy assessment and test enhancement Testing for fault tolerance Load testing/Performance measurements In-use testing and debugging (dynamic testing)

27. Test and Management of Internet Services27 Load Testing network C1 C2 Server On: Avg. Latency Effect of: Avg. Load Clients

28. Test and Management of Internet Services28 Server Statistics

29. Test and Management of Internet Services29 Client Statistics

30. Test and Management of Internet Services30 Low intrusion Selective on-line monitoring Selective on-line filtering To be applied in the Measurement of the latency and CPU consumption (and others) of individual methods for use in capacity planning Demonstration of performance improvement techniques Performance Instrumentation Goals

31. Test and Management of Internet Services31 Areas of Concern Test Methodology Adequacy assessment and test enhancement Testing for fault tolerance Load testing/Performance measurement In-use testing and debugging (dynamic testing)

32. Test and Management of Internet Services32 Dynamic Testing Question: How to test an Internet Service while it is in use? Answer: Use the dynamic testing procedure. What is the dynamic testing procedure?

33. Test and Management of Internet Services33 Dynamic Testing Faulty Server group Faulty server Test Client Ra 2 Client 1 Isolated server 3

34. Test and Management of Internet Services34 Limitations of Dynamic Testing Test client might generate undesirable actions: Persistent data modification. Irreversible actions. Application limited to: Closed and well understood domains. Simulated or isolated service environments.

35. Test and Management of Internet Services35 Areas of Concern [2] Management Uniform interface to a heterogeneous environment Generic event and control specification and processing mechanisms Variable overhead mechanisms

36. Test and Management of Internet Services36 Organization of the Service Domain Why organize ? Efficient and scalable management Personalized management Assignment of individual responsibilities

37. Test and Management of Internet Services37 Dimensions of Organization Geography Component Client

38. Test and Management of Internet Services38 Sample Organization

39. Test and Management of Internet Services39 Areas of Concern [2] Management Uniform interface to a heterogeneous environment Generic event and control specification and processing mechanisms Variable overhead mechanisms

40. Test and Management of Internet Services40 Monitoring Why? Resource planning Control Testing and debugging What? State of individual or a group of components at different levels of abstraction.

41. Test and Management of Internet Services41 Control Why? Resource allocation Prevention of unauthorized use Testing and debugging What? Component behavior and location.

42. Test and Management of Internet Services42 Monitoring: Strategy UserEvent(s) specifies State MonitorApplication State monitors Event detectorUser specified events detects Event notification Waiting user notifies

43. Test and Management of Internet Services43 Control: Strategy UserControl action(s), trigger event(s) specifies Event notifierController signals ControllerControl action(s) executes

44. Test and Management of Internet Services44 Areas of Concern [2] Management Uniform interface to a heterogeneous environment Generic event and control specification and processing mechanisms Variable overhead mechanisms

45. Test and Management of Internet Services45 Observer-listener architecture O1O1 O2O2 C1C1 O3O3 O4O4 C2C2 L1L1 M1M1 O5O5 O6O6 C3C3 O7O7 O8O8 C4C4 L2L2 M2M2 Zonal Manager GUI

46. Test and Management of Internet Services46 LLI MCAR db Zonal Manager LLLL CS LOG Host 1 C Architecture of Wabash 2.0 [1] Wabash GUI Request Client sends a request to a managed object LL determines whether request can be passed or not If the request is allowed, LL forwards it to the CORBA Server after time-stamping it Response LL stores information about the request, records it in a log, sends the response back to client LL gets the response from the CORBA Server If the request is not allowed, LL throws exception and does not forward request to the CORBA Server GUI requests data from the Zonal Manager Zonal Manager requests data from corresponding LL LL returns data to Zonal Manager Zonal Manager returns data collected from LL to GUI

47. Test and Management of Internet Services47 LLI MCAR db Zonal Manager LL CS LOG Host 1 C Wabash GUI Request Architecture of Wabash 2.0 [2] Manager sends command (‘ANY’, DENY, OBJECT_A) Command is forwarded to corresponding LL for OBJECT_A Client sends request to OBJECT_A LL determines that request has to be denied LL does not forward request to CS Throws exception back to client

48. Test and Management of Internet Services48 Architecture of Wabash 3.0

49. Test and Management of Internet Services49 Ongoing Research [1] Non-intrusive procedures for dynamic testing. Generalized event-control model and its implementation. Implementation of the unified architecture to assist with the management of JMX, JINI, and CORBA objects. Light-version of Wabash for SmartHome management.

50. Test and Management of Internet Services50 Ongoing Research [2] Automatic generation of test inputs. Test capture and replay.

51. Test and Management of Internet Services51 The Wabash Project: History Progress: August 1998: Wabash project launched. August 1999 TDS 1.1 available to SERC affiliates. August 2000 Wabash 2.0 available to SERC affiliates. Experiments to assess goodness of proposed interface testing criteria completed. December 2000 Uniform interface for Jini/JMX/CORBA objects.