1. N ORTH D AKOTA S TATE U NIVERSITY D EPARTMENT OF C OMPUTER S CIENCE © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Engineering Self-Testable Autonomic Software Presenter: T ariq M. King EASe 2011 April 27-29, Las Vegas, NV, USA A COMPARATIVE CASE STUDY

2. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Outline  Motivation  Background  Applications  Experiments  Lessons Learned  Related Work  Conclusion 2

3. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Motivation  Self-* features in autonomic computing systems may incorporate dynamic software adaptation  System adds, removes, replaces its own components @ runtime  Raises reliability concerns as new faults may be introduced due to dynamic adaptation  Most research neglects that self-test should be an integral part of these types of systems 3 A NEGLECTED SELF-* CHARACTERISTIC Self- Configure Self- Optimize Self- Heal Self- Protect Self- Test FAULTS? FAILURES?

4. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  King (2009) describes an approach that seeks to seamlessly integrate runtime testing into the workflow of Autonomic Managers (AMs)  Idea: Instantiate new AMs that are tailored for testing activities  Test Managers (TMs) monitor, intercept & validate the change requests generated by AMs to realize AST Managed Resource Background 4 AUTONOMIC SELF-TESTING (AST) Monitor AnalyzePlan Execute Knowledge Effector Sensor Autonomic Manager Monitor AnalyzePlan Execute Knowledge Autonomic Manager Test Manager

5. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Under AST, test managers can operate according to two different validation strategies:  Replication with Validation (RV) – tests changes using separate copies of managed resources  Safe Adaptation with Validation (SAV) – tests changes in-place, directly on managed resources  Supported by an OO framework for implementing prototypes of self-testable autonomic software  Java, Reflection, Threads, XML-Based Policies  Interfaces with Automated Software Testing Tools Background 5 AST STRATEGIES & TOOL SUPPORT

6. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  During preliminary investigations, we sought to locate freely available autonomic software that could be used to demonstrate our ideas on AST  When no such applications could be found, focus shifted to development through our own avenues:  REU on Autonomic Computing http://www.cis.fiu.edu/reu/  CVM Research Project http://cvm.cs.fiu.edu/ Applications 6 PRELUDE

7. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Stevens (2007) developed a self-configuring data structure to demonstrate ideas on AST Applications 7 AUTONOMIC CONTAINER ACT Lightweight Structural Adaptations RV Strategy

8. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Applications 8 AUTONOMIC JOB SCHEDULER

9. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Applications 9 AUTONOMIC JOB SCHEDULER Structural & Behavioral Adaptations AJS Lightweight (More Complex than ACT) RV Strategy

10. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  CVM is a model-driven platform for realizing communication services  Collaboration: FIU & Miami Children’s Hospital  Separates concerns of modeling, synthesis, coordination, and delivery of user communication  Users specify their communication requirements, which are represented as schemas  Schemas are then transformed into executable control scripts, which are used to configure the underlying network devices and protocols Applications 10 COMMUNICATION VIRTUAL MACHINE

11. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Allen (2009) leverage AC and open-platform APIs for integrating communication services into CVM Applications 11 COMMUNICATION VIRTUAL MACHINE CVM Real-World Behavioral Adaptations SAV Strategy NCB LAYER

12. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Conducted three sets of experiments as part of the case study:  Development Experiments – compared the relative effort required to build the autonomic and self-test infrastructures, and various prototypes  Performance Experiments – compared the runtime performance of different versions and variants of the prototypes  Test Quality Experiments – compared the effectiveness of test sets in revealing faults and exercising the program code of the prototypes Experiments 12 OVERVIEW

13. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Experiments were performed on windows-based Intel® Core 2 Duo 2.4GHz PCs with 2–4 GB RAM  Several tools were used to support automatically collecting results:  Eclipse Metrics – source code size and other complexity metrics for development experiments  Eclipse TPTP – thread and memory analysis for performance experiments  JUnit, Cobertura, MuJava – test failures, code coverage, and mutation analysis for test experiments Experiments 13 SETUP ENVIRONMENT

14. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Experiments 14 DEVELOPMENT COMPARISON OF AUTONOMIC & TEST INFRASTRUCTURES

15. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Experiments 15 DEVELOPMENT (CONT.) COMPARISON OF PROTOTYPE DEVELOPMENT

16. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Non-Self-Test vs. High Transparency Self-Test  Used Java RMI to develop a self-test version of AJS with a distributed RV self-testing process  Performed identical sets of user actions on both systems and compared memory and thread usage  Results: Less than 1% difference  Naive vs. “Enhanced” Interleaving Self-Test  Introduced timeout into TM threads of CVM  Purpose: Seek to achieve better interleaving in favor of core system over self-test during SAV self-testing Experiments 16 PERFORMANCE

17. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Experiments 17 PERFORMANCE (CONT.) SKYPE TWO-WAY TO SMACK THREE-WAY COMPARISON OF SELF-TEST VARIANTS CVM

18. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Experiments 18 TEST QUALITY MUTATION SETUP

19. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Experiments 19 TEST QUALITY (CONT.) MUTATION RESULTS CODE COVERAGE RESULTS

20. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  RV can be used to provide a transparent self- testing process, while SAV requires optimizations.  Using TMs as a test harness was convenient but problematic: Complexity, Synchronization  Safety mechanisms (locks) were instrumental  Selecting and tailoring open-source testing tools presented unnecessary difficulties  Benefits: Prevention of Runtime Faults Testing Aligned with Development Lessons Learned 20

21. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  “Testing and assurance are probably the least focused phases in engineering self-adaptive software…” – Salehie (2009), ACM Trans…  King et al. (2007), and Zhang et al. (2007)  Da Costa et al. (2010)  JAAF+T: Java Self-Adaptive Framework + Test  Modify MAPE structure to include a self-test activity  Munoz & Baudry (2009)  Artificial Table Testing of Adaptive Software Related Work 21

22. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Presented a case study on the engineering of autonomic software with self-testing features  Reached a level of “enablement”, still many open research issues:  Dynamic regression test case selection and scheduling based on runtime testing goals  Runtime test case maintenance  Reducing overhead of runtime testing  Current Directions: Cloud, Virtualized RV. Conclusion 22

23. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Acknowledgements 23  Dr. Masoud Milani, FIU  Dr. S. M. Sadjadi, FIU  Participants of the REU Summer 2006 and 2007 Programs on Autonomic Computing at FIU  Students and collaborators on CVM project This work has been supported in part by the NSF under grants IIS-0552555, and HRD-0833093

24. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Thank You! 24 Questions? ¿Preguntas? 問題 Sawwal вопросы 質問 domande ερωτήσεις