1. A Systematic Mapping Study on Software Engineering Testbeds Emanoel Barreiros Advisor: Sérgio Soares

2. System Solution Software + Hardware + Documentation (run, transform...) Software + Hardware + Documentation (run, transform...) 2

3. Software Developed ≠ Produced Manufacturing Human Creativity Manufacturing Human Creativity 3

4. Software Engineering Software runs on the real world Empirical Studies better evaluate predict understand control improve 4

5. What do we need? Guidelines Experience reports Replications Comparisons Systematic evaluation Guidelines Experience reports Replications Comparisons Systematic evaluation 5

6. Testbeds Systematic evaluation of technologies Holibaugh et al. (1988) “Phase I Testbed Description: Requirements and Selection Guidelines” 6

7. ? ? Research Problem How can testbeds help researchers in evaluating technologies in SE? What should be considered in order to build a new testbed in SE? How can testbeds help researchers in evaluating technologies in SE? What should be considered in order to build a new testbed in SE? 7

8. Objectives Systematic mapping study Facilitate creation of testbeds in SE Foster empirical studies Systematic mapping study Facilitate creation of testbeds in SE Foster empirical studies 8

9. BACKGROUND Experimentation in SE, Testbeds in SE and Evidence-Based Software Engineering

10. Empirical Studies in SE Software Engineering Human Interaction Social Sciences 10

11. How? Near real scenarios Risky! Near real scenarios Risky! 11

12. Empirical Studies Systematic Disciplined Quantifiable Controlled Systematic Disciplined Quantifiable Controlled 12

13. Difficulties Definition and encapsulation Control of variables Human involvement High cost Definition and encapsulation Control of variables Human involvement High cost 13

14. One Experimental Framework 1.Definition: motivation, object, purpose, perspective, domain, scope; 2.Planning: design, criteria, measurement; 3.Operation: preparation, execution, analysis; 4.Interpretation: context, extrapolation, impact; 5.Packaging 1.Definition: motivation, object, purpose, perspective, domain, scope; 2.Planning: design, criteria, measurement; 3.Operation: preparation, execution, analysis; 4.Interpretation: context, extrapolation, impact; 5.Packaging Basili et al. (1986) “Experimentation in software engineering” 14

15. Why Testbeds in SE? Maturation may take 15+ years Systematic studies  Replicable, example Lower risks Easier knowledge transfer Maturation may take 15+ years Systematic studies  Replicable, example Lower risks Easier knowledge transfer 15

16. SE Testbeds Usually Have Assets repository Guidelines Mestrics suite Knowledge base Applications Assets repository Guidelines Mestrics suite Knowledge base Applications 16

17. SE Testbed Example Mikael Lidvall et al. (2007) “Experimenting with software testbeds for evaluating new technologies” – TSAFE tool – Soft. Architecture Evaluation – Seeded defects – Requirements specs., architecture docs., source code, installation guide, flight data 17

18. Evidence-Based Software Engineering (EBSE) Medical sciences Stand on the shoulder of giants SLRs and SMSs Medical sciences Stand on the shoulder of giants SLRs and SMSs 18

19. A”A” A”A” “ “ EBSE Goal To provide the means by which current best evidence from research can be integrated with practical experience and human values in the decision making process regarding the development and maintenance of software. Kitchenham et al. (2004). “Evidence-based software engineering” 19

20. EBSE Advantages Reduced bias Wide range conclusions Quantitative studies can detect real effects Reduced bias Wide range conclusions Quantitative studies can detect real effects 20

21. SMSs ≠ SLRs SMSs have broader questions Search terms are less focused Data extraction is broader Summaries rather than deep analyses More limited than SLRs SMSs have broader questions Search terms are less focused Data extraction is broader Summaries rather than deep analyses More limited than SLRs 21

22. METHODOLOGY Classification, Research Steps and Protocol

23. Methodology 23

24. Classification According to Cooper 24 Harris M. Cooper (1988). “Organizing Knowledge Syntheses: A Taxonomy of Literature Reviews”

25. Research Steps 25

26. Selection/Evaluation Process 26

27. Scope Population: Published literature that report the definition of software engineering testbeds; Intervention: Empirical studies that defines a SE testbed; Comparison: N/A; Outcomes: Quantity and type of evidence regarding the empirical evaluation of software engineering technologies; Context: Empirical studies on SE are run mainly in academic environments, which characterizes the current mapping study as a being of academic context. 27

28. Research Questions RQ1: Which specific software engineering fields are investigated using testbeds? RQ2: Which elements the studies define for a testbed in software engineering? RQ3: Which are the benefits of having a software engineering testbed? RQ4: Which methods are used to evaluate the proposed testbed? RQ1: Which specific software engineering fields are investigated using testbeds? RQ2: Which elements the studies define for a testbed in software engineering? RQ3: Which are the benefits of having a software engineering testbed? RQ4: Which methods are used to evaluate the proposed testbed? 28

29. Automated Search IEEE Computer Society Digital Library ACM Digital Library Scopus Citeseer Springer Link IEEE Computer Society Digital Library ACM Digital Library Scopus Citeseer Springer Link 29

30. Manual Search (past 12 months) International Conference on Software Engineering (ICSE), 2010 Empirical Software Engineering and Measurement (ESEM), 2009 Evaluation and Assessment in Software Engineering (EASE), 2010 Empirical Software Engineering Journal (June 2009 to June 2010) International Conference on Software Engineering (ICSE), 2010 Empirical Software Engineering and Measurement (ESEM), 2009 Evaluation and Assessment in Software Engineering (EASE), 2010 Empirical Software Engineering Journal (June 2009 to June 2010) 30

31. Search String “software engineering” AND (testbed OR “family of experiments” OR “technology evaluation” OR “technology comparison” OR “technology transition” OR “software adoption” OR “software maturity” OR “replicability” OR “replicate” OR “replication” OR “experiment protocol” OR “study protocol” OR “experiment process” OR “experiment guidelines” OR “study process” OR comparability OR comparable OR compare) 31

32. Selection Criteria – Phase 1 Title, keywords and abstracts 2 researchers for each paper Remove out of scope papers Title, keywords and abstracts 2 researchers for each paper Remove out of scope papers 32

33. Selection Criteria – Phase 2 Full paper Real Not a duplicate Defines a testbed Related to SE Full paper Real Not a duplicate Defines a testbed Related to SE 33

34. Data Extraction (Primary Studies) Form A ID Source Year Title Author Institution Country Form C Evaluation Date Researcher Research Question 1 Research Question 2 Research Question 3 Research Question 4 Notes 34

35. Data Extraction (Excluded Studies) Form B – ID – Source – Year – Title – Author – Criteria used for exclusion 35

36. Quality Assessment 1. Does the paper define the concept of testbed? 2. Has the paper evaluated the testbed it proposes? 3. Does the testbed evaluation allows replication? 4. Does the study define a process for using the testbed? 5. Does the study point the benefits of having and/or using a testbed? 1. Does the paper define the concept of testbed? 2. Has the paper evaluated the testbed it proposes? 3. Does the testbed evaluation allows replication? 4. Does the study define a process for using the testbed? 5. Does the study point the benefits of having and/or using a testbed? 36

37. Grading Quality Yes: question fully answered = 1.0 point Partly: question is not fully answered = 0.5 point No: question is not answered = 0.0 point Yes: question fully answered = 1.0 point Partly: question is not fully answered = 0.5 point No: question is not answered = 0.0 point [0.0; 1.0] [1.5; 2.5] [3.0; 4.0] [4.5; 5.0] 37

38. EXECUTION AND RESULTS General Data, Answers to Research Questions, Analysis and Discussion

39. Accounting 39

40. Accounting 40

41. Authors 41

42. Studies Distribution 42

43. Quality Assessment 43

44. ? Research Question 1 Which specific software engineering fields are investigated using testbeds? 44

45. Research Question 1 45

46. Research Question 1 46

47. ? Research Question 2 Which elements the studies define for a testbed in software engineering? 47

48. Research Question 2 Step-Based Testbeds Clear ordered steps Easier to understand Guidelines Block-Based Testbeds Blocks of activities May present block relations Harder to understand Step-Based Testbeds vs Block-Based Testbeds New! 48

49. Example of Step-Based Testbed Mikael Lindvall et al. (2005) “An evolutionary testbed for software technology evaluation”: 1.Select application in relevant domain; 2.Select family of technologies; 3.Select one technology within the family; 4.Prepare artifacts necessary for experimentation; 5.Conduct study to create a testbed baseline; 6.Define faults to be seeded; 7.Design and conduct experiment; 8.Analyze and document results from experiment; 9.Improve testbed based on analysis; 10.Improve technology based on analysis; 11.Verify testbed usefulness; 49

50. Example of Block-Based Testbed Marcus Ciolkowski et al. (2002) “A family of experiments to investigate the influence of context on the effect of inspection techniques” 50

51. ? Research Question 3 Which are the benefits of having a software engineering testbed? 51

52. Research Question 3 20 benefits cited Most papers claim 2+ benefits 20 benefits cited Most papers claim 2+ benefits 52

53. Research Question 3 53

54. Most Cited Benefits (cited by 2+ papers) 54

55. ? Research Question 4 Which methods are used to evaluate the proposed testbed 55

56. Research Question 4 56

57. Research Question 4 57

58. Analysis and Discussion 61.54% of papers ≥ Good Testbeds are versatile (16 research topics) 61.54% of papers ≥ Good Testbeds are versatile (16 research topics) 58

59. Analysis and Discussion Controlled Experiment dominates 59

60. Analysis and Discussion Testbeds are very different from each other Half (10 of 20) of the benefits cited by 2+ papers Testbeds are very different from each other Half (10 of 20) of the benefits cited by 2+ papers 60

61. FINAL CONSIDERATIONS Threats to Validity, Future Work and Conclusions

62. Threats to Validity Search strategy Small number of papers Quality assessment Data extraction Search strategy Small number of papers Quality assessment Data extraction 62

63. Future Work AO software maintenance testbed Integration with the BF* Extension to corporate environment AO software maintenance testbed Integration with the BF* Extension to corporate environment * Marcelo Moura (2008) “Um benchmarking framework para avaliação da manutenibilidade de software orientado a aspectos” 63

64. Future Work (improve the mapping) Backward search More thorough quality assessment Backward search More thorough quality assessment 64

65. Conclusion 4239 papers analyzed 13 primary studies Protocol reported Scopus performed well 4239 papers analyzed 13 primary studies Protocol reported Scopus performed well 65

66. Conclusion Similar benefits Testbeds are not well explored in SE Testbed classification Similar benefits Testbeds are not well explored in SE Testbed classification 66

67. Acknowledgements 67 Friends Family CIn CNPQ Friends Family CIn CNPQ

68. THANK YOU! 68