1. ISERN Survey & Benchmark 10 th anniversary meta-experiment project Session Chair, Stefan Biffl Marcus Ciolkowski, Forrest Shull, and Dieter Rombach 1.Strategy 2.Organization 3.Survey Design 4.Benchmark Design 5.Next Steps contact/info: ciolkows@informatik.uni-kl.de

2. Meta-experiment Goal ISERN develops empirical methodology, i.e. how to use experiments (case studies, controlled experiments). 1992: What is an experiment? 2000: Many experiments conducted 2002: How to conduct a meta-experiment; Data Collection from the distributed individual surveys and benchmarks; Presentation of initial results, Meta-analyis with historic experiments

3. Research Problem: History ISERN has successful history on experiments with inspections, but mostly in academic environments –Variations in replications not systematically recorded: Make best use of existing information –Influence factors for inspection from an industry point of view should be elicited Methodic organization –How to acquire knowledge –Demonstrate added value

4. ISERN Inspection Survey & Benchmark Common empirical study in ISERN –One large survey study –Several options for inspection benchmarks –All interested ISERN-organizations collaborate Run a series of small but related studies –Each will provide interesting results for the host site –In the aggregate, a chance to compare the impact of various environmental variables on the results. –Also include historic experiments.

5. Benefit Common ISERN experiment –Pragmatic 1 st round – low risk context; reasonable return on investment (ROI) Low risk/cost for software inspection due to reuse –Experimental framework (baseline document, baseline systematic inspection technique) –Package with baseline inspection materials Reasonable ROI due to scalable investment –Allows organizations to commit to various levels of effort, based on their constraints

6. Opportunities through ISERN Survey and Benchmark Work with industry as well as universities Benefit for industry: –Learn: under which constraints are results (positive trends) transferable? –Study: Demonstrate benefit and gain knowledge not by chance, but based on testable hypotheses (study design). –QIP for experiments: Maximize benefits from experiments. –Industry can introduce own questions and own factors –External experience often basis for introduction of techniques –Measurement data important for regular feedback on and reinforcement of successful techniques in practice.

7. Organization: Roles in the Experiment

8. Organization: People for roles Steering Committee: contributors to experiment Experiment Preparation Committee: authors of experiment documentation Experiment Coordinator: knows experiment status Librarian: administers experiment documentation QA for Data Repository: checks incoming data Meta-analysis Committee: plans data analysis

9. Working Group Inspection Survey Goal: Overview on state-of-the-practice in inspections in industry Characterize documents and defects in practice Measure: How well does reading technique fit to document/defect profile/inspector skills? So far: studies/questionnaires from Lund, Sweden Working group chair: Ross Jeffery

10. Working Group Experiment Design Goal: Create experimental design and materials for distributed experiment Provide standard inspection techniques and baseline documents for inspection So far: three levels of commitment Working group chair: Forrest Shull, FC MD

11. ISERN Experiment Support Preparation –Baseline document and inspection material –Training material Conduct –Guidance through local researcher Data Analysis and Presentation –Data collection (guaranty of anonymity) –Evaluation and customized feedback to organization

12. Next Steps Negotiation and Agreement on Procedure Commitment to conduct studies Milestones for individual study –Contact to organization –Negotiate options –Conduct study –Data collection in ISERN repository Data meta-analysis (including historical data)

13. Questions & Open Issues Clear, can be discussed now: Unclear, should be discussed off-line: Working groups to resolve unclear issues into a set of clear options for participants.

14. Sign-up for participation Local researcher: ____________________ Contact: ___________________________ Max. number of partners...: –... for survey:___ –... for study of standard process:___ –... for semi-controlled experiment:___ Opportunities for collaboration (shadow experiments): _____________________