1. CSE USC http://cebase.org

2. Fraunhofer USA Center for Experimental Software Engineering, Maryland February 2001 2 Empiricism in Software Engineering Empiricism: “An understanding of development methods based on real-world data will help software developers to produce “no surprise” software by choosing appropriate methods for producing software of the required quality and reliability.” Aims to: –strengthen available tools and methods by identifying current problems in practice as well as promising areas of research –provide validation for mature technologies –provide indications for the development of less mature technologies A commitment to advancing the field through empiricism is a fundamental part of CeBASE.

3. Fraunhofer USA Center for Experimental Software Engineering, Maryland February 2001 3 Experimentation (pre-CeBASE) A number of rigorous individual studies have been run and added to our knowledge of software engineering techniques –E.g. PBR, a procedural and specific inspection technique, found more defects in a study at NASA GSFC than less structured approaches. However, drawing general conclusions from empirical studies remains difficult. Hard to know if results will hold: –For developers at other levels of experience –For development teams that are composed differently –If used with different development methodologies –… Answering such questions requires: –Well-defined and interesting hypotheses. –Well-specified models of process, effectiveness, and context. –A community of researchers willing and able to replicate studies.

4. Example: Experimentation in Inspection Techniques for Inspections Reading Techniques Of Requirements Of OO Designs Perspective-Based Reading Study at NASA/ GSFC: Studies in UMCP classes: H: Procedural approaches to inspections are more effective than ad-hoc approaches. OO Reading Techs. (OORTs) H: Guidelines for effectively creating reading techniques exist. for Other Tasks... Studies in UMCP classes: H: PBR is more effective than ad- hoc reading. H: PBR is more effective than ad-hoc reading for various classes of users. H: OORTs are a feasible means of detecting real defects in OO designs.

5. Detailed Example: Experimentation to Refine OORTs

6. Fraunhofer USA Center for Experimental Software Engineering, Maryland February 2001 6 Current Efforts in CeBASE Tying together disparate studies to explore more hypotheses. –Obtain more generalizable results. For example, looking across a number of different studies in different contexts to understand general success factors influencing inspection effectiveness. –Obtain more specific results. Running studies in more contexts allows us to better answer industry’s questions. (E.g. Will a particular technique work in my environment?) Collecting materials already developed and making them available to –Grow the community –Increase the pace of experimentation Results so far: –Expanded to include new collaborators and begun replication of studies. –Building an experience base of materials and results. –Beginning larger scale analyses across studies.

7. Example: Experimentation in Inspection Techniques Of Requirements Perspective-Based Reading Study at NASA/ GSFC Studies in UMCP classes Studies at CPQD ISERN distributed experiment Studies at USC Lab Package: Reusable materials, experimental design, analyses... H: PBR is more effective than ad hoc... H: Cultural factors influence the effectiveness of PBR & other inspection techs. H: PBR can be effectively adapted to Fagan inspection methods. H: Factors across environments and techniques influence inspection effectiveness.

8. Example: Experimentation in Inspection Techniques Of OO Designs OO Reading Techs. (OORTs) Studies by ISERN collaborators: Studies in NTNU: Studies in UMCP classes Lab Package: Reusable materials, experimental design, analyses... H: OORTs are effective and feasible... H: Cultural factors influence the effectiveness of OORTs. H: Procedural inspection techniques are more effective than ad hoc for OO inspections.

9. Fraunhofer USA Center for Experimental Software Engineering, Maryland February 2001 9 Future goals and issues In contrast to previous work, CeBASE –is helping us move to a national as well as international focus –is facilitating work with industry as well as universities Through CeBASE we are reaching out to research communities other than software engineers. To achieve our goals we are exploring how to better facilitate replication. –How to present materials and results in a way that is usable to various communities? University collaborators are interested in undertaking replications. Industrial collaborators interested in contextual information of results.