1. By Hyunsook Do, Sebastian Elbaum, Gregg Rothermel
Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact
By Hyunsook Do, Sebastian Elbaum, Gregg Rothermel
Presented by Yuyang Liu
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

2. Overview
There is seldom enough good empirical work appearing in top SE conference venues.
Goal is to help authors and reviewers of top SE venues improve this situation.
Provide an infrastructure, including testing artifacts and tools, to support controlled experimentation in software testing research.
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

3. Contents Introduction Survey of Software Testing Studies
Challenges for Experimentation
Infrastructure
Conclusion
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

4. Introduction (1/2) Motivation:
Testing, especially regression testing, take a significant portion of the costs of developing and maintaining software.
Research on software testing technique is helpful to cut down the cost and understanding factors that influence testing techniques.
To measure and compare the fault-revealing capabilities of test suites created by various testing methodologies
to measure the cost of executing the test suites created by different methodologies
To evaluate the influence of choices in test suite design on testing cost-effectiveness
Controlled Experimentation is an indispensable research methodology in creation, understanding, and assessment of software testing and regression testing techniques.
Controlled experiment: a controlled experiment is done when the investigator consciously changes one variable (the independent or manipulated variable), which will likely cause another variable to change (dependent or responding variable). To the extent possible, all other variables are kept the same (constants).
It focus on rigorous control of variables in an attempt to preserve internal validity and support conclusions about causality.
Regression testing
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

5. Introduction (2/2) Goal:
Obtaining testing objects and organizing them that supports controlled experimentation is a difficult task and expensive.
testing objects include:
software systems,
test suites,
fault data,
multiple versions of software systems (regression testing experimentation)
Goal:
Providing an infrastructure to support controlled experimentation in testing and regression testing techniques research.
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

6. Contents Introduction Survey of Software Testing Studies
Challenges for Experimentation
Infrastructure
Conclusion
if a study utilizes a single program or version and it has no control factor then it is clearly a case study or an example; if a study utilizes multiple programs and versions and it has multiple control factors then it is a controlled experiment.
If a study did not manipulate any factors, then be classified as a case study or an example.
On the other hand, if a study involved a single program but its experiment design manipulated and controlled factors such as versions or tests, we classified it as an experiment.
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

7. Survey of Testing Studies (1/3)
Survey in recently research papers in 6 top SE conference venues (2 journals and 4 conferences) : TSE, TOSEM, ISSTA, ICSE, FSE, ICSM, in the period of
Type of paper
amounts
Total papers
1995
Software testing
224
With empirical studies
107
Controlled experiment 37
TSE
TOSEM
ISSTA
ICSE
FSE
ICSM
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

8. Survey of Testing Studies (2/3)
Among 107 empirical papers, five factors are important to controlled experimentation on testing and regression testing.
among 24 papers, 21 use the same program
Multiple Programs: the number of programs used as sources of data.
Multiple Versions: the number of versions used as sources of data.
Test Suites: whether test suites were utilized.
Fault Data: whether fault data was utilized.
Shared Artifacts: whether the study involved artifacts provided by or made available to other researchers.
Number of programs used as sources of data.
The number of versions used as sources of data.
Whether test suites were utilized
Whether fault data was utilized
Whether the study involved artifacts provided by or made available to other researchers.
in the small number of controlled experiments, the small percentage
of studies utilizing multiple programs, versions, and faults data, and the limited artifact sharing evident.
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

9. Survey of Testing Studies (3/3)
--It’s lack of resources which are used for software testing researchers to do experiment.
Why?
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

10. Contents Introduction Survey of Software Testing Studies
Challenges for Experimentation
Infrastructure
Conclusion
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

11. Challenges for Experimentation (1/4)
Several challenges are faced by researchers who wish to perform controlled experimentation with testing techniques, and this have limited progress in testing area.
These challenges are:
Supporting replicability across experiments.
Supporting aggregation of findings.
Reducing the cost of controlled experiments.
Obtaining sample representativeness.
Isolating the effects of individual factors.
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

12. Challenges for Experimentation (2/4)
Challenge of “Supporting replicability across experiments”
Artifacts utilized by researchers are rarely homogeneous.
Artifacts are provided in widely varying levels of detail.
Experiment design and process details are often not standardized or reported in sufficient detail.
Challenge of “Supporting aggregation of findings”
Highly correlated with the replicability of an experiment, but can not be guaranteed by it.
Systematic capture of experimental context does not yet occur in the domain of testing experimentation.
1.A scientific finding is not trusted unless it can be independently replicated.
2.Individual experiments may produce interesting findings, but can claim only limited validity under different contexts. In contrast, a family of experiments following a similar operational framework can enable the aggregation of findings, leading to generalization of results and further theory development.
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

13. Challenges for Experimentation (3/4)
Challenges of “Reducing the cost of controlled experiments”
Manually collecting experiment data is very expensive.
For open source artifacts:
Artifact organization is not standardized.
For business artifacts:
Artifacts are “sanitized”, especially for fault data.
Artifacts require manual handling.
Challenges of “Obtaining sample representativeness”
Sample size is limited (for cost reason).
Samples are biased. Some attributes of collected programs are similar, such as size, faults seeded, code changes considered, and etc.
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

14. Challenges for Experimentation (4/4)
Challenges of “Isolating the effects of individual factors”
Artifacts may not offer the same opportunities for manipulation.
Artifacts may make it difficult to decouple factors.
Although these challenges, willingness of researchers to conduct controlled experiments, and utilization of shared artifacts are increasing.
Understanding causality relationships between factors lies at the core of experimentation.
Blocking and
manipulating the effects of a factor increases the power of an experiment to explain causality.
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

15. Contents Introduction Survey of Software Testing Studies
Challenges for Experimentation
Infrastructure
Conclusion
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

16. Infrastructure
Artifacts: objects which are used in controlled experiments in software testing, like source code, test suites, versions, and so on.
Provide guidelines to select, organize, and setup artifacts.
Provide automatic tools to check test techniques, from test case to finding faults
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

17. Infrastructure—Overview
Assist in the construction of a sample of complete artifacts.
Object Selection
Object Organization
Object Setup
Provide a consistent context for all artifacts, facilitating the development of generic experiment tools, and
reducing the experimentation overhead for researchers.
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

18. Infrastructure—Object Selection
Object selection guidelines direct persons assembling infrastructure in the task of selecting suitable objects.
Objects here means “programs” or “projects”
Two levels of required qualities for objects:
1st-tier required-qualities
Minimum lines of code required
Source freely available
Five or more versions available
2nd-tier required-qualities
Runs on platforms we utilize
Can be built from source
Allows automation of test input application and output validation
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

19. Infrastructure—Object Organization (1/2)
Object Organization provide a guideline to organize objects and associated artifacts into a directory structure that supports experimentation.
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

20. Infrastructure—Organization (2/2)
source—Temporarily place. During experiments, a version of a program that we're working with is placed in here, then deleted after we're finished with it.
versions.alt—Contains “variants” of the source for building program versions
versions.orig— the basic variant, that any object will have.
versions.seeded—in which faults are provided.
testplans.alt—Contains each version of testing information for objects.
general files: *.tsl, *.frame, *.universe
link files: files that links to a previous version of general files.
*.tsl—TSL(Testing Specification Language) specification files.
*.frame—sets of test frames generated from .tsl.
*.tsl.universe/*.cov.universe—STImpL files, which list test cases.
STImpL (Simple Test Implementation Language)
info—other information about the program, especially information gathered by analysis tools and requiring saving for experiments.
e.g. fault matrix file
Variants are needed when, say, we have to modify the code to accomodate some tool. The basic variant, that any object will have, is contained in the subdirectory of versions.alt called:
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

21. Infrastructure—Object Setup
1. Construct Test Suite
setup test suite by TSL tool
input *.tsl
output *.frame
setup cover-based test suite
2. Fault
activate hand-seeded faults
5/23/2019
Supporting Controlled Experimentation with Testing Techniques

22. Conclusion
This infrastructure and its artifacts have been widely used, since 1999.
Evaluating improved test suites by comparing provided test suites with code-coverage test suites. [By Harder et al. 2003]
Investigate whether automatically generated faults mutants can be used to assess the fault detection effectiveness of testing techniques. [By Andrews et al ]
Measure the effectiveness of prioritization techniques in a Junit testing environment. [By Do et al. 2004]
Directly collaborate with other researchers to add more artifacts into the infrastructure.
5/23/2019
Supporting Controlled Experimentation with Testing Techniques