1. Quality and applicability of automated repair
Yuriy Brun, UMass Amherst
Ted Smith Yalin Ke Manish Motwani Mauricio Soto
Earl Barr Prem Devanbu Claire Le Goues René Just Katie Stolee

2. Quality of the patches produced by automated repair
Applicability of automated repair to important and hard defects

3. What does it mean for automated repair to be successful?
How many bugs can it fix?
that is, make all known test cases pass
But is making test cases pass the ultimate success?
No! It conflates training and evaluation data!
notable exceptions:
Fry et al. ISSTA’12
Kim et al. ICSE’13
Qi et al. ISSTA’15
Martinez et al. EMSE’16
Don’t get me wrong:
Tweaking code to pass all tests is really cool!
It’s just not the end-all of automated repair

4. Lingering questions Are the produced repairs any good?
How can we tell quality objectively?

5. How can we know if APR repairs
Look at the produced patches by hand [Qi, Long, Achour, Rinard ISSTA’15] [Martinez, Durieux, Sommerard, Xuan, Monperrus ESEM’15]
Have others look at the produced patches by hand [ Fry, Landau, Weimer ISSTA’12] [Kim, Nam, Song, Kim ICSE’13]
Produce patches with test suite T, evaluate them on independent test suite T' [Brun, Barr, Xiao, Le Goues, Devanbu 2013] [Smith, Barr, Le Goues, Brun ESEC/FSE 2015]
objective
repeatable
Zichao, Fan, Sara, Martin
MartAn’s team

6. IntroClass Benchmark Requires a large set of bugs
for programs with 2 independent test suites
and the test suites need to be good
IntroClass: 998 bugs in very small, student-written C programs, with a KLEE-generated test suite, and a human-written test suite.
[Le Goues, Holtschulte, Smith, Brun, Devanbu, Forrest, Weimer TSE’15]
450 unique bugs

7. Cobra effect
The British government was concerned about the number of venomous cobra snakes in Delhi. They put a bounty on the cobra heads. Business-minded individuals began breeding cobras to harvest them for the heads. The British government cancelled the program and the breeders released all their snakes, greatly increasing the population.

8. Do GenProg and TrpAutoRepair patches pass kept-out tests?
Overfitting is common!
Repair quality is not to be taken for granted.

9. Repair quality distribution
GenProg TrpAutoRepair

10. More GenProg and TrpAutoRepair findings
The better the test suite coverage, the better the patch
APR causes harm to high-quality programs, but is helpful for low-quality programs
Human-written tests lead to better patches
More answers and details in “Is the Cure Worse Than the Disease? Overfitting in Automated Program Repair” by Smith, Barr, Le Goues, Brun ESEC/FSE 2015

11. Why study patch quality?
To build repair techniques that produce high-quality repairs!

12. Can we improve the patch quality?
Recent work:
SearchRepair [Ke, Stolee, Le Goues, Brun ASE’15]
SPR [Long and Rinard ESEC/FSE’15]
Prophet [Long and Rinard POPL’16, ICSE’16]
SearchRepair, SPR, Prophet produce higher quality patches than GenProg, TrpAutoRepair, AE

13. Applicability What kinds of defects can automated repair repair?
Can it repair defects hard for humans to fix?
Can it repair defects humans think are important?
Can is synthesize feature requests?

14. Defect sets
ManyBugs [Le Goues, Holtschulte, Smith, Brun, Devanbu, Forrest, Weimer TSE’15]
Defects4J [Just, Jalali, Ernst ISSTA’14]

15. How do you measure importance?
Importance: priority, time from report to fix, project versions affected, etc.
Complexity: size of the developer-written patch
Test effectiveness: relevant tests, triggering tests, coverage, etc.
Independence: does the defect depend on other defects?
Developed a methodology for measuring a defect’s importance, complexity, test-suite effectiveness, and independence using:
bug trackers,
version control history, and
human-written patches.

16. Seven repair techniques
Java:
GenProg
GenProgJ
TrpAutoRepair
KaliJ AE
Nopol
Kali
SPR
Prophet
Our focus was measuring defect properties, so we used prior results for which defects can be repaired.

17. Importance, complexity, tests
APR is more likely to patch defects of a higher priority. (surprising!)
Human time to fix didn’t correlate with APR’s ability to patch. (encouraging!)
APR is less likely to repair defects for which human patches were large, but fixed some hard defects.
More tests can reduce ability to produce patches. (concerning!)

18. Quality, patch attributes
APR1 is less likely to produce correct patches for more complex defects.
but the effect is smaller for producing high-quality patches than low-quality ones. (surprising and encouraging!)
Higher-coverage test suites resulted in higher-quality patches.
APR struggled with inserting loops, new variables, new function calls, and changing signatures.
1Only Prophet and SPR produce a sufficient number of high-quality patches to evaluate.

19. Contributions Repair Quality
Research needed focusing on repair quality.
Repeatable, automated, objective methodology for evaluating automated repair quality.
including the IntroClass dataset
Repair Applicability
Defects humans and automated repair find difficult are different.
Methodology and dataset for measuring repair applicability to hard and important defects.