1. University of Maryland Bug Driven Bug Finding Chadd Williams

2. University of Maryland 2 Motivation Finding bugs in software is important Statically checking code has been effective –finds complex errors –no need to run the code Many static checkers available –some with specific bug patterns to find –some allow the user to define the patterns –what kinds of bugs are really out there? Lots of false positive error reports –can we rank the errors better? –can previous bug history help? Where to start? –Bug reporting databases –CVS commit messages

3. University of Maryland 3 Bug Database Inspect fixed bugs –review bug discussions –tie fixed bug to source code change –classify the type of the bug –look for bugs that can be found statically UsersDevelopers Bug Database

4. University of Maryland 4 Bug Database: Practical Experience We inspected the Apache httpd bug database –inspected 200 bug reports marked as fixed –not as helpful as we expected Only 24% tied directly back to a source code change –bug reports include a discussion of the problem –rarely is a diff or a CVS revision noted Most are logic errors/feature requests –not the type found by static checkers

5. University of Maryland 5 Bug Database Bug Types Most classified bugs are logic errors

6. University of Maryland 6 Bug Database: Practical Experience Most bug reports originate from users –197 out of 200 –does not capture bugs found by developers Most bug reports came against a release of the software, not a CVS-HEAD –198 out of 200 –does not capture bugs between releases What about the bugs that don’t make it into the release? –they may be in the CVS repository…

7. University of Maryland 7 CVS Repository Commits may contain useful data –any bug fix must show up in a commit –will commit messages lead us to bug fixes? Shows bugs fixed between releases Bugs caught by developers –bugs that could be found by static checking 1.1 1.2 1.3 CVS Repository

8. University of Maryland 8 CVS Repository: Practical Experience Inspected commit messages –looked for ‘fix’, ‘bug’ or ‘crash’ –ignored those with bug number listed –looked at mature source files Commit messages are useful –trivially tied to source code change –less logic errors Common errors found –NULL pointer check –failing to check the return value of a function before use

9. University of Maryland 9 CVS Repository Bug Types NULL pointer bugs and return value bugs can be found by static analysis

10. University of Maryland 10 Return Value Check Bug Returning error code and valid data from a function is a common C idiom int foo(){ … if( error ){ return error_code; } …. return data; } … value = foo(); newPosition + = value; // ??? –the return value should be checked before being used –lint checks for this error Error types –completely ignored foo(); –return value used directly as an argument bar(foo()); –others …

11. University of Maryland 11 Return Value Checker Some functions don’t need their return value checked –no error value returned –could lead to many false positives Naively flagging all unchecked return values leads to many false positives –over 7,000 errors reported for the Apache httpd-2.0 source Need to determine which are most likely true errors –use historical data –present this data to the user

12. University of Maryland 12 Which return values need checked? Infer from historical data –look for an add of a check of a return value in a CVS commit –implies the programmer thinks it’s important Infer from current usage –does the return value of a function get checked in the current version of the software –how often? … value = foo(); newPosition + = value; // ??? … value = foo(); if( value != Error) { // Check newPosition + = value; } … Commit Bug Fix

13. University of Maryland 13 Our Tool Static checker that looks for return value check bugs –built on ROSE by Dan Quinlan, et al. Classify each error by category –ignored return value –return value used as argument, etc. Produce a ranking of the errors –group errors by called function –rank most promising errors higher rank functions that most likely need their return value checked higher

14. University of Maryland 14 Return Value Checker: Ranking Rank errors in two ways Split functions into two groups –functions flagged with a CVS bug fix commit at least one CVS commit adds a check of the function’s return value –functions not flagged with CVS bug fix commit Within each group: –rank by how often the function’s return value is checked in the current software distribution –checked more often means rank higher

15. University of Maryland 15 Case Study Apache httpd-2.0 on Linux –core system –modules –Apache Runtime Library Checked all the CVS commits for a return value check bug fix –6100 commits checked –2600 commits failed to go through our tool wrong (too new) version of autoconf parser problems compile bugs in the CVS commits

16. University of Maryland 16 Case Study: Results Our checker marked over 7,000 errors –individual call site for non-void function where the return value is not checked Too many too look at! –expect many are false positives Rank errors –inspect CVS bug fix commit flagged functions –inspect functions with return value checked more than 50% of the time in the current source tree value = foo(); // ERROR newPosition + = value; … result = foo(); // ERROR zoo(result);

17. University of Maryland 17 Case Study: Error Breakdown Inspected 453 errors (of 7,000) –found 98 that may be bugs! 231 errors associated with a CVS bug fix flagged function –61 of the 98 bugs found here –false positive rate of 74% 222 errors associated with a function that has its return value checked > 50% of the time –37 of the 98 bugs found here –false positive rate of 83%

18. University of Maryland 18 Case Study: A Bug We investigated an error and found it did crash httpd –error reported near the top of the ranking The called function builds a filename –arguments represent file and pathname –a char array is returned and directly used as an argument to strcmp() –strcmp(foo()) –NULL return value will cause a seg fault –return value is NULL if the path is too long!

19. University of Maryland 19 Analysis False positive rate too high! –overall false positive rate: 78% (1-(98/453)) A false positive rate closer to 50% would be acceptable –the user is likely as not to find a true error –cluster them near the top of the ranking We did cull 7,000 errors down to 453 –lint would have flagged only the ‘ignored’ errors and not ranked them

20. University of Maryland 20 Conclusion Bug databases are not useful in understanding much about low-level bugs –good for logic errors –good for misunderstood specifications CVS commit messages give a better picture of low-level bugs –especially bugs that don’t enter a release CVS commits can give useful data to help classify error reports

21. University of Maryland 21 Future Work What other types of bugs are common? What other checkers can benefit from CVS data? How can we cut the false positive rate? Can we dynamically gather data on functions called via function pointers? –many of the error messages involved calls through function pointers –Dyninst will allow us to instrument function pointer call sites and gather data