1. Public Deployment of Cooperative Bug Isolation
Ben Liblit, Mayur Naik, Alice Zheng, Alex Aiken, and Michael Jordan
UC Berkeley, Stanford University and University of Wisconsin (pending)
Motivate research
Review instrumentation strategy
Engineering issues exposed working toward public deployment
Eye candy
Screen shots
Deployment status
Sneak peak at analysis results

2. Our Goal: Measure Reality
We measure bridges, airplanes, cars…
Where is ight data recorder for software?
Users are a vast, untapped resource
96,000 new Kazaa users during this workshop
Users know what matters most
Opportunity for reality-directed debugging
Implicit bug triage for an imperfect world
Users define what matters most.
Office XP: 60 million licenses in first year; nearly 2 per second
Kazaa: “over 1.9 million downloaded last week”; more than 3 per second

3. Bug Isolation Architecture
Guesses
Shipping Application
Program Source
Sampler € ƒ ‚
Compiler
Note: no programmer annotation required! Just pick which instrumentation schemes to activate.
Review of instrumentation strategy
Engineering challenges & solutions
Deployment status
Preliminary results
Statistical Debugging
Prole & J/L
Top bugs with likely causes

4. Predicates on Program Behavior
Guess what might be interesting
Branches: Left? Right?
Function returns: Negative? Zero? Positive?
Pairs of variables: Less? Equal? Greater?
Reference counts: Alive? Dead?
Count how often guesses are true
Feedback: vector of counts + outcome label
These are just examples; we have several more schemes too.

5. Sampling the Bernoulli Way
“Next sample” countdown
Geometric distribution
Split into acyclic regions
Finite threshold weight
Martin Hirzel & Trishul Chilimbi: optimal threshold check placement is NP-hard
optimality = minimal set of check nodes such that every cycle contains at least one
by reduction from Vertex-Cover

6. Sampling the Bernoulli Way
“Next sample” countdown
Geometric distribution
Split into acyclic regions
Finite threshold weight 4 3 1 2 1 2 1
Martin Hirzel & Trishul Chilimbi: optimal threshold check placement is NP-hard
optimality = minimal set of check nodes such that every cycle contains at least one
by reduction from Vertex-Cover 1

7. Sampling the Bernoulli Way
“Next sample” countdown
Geometric distribution
Split into acyclic regions
Finite threshold weight
Clone acyclic regions
“Fast” & “slow” variants
Choose at run time
Result:
Subset of dynamic behavior
Statistically fair sample
>4?
Related work:
Arnold & Ryder [2001]

8. Multithreaded Programs
Global next-sample countdown
High contention, small footprint
Want to use registers for performance
Thread-local: one countdown per thread
Global predicate counters
Low contention, large footprint
Optimistic atomic increment

9. Multi-Module Programs
Forget about global static analysis
Plug-ins, shared libraries
Instrumented & uninstrumented code
Self-management at compile time
Locally derive identifying object signature
Embed static site information within object le
Self-management at run time
Report feedback state on normal object unload
Signal handlers walk global object registry

10. Native Compiler Integration
Instrumentor must mimic native compiler
You don’t have time to port & annotate by hand
Our approach: source-to-source, then native
Hooks for GCC:
Stage wrapping via scripts
Flag management via specles
Compiler
Sampler
Guesses
Shipping Application
Program Source
Compiler Tool Chain

11. Keeping the User In Control

12. Public Deployment, To Date

13. Public Deployment, To Date

14. Sneak Peak: Data Exploration
Entire band: total number of runs in which the predicate is observed to be true
Black segment: P(fail | line of code on which the predicate occurs is reached)
White segment: P(succeed | predicate is observed to be true)
Red + pink segments: increase in P(crash | predicate is observed to be true) beyond P(crash line of code on which the predicate occurs is reached)
Red segment only: increase for which we are 95% confident

15. Conclusions Public deployment is challenging
Real world code pushes tools to their limits
Large user communities take time to build
But the results are worth it:
“Thanks to Ben Liblit and the Cooperative Bug Isolation Project, this version of Rhythmbox should be the most stable yet.”

16. The Cooperative Bug Isolation Project
Join the Cause!
The Cooperative Bug Isolation Project