1. Probabilistic Calling Context Michael D. Bond Kathryn S. McKinley University of Texas at Austin

2. Why Context Sensitivity? at com.mckoi.db.jdbcserver.JDBCInterface.execQuery():213 Static program location not enough

3. at com.mckoi.db.jdbcserver.JDBCInterface.execQuery():213 at com.mckoi.db.jdbc.MConnection.executeQuery():348 at com.mckoi.db.jdbc.MStatement.executeQuery():110 at com.mckoi.db.jdbc.MStatement.executeQuery():127 at Test.main():48 Static program location not enough Why Context Sensitivity?

4. at com.mckoi.db.jdbcserver.JDBCInterface.execQuery():213 at com.mckoi.db.jdbc.MConnection.executeQuery():348 at com.mckoi.db.jdbc.MStatement.executeQuery():110 at com.mckoi.db.jdbc.MStatement.executeQuery():127 at Test.main():48 Motivated by  Complex programs  Small methods  Virtual dispatch Static program location not enough Why Context Sensitivity?

5. at com.mckoi.db.jdbcserver.JDBCInterface.execQuery():213 at com.mckoi.db.jdbc.MConnection.executeQuery():348 at com.mckoi.db.jdbc.MStatement.executeQuery():110 at com.mckoi.db.jdbc.MStatement.executeQuery():127 at Test.main():48 Motivated by  Complex programs  Small methods  Virtual dispatch Static program location not enough C/Fortran method Java/C# method return call return Why Context Sensitivity?

6. Context Is Nontrivial API calls ProgramCall sitesDistinct contexts antlr4,184128,627 bloat3,306600,947 chart2,335202,603 eclipse9,611226,020 fop2,22537,710 hsqldb94716,050 jython1,830628,048 luindex654102,556 lusearch507905 pmd1,890847,108 xalan1,53017,905

7. Example: Residual Testing class EditorWindow { close() {... } Does behavior occur at production time that did not occur at testing time? class SimpleWindow { close() {... }

8. Example: Residual Testing class EditorWindow { close() {... } Does behavior occur at production time that did not occur at testing time? class SimpleWindow { close() {... } inputHandler() {... case CLICK_EXIT: w.checkUnsaved(); w.close();... } autoUpdate() {... for all windows w w.close();... }

9. class EditorWindow { close() {... } class SimpleWindow { close() {... } Example: Residual Testing Does behavior occur at production time that did not occur at testing time? inputHandler() {... case CLICK_EXIT: w.checkUnsaved(); w.close();... } autoUpdate() {... for all windows w w.close();... } Bug!

10. class EditorWindow { close() {... } class SimpleWindow { close() {... } Example: Residual Testing Does behavior occur at production time that did not occur at testing time? inputHandler() {... case CLICK_EXIT: w.checkUnsaved(); w.close();... } autoUpdate() {... for all windows w w.close();... } Bug! New behavior indicates bugs Context sensitivity helps find new behavior

11. Two-Phase Dynamic Analyses TrainingProduction Behavior observed New or anomalous behavior detected

12. Two-Phase Dynamic Analyses Residual testing Anomaly-based intrusion detection Anomaly-based bug detection TrainingProduction Behavior observed New or anomalous behavior detected What behavior occurs at production time that did not occur at testing time? [Vaswani et al. ’07] What new behavior occurs during a buggy program run? [Hangal & Lam ’02] Does a program exhibit anomalous behavior? [Inoue ’05]

13. Probabilistic Calling Context TrainingProduction Behavior observed New or anomalous behavior detected Adds context sensitivity to dynamic analyses Maintains value representing context  Unique with high probability  New value  new context  walk stack High accuracy: <0.1% false negatives Low overhead: 3% overhead, 0-8% for clients

14. Outline Introduction Previous approaches Maintaining the PCC value Evaluation  Accuracy  Performance

15. Previous Approaches Tracking context [Ammons et al. ’97] [Spivey ‘04]  Maintain CCT position at each call/return Walking the stack [Nethercote & Seward ‘07] Path profiling [Ball & Larus ’96] [Melski & Reps ’99]  Call graphs large  path explosion  Virtual dispatch complicates instrumentation

16. Previous Approaches Tracking context [Ammons et al. ’97] [Spivey ‘04]  Maintain CCT position at each call/return Walking the stack [Nethercote & Seward ‘07] Path profiling [Ball & Larus ’96] [Melski & Reps ’99]  Call graphs large  path explosion  Virtual dispatch complicates instrumentation Sampling [Zhuang et al. ’06]  Sacrifices coverage for low overhead

17. Outline Introduction Previous approaches Maintaining the PCC value Evaluation  Accuracy  Performance

18. PCC Function f ( V, cs )  V is PCC value  cs is call site ID

19. PCC Function f ( V, cs ) V ← f ( V, cs 1 ) V ← V saved V ← f ( V, cs 2 ) cs 2 cs 1  V is PCC value  cs is call site ID

20. PCC Function f ( V, cs ) ≡ 3V + cs (mod 2 32 )  V is PCC value  cs is call site ID

21. PCC Function f ( V, cs ) ≡ 3V + cs (mod 2 32 ) Motivated by MPI datatype hashing [Langou et al. ’05] [Gropp ’00] Cheap to compute Desirable properties:  Non-commutative  Composition efficient to compute

22. Differentiating Similar Contexts C A B V ← 3V + cs 1 V ← 3V + cs 2 C A B V ← 3V + cs 1 V ← 3V + cs 2 …  A()  B()  … …  B()  A()  …

23. Differentiating Similar Contexts Non-commutative f ( f (V, cs 1 ), cs 2 ) ≠ f ( f (V, cs 2 ), cs 1 ) C A B V ← 3V + cs 1 V ← 3V + cs 2 C A B V ← 3V + cs 1 V ← 3V + cs 2

24. Efficiency at Inlined Calls C A B V ← 3V + cs 1 V ← 3V + cs 2

25. Efficiency at Inlined Calls C A B V ← 3 ( 3V + cs 1 ) + cs 2 C A B V ← 3V + cs 1 V ← 3V + cs 2

26. Efficiency at Inlined Calls C A B V ← 9V + 3cs 1 + cs 2 C A B V ← 3V + cs 1 V ← 3V + cs 2

27. Efficiency at Inlined Calls Composition efficient to compute C A B V ← 9V + 3cs 1 + cs 2 C A B V ← 3V + cs 1 V ← 3V + cs 2

28. Outline Introduction Previous approaches Maintaining the PCC value Evaluation  Methodology  Evaluating potential clients  Accuracy  Performance

29. Methodology Implementation in Jikes RVM 2.4.6  Available on Jikes RVM Research Archive Deterministic calling context profiling  Maintains CCT node at each call & return Benchmarks: DaCapo, SPEC JBB2000, SPEC JVM98 Platform: 3.6 GHz Pentium 4 w/Linux

30. TrainingProduction Behavior observed New or anomalous behavior detected Record values New value  new context  walk stack How Clients Use PCC

31. TrainingProduction Behavior observed New or anomalous behavior detected Evaluating Potential Clients Record values Check values (no new values) Global hash table

32. TrainingProduction Behavior observed New or anomalous behavior detected Evaluating Potential Clients Record values Check values (no new values) Global hash table Memory overhead: proportional to contexts

33. Evaluating Potential Clients Residual testing Check PCC value at Java API calls (calls to java.* ) Check PCC value at system calls (Network, I/O, OS) Anomaly-based intrusion detection Upper bound Check PCC value at all calls

34. Ideal Accuracy PCC maps context to value  New PCC value  new context  Familiar PCC value  probably familiar context

35. Ideal Accuracy Expected conflicts (false negatives) Distinct contexts32-bit values64-bit values 100,0001 (0.0%)0 (0.0%) 1,000,000116 (0.0%)0 (0.0%) 10,000,00011,632 (0.1%)0 (0.0%) 100,000,0001,155,170 (1.2%)0 (0.0%) 1,000,000,000107,882,641 (10.8%)0 (0.0%) 10,000,000,0006,123,623,065 (61.2%)3 (0.0%) PCC maps context to value  New PCC value  new context  Familiar PCC value  probably familiar context

36. Ideal Accuracy Expected conflicts (false negatives) Distinct contexts32-bit values64-bit values 100,0001 (0.0%)0 (0.0%) 1,000,000116 (0.0%)0 (0.0%) 10,000,00011,632 (0.1%)0 (0.0%) 100,000,0001,155,170 (1.2%)0 (0.0%) 1,000,000,000107,882,641 (10.8%)0 (0.0%) 10,000,000,0006,123,623,065 (61.2%)3 (0.0%) PCC maps context to value  New PCC value  new context  Familiar PCC value  probably familiar context API calls

37. Ideal Accuracy Expected conflicts (false negatives) Distinct contexts32-bit values64-bit values 100,0001 (0.0%)0 (0.0%) 1,000,000116 (0.0%)0 (0.0%) 10,000,00011,632 (0.1%)0 (0.0%) 100,000,0001,155,170 (1.2%)0 (0.0%) 1,000,000,000107,882,641 (10.8%)0 (0.0%) 10,000,000,0006,123,623,065 (61.2%)3 (0.0%) PCC maps context to value  New PCC value  new context  Familiar PCC value  probably familiar context All calls

38. Ideal Accuracy Expected conflicts (false negatives) Distinct contexts32-bit values64-bit values 100,0001 (0.0%)0 (0.0%) 1,000,000116 (0.0%)0 (0.0%) 10,000,00011,632 (0.1%)0 (0.0%) 100,000,0001,155,170 (1.2%)0 (0.0%) 1,000,000,000107,882,641 (10.8%)0 (0.0%) 10,000,000,0006,123,623,065 (61.2%)3 (0.0%) PCC maps context to value  New PCC value  new context  Familiar PCC value  probably familiar context Near-perfect accuracy

39. PCC’s Accuracy System callsJava API calls ProgramDynamicDistinctConf.DynamicDistinctConf. antlr211,4901,567024,422,013128,6273 bloat121001,159,281,573600,94740 chart63620258,891,525202,6034 eclipse14,1101970132,507,343226,0205 fop181709,918,27537,7100 hsqldb12 081,161,54116,0500 jython5,9294,2890543,845,772628,04848 luindex2,61514039,733,214102,5560 lusearch141110113,511,3119050 pmd1,045250537,017,118847,10879 xalan137,8955902,105,838,67017,9050

40. PCC’s Accuracy System callsJava API calls ProgramDynamicDistinctConf.DynamicDistinctConf. antlr211,4901,567024,422,013128,6273 bloat121001,159,281,573600,94740 chart63620258,891,525202,6034 eclipse14,1101970132,507,343226,0205 fop181709,918,27537,7100 hsqldb12 081,161,54116,0500 jython5,9294,2890543,845,772628,04848 luindex2,61514039,733,214102,5560 lusearch141110113,511,3119050 pmd1,045250537,017,118847,10879 xalan137,8955902,105,838,67017,9050

41. PCC’s Accuracy All calls ProgramDynamicDistinctConf. antlr490,363,2111,006,578118 bloat6,276,446,0591,980,205453 chart908,459,469845,43291 eclipse1,266,810,5044,815,9012,652 fop44,200,446174,9552 hsqldb877,680,667110,7951 jython5,326,949,1583,859,5451,738 luindex740,053,104374,20112 lusearch1,439,034,3366,0390 pmd2,726,876,9578,043,0967,653 xalan10,083,858,546163,2056

42. PCC’s Execution Time Overhead 3%

43. PCC’s Execution Time Overhead 3%

44. Summary PCC maintains calling context value  New value indicates new behavior Low overhead  Maintaining PCC value adds 3%  Checking PCC value 0-8%  Memory overhead proportional to contexts High accuracy  Less than 0.1% false negative rate PCC adds context sensitivity to clients that detect anomalous behavior

45. Summary PCC maintains calling context value  New value indicates new behavior Low overhead  Maintaining PCC value adds 3%  Checking PCC value 0-8%  Memory overhead proportional to contexts High accuracy  Less than 0.1% false negative rate PCC adds context sensitivity to clients that detect anomalous behavior Thank you!

46. Extra slides

47. Do paths capture enough behavior? C/Fortran method Java/C# method return call return Context Sensitivity Mostly Unused