1. MOP: An Efficient and Generic Runtime Verification Framework Feng Chen and Grigore Rosu University of Illinois at Urbana-Champaign

2. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 2 Monitoring of Formal Specifications in Software Runtime verification (RV 2001 – 2007) –Observe a run of the system Instrumentation –Check it against desired properties Explicit or implicit properties –React /report if needed Error messages, throw exceptions, recover

3. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 3 Runtime Verification Systems ≤ 2001 –MAC (UPenn), PAX* (NASA), TimeRover (commercial) 2002-2004 –HAWK/Eagle (NASA), MOP* (UIUC), POTA (UTA) ≥ 2005: –PQL (Stanford), Tracematches (Oxford), PTQL –PEX (Microsoft) –MOP* (added universal parameters, more logics)

4. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 4 Overview What is MOP –Philosophy, Examples, Features Decentralized Monitors and Indexing Experiments and Evaluation Current limitations Conclusions and future work

5. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 5 (MOP)* Monitoring-Oriented Programming http://fsl.cs.uiuc.edu/mop - proposed in 2003 – RV’03, ICFEM’04, RV’05, CAV’05, TACAS’05, CAV’06, CAV’07… *(not to be confused with Meta-Object Protocol) http://fsl.cs.uiuc.edu/mop

6. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 6 What is MOP? Framework where the benefits of monitoring specifications may encourage us to use them –Monitoring is basic design discipline –Recovery allowed and encouraged –Provides to programmers and hides under the hood a large body of formal methods knowledge/techniques –Generic for different application domains Tool supported –JavaMOP – “compiles” MOP for Java into AspectJ

7. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 7 Overview What is MOP –Philosophy, Examples, Features Decentralized Indexing Experiments and Evaluation Current limitations Conclusions and future work

8. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 8 class Resource { /*@ scope = class logic = PTLTL { Event authenticate: end(exec(* authenticate())); Event use: begin(exec(* access())); Formula : use -> authenticate } violation Handler { @this.authenticate(); } @*/ void authenticate() {...} void access() {...}... } JavaMOP Example: “Enforce authentication before use”

9. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 9 /*@ scope = class logic = ERE { Event openRegKey: end(exec(* openRegKey())); Event closeHandle: begin(exec(* closeHandle())); Formula : (openRegKey closeHandle)* } validation handler { @this.closeRegKey(); return; } @*/ Method openRegKey should be followed by closeRegKey, not by closeHandle JavaMOP Example: Correcting method matching

10. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 10 /*@ scope = class logic = ERE [ int count = 0; int writes = 0;] { Event open : end(call(* open(..))) {writes = 0;}; Event write : end(call(* write(..))) {writes ++;} ; Event close : end(call(* close(..))); Formula : (open write+ close)* } violation handler{ @RESET; } validation handler{ count ++; File2.log(count + ": " + writes); } @*/ JavaMOP Example: Profiling How many times a file is open, written to, and then closed?

11. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 11 Multi-Parameter Specifications Until now, MOP was limited to one- parameter specifications; the examples above showed only such specifications: parameter = current object Because of that restriction monitor states were always stored as fields into the monitored objects in MOP Situation is more complex for multiple- parameter specifications …

12. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 12 Fail Fast Iterators in Java Vector v = new Vector(); v.add(new Integer(1)); Iterator i = v.iterator(); v.add(new Integer(2)); Following code throws exception in Java: No exception raised if one uses Enumeration instead of Iterator –Java language decision, showing that properties referring to sets of objects are important

13. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 13 JavaMOP Example: Safe Enumeration (borrowed from Tracematches) –We thank Tracematches team for their useful collection of regular pattern properties! Safety property: –If nextElement() invoked on an enumeration object, then the corresponding collection (vector) is not allowed to change after the creation of the enumeration object create next* updatesource+ next

14. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 14 /*@ scope = global logic = ERE SafeEnum(Vector v, Enumeration+ e) { Event create : end(call(Enumeration+.new(v,..))) with e; Event updatesource : end(call(* v.add*(..))) \/ … ; Event next : begin(call(Object e.nextElement())); Formula : create next* updatesource+ next } validation handler { System.out.println("Error..."); } @*/ Multiple-Parameters in JavaMOP Safe Enumeration JavaMOP generates >200 AspectJ LOC from above

15. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 15 Overview What is MOP –Philosophy, Examples, Features Decentralized Monitors and Indexing Experiments and Evaluation Current limitations Conclusions and future work

16. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 16 MOP Features: Extensible logic framework Observation: no silver-bullet logic for specs MOP logic plugins (the “How”): encapsulate monitor synthesizers; so far we have plugins for –ERE (extended regular expressions) –PtLTL (Past-time LTL) and FtLTL (Future-time LTL) –ATL (Allen temporal logic), –JML (fragment of Java modeling language); –NEW: CFG and ptCaRet plugins (need stack) Generic universal parameters –Allow monitor instances per groups of objects

17. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 17 MOP Features: Configurable monitors Working scope (the “Where”) –Check point: check spec at defined place –Method: pre-post condition –Class: check spec everywhere during obj lifetime –Interface: check spec at boundaries of methods –Global: may refer to more than one object Running mode –Inline: shares resources with application –Outline: communicates with application via sockets –Offline: generated monitor has random access to log

18. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 18 MOP Features: Programmable Actions Monitor variables Event-triggered actions (any code) Violation/validation handlers (any code) Logic-based AOP instance –Aspects are formal specifications –MOP aim: generate optimal AOP code

19. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 19 Overview What is MOP –Philosophy, Examples, Features Decentralized Monitors and Indexing Experiments and Evaluation Current limitations Conclusions and future work

20. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 20 Decentralized Monitoring/Indexing (works only for inline MOP) The problem: how to monitor a universally quantified specification efficiently! create udatesource next create next* updatesource+ next (  v,e ) 

21. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 21 “Super-Logic” Approach Eagle, PQL, PTQL, Tracematches, … Choose a universally quantified logic Device monitor synthesizers for it: Doable in MOP, too, but against its spirit –Logic plugins only know how to generate monitors for  –No relationship between monitors for  and for (  p )  (p) (p)  Mon (  p ) 

22. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 22 MOP’s Decentralized Monitors Monitor instances (one per parameter instance) M  p1 M  p2 M  p3 … M  p1000

23. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 23 Indexing … The next problem: how can we retrieve all needed monitor instances efficiently? M  p1 M  v,e1 M  v,e2 … M  p1000 udatesource Naïve implementation can be very inefficient (both time- and memory-wise)

24. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 24 MOP’s Centralized Indexing One index tree per parameter set Weak references SafeEnum events create udatesource next

25. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 25 MOP’s Decentralized Indexing Monitors scattered all over the code Monitor states piggybacked to object states SafeEnum events create udatesource next

26. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 26 Overview What is MOP –Philosophy, Examples, Features Decentralized Monitors and Indexing Experiments and Evaluation Current limitations Conclusions and future work

27. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 27 The JavaMOP Implementation Short story: 1.Parse the MOP specifications –Generate all monitors using the logic plugins –Find instrumentation points 2.Generate standard AspectJ code 3.Wave everything (observation, monitoring, handlers) together using AspectJ compiler

28. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 28 MOP Evaluation on DaCapo: Statistics

29. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 29 MOP Evaluation on DaCapo: Runtime Overhead Summary: MOP runtime overhead larger than 10% in only 8% of programs

30. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 30 Evaluation of MOP on TM MOP is generic (logic, etc.), but its generality does not come at a performance cost! Following programs part of Tracematches’ (TM) benchmarks (hand optimized by TM)

31. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 31 Overview What is MOP –Philosophy, Examples, Features Decentralized Monitors and Indexing Experiments and Evaluation Current limitations Conclusions and future work

32. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 32 Limitations of the Current Implementation of JavaMOP No nested or existential parameters Monitor creating events must contain all the parameters of the specification No support for real-time Inherit the limitations of AspectJ –Would like to have dynamic aspects … –Though shouldn’t miss events that trigger actions

33. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 33 Conclusions and Future Work MOP is a generic, yet efficient RV framework Experiments showed that RV is feasible Proposed a logic-independent approach to support universally quantified specifications –With both centralized and decentralized indexing Future work –Use MOP for improved and rigorous testing Test case generation, sliced causality –Real-time logic plugins –Static analysis (plugins) to reduce runtime overhead

34. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 34 Related Work Extensively discussed in paper Programming languages –PQL (Lam et al., OOPSLA ’05) –Tracematches (de Moor et al., OOPSLA ’05, ‘07) –PTQL (Goldsmith et al., OOPSLA’05) Runtime verification –MAC (Lee et al., RV’01) –PAX* (Havelund, Rosu, RV’01) –TimeRover (Drusinski, RV’02) –HAWK/Eagle (Barringer et al., VMCAI’04) –PEX (Schulte, Tilman; Microsoft)

35. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 35

36. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 36 Backup slides

37. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 37 Application to Verification isSorted? No Yes FancySort InsertionSort /*@ logic = JML { ensures isSorted(array); } Violation Handler { insertionSort(array); } @*/ Here is an MOP approach to validate a sorting program:

38. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 38 Monitoring-based Verification? Monitor & Recover: presumably reduces the complexity of program verification Monitor Incorrect Correct Complex Component Simple Component Architecture φ p = φ c & (φ m \/ φ s ) φcφc φmφm φsφs Is sorted? Correct sorting Array elements unchanged

39. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 39 Thinking MOP Different applications suggest different ways to think of MOP 1.Merging specification and implementation 2.Extending PLs with logical statements 3.Logical aspects 4.Light-weight formal method If used properly, may reduce complexity of verification

40. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 40 Why Monitoring I Monitoring is well-adopted in many engineering disciplines –Fuses, watchdogs, fire-alarms, etc. Monitoring adds redundancy –Increases reliability, robustness and confidence in correct behavior, reduces risk

41. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 41 Why Monitoring II Ideally, one would like to prove a program correct with respect to its specification … –Known to be hard Provably correct systems can fail, too –Unexpected environment, wrong/strong assumptions, hardware or OS errors, etc.

42. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 42 Monitoring of Formal Specifications in Software Runtime verification (RV 2001 – 2007) –Observe a run of the system Instrumentation –Check it against desired properties Explicit or implicit properties –React /report if needed Error messages, throw exceptions, recover

43. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 43 MOP Conceptual Instances MOP[logic, working-scope, running-mode] JPaX = MOP[LTL, class, outline/offline] –NASA (joint work with Havelund) TemporalRover = MOP[MiTL, class, inline] –Commercial (Drusinski) MAC = MOP[PastLTL, class, inline/outline] –UPenn (Lee, Sokolski,...) Hawk = MOP[Eagle, global, inline] –NASA (Havelund and collaborators) Tracematches = MOP[RegExp, global, inline] –Oxford (de Moor and students) PQL = MOP[CFG, global, inline] –Stanford (Lam and students) PTQL = MOP[SQL, global, outline] –Berkeley, Stanford (Aiken and students)

44. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 44 Why “Monitoring-Oriented” ? We think one can write reliable programs if one adopts a “monitor and recover” style –Not a substitute for writing good code upfront –Write good code, but, “just in case” also say what to do if requirements are violated/validated MOP tools provide support for that … may reduce complexity of verification: prove what is easy, monitor the rest

45. OOPSLA’07 – 25 Oct 2007 MOP – Grigore Rosu 45 MOP Approach to Monitoring: Divide & Conquer What properties/specifications to check? –Combine formal specification and implementation Where to check it? –Automatic monitor integration Currently using Aspect-Oriented Programming How to express the property? –Declarative, automatic monitor synthesis What if specification violated or validated? –Execute user provided “recovery” code