1. CMV: Automatic Verification of Complete Mediation for Java Virtual Machines A. Prasad Sistla, V.N. Venkatakrishnan, M. Zhou, H. Branske University of Illinois at Chicago

2. Introduction Java language and Java Platform  Java Language: With powerful features, gained considerable success  Java Platform: Enable users to write and run applications written in Java languages directly on top it Java Applications and Platforms are Used Everywhere Almost every browser runs JVM and supports running Java Applets Java Applet: A Java program that is run from inside a web browser OS (e.g. Solaris TM, Linux, Windows) Java Virtual Machine (such as HotSpot, Harmony) Java libraries (including java.io, java.net, jave.lang and other packages) Java compiler and other tools Java applications Java Platform

3. Internet Explorer browses a webpage containing a Java Applet Webpage content: …. An Example of Java Applet - From http://www.realapplets.com/applets/quizmaster/

4. Risks What can happen if a Java applet/Java program is allowed to access any system resources?  Compromise the security of the local host Disclose the confidential information: Transmit information Destroy system files Display annoying pictures on a user screen How the JVM prevents this from happening?

5. Java Fine-grained Access Control Model Security Policies  Map code to sets of permissions to access security sensitive resources grant codeBase “URL” { permission java.io.FilePermission “/home/tmp/", "read"; } Runtime Monitoring in JVM  Through a call to library class SecurityManager, before accessing any sensitive resources

6. A Typical Use of SecurityManager public FileInputStream(File file) throws FileNotFoundException { String name = (file != null ? file.getPath() : null); // Existing Solutions SecurityManager security = System.getSecurityManager(); // checks for SM if (security != null) { security.checkRead(name);  permission check } …… open(name);  sensitive operation implemented as a native method } Potential Problem: Without checking the permissions, any file specified by the application can be opened, leading to potential confidential information leaks. Challenge: Ensure that the SecurityManager is consulted on all paths that lead to sensitive operations

7. Our Goal Ensure Java Standard Libraries are trustworthy  They satisfy Complete Mediation Property: SecurityManager is consulted on all paths that lead to sensitive operations  This Assurance will benefit millions of Java users Sensitive Operations  Implemented as native methods Code that is specific to a hardware and operating system platform and are written in other languages, such as C and C++. Called by Java code. Challenges to Our Goal: Large code base - Thousands of classes in Java libraries Existing Model Checking tools (e.g. SLAM, MOPS) are general purpose oriented, while our goal is specialized to verify complete mediation property. These generic tool may not be scalable to a large code base such as the Java libraries

8. Our Solution Security Property Verifier No All paths from public methods to sensitive operations are guarded by security checks. Yes Some paths from some public methods to sensitive operations are missing security checks. Input: Java Standard Libraries Output: An automated static analysis technique. Sound. Not Complete

9. Our Contributions A scalable automated model checking technique that verifies satisfaction of complete mediation in open systems A tool, Complete Mediation Verifier (CMV)  Efficiently analyzes JVM library classes Experimental results in two shrink- wrapped JVM implementations -- HotSpot and Harmony VMs.

10. A Code Example to Walk through Various Verification Approaches public void X () { // do some operations Y(); //sensitive-operation FileRd1(); } private void Y() { if (….){ SecurityManager sm = System.getSecurityManager(); if (sm != null) { sm.checkPermission( FilePermission(pathname”read”)); } // sensitive-operation FileRd2(); } else { // other operations } A branch w/o a security check Risky

11. Analysis : Naive First Approach - private void Y() { 0. if (….){ 1. SecurityManager sm = System.getSecurityManager(); 2. if (sm != null) { 3. sm.checkPermission( FilePermission(pathname, ”read”)); } // sensitive-operation 4. FileRd2(); } else { 5. // other operations } Y1 Y3 Y4 Y0 Y2 Y5 RET safe

12. First Approach (Contd.) - Presence of Method Calls Expand and analyze CFG inline to make Extended CFGs(ECFG) Inline Expansion Drawbacks:  Recursion  Size of ECFG(M) – can be exponential public void X(int x) { 0: // do some operations 1: Y(); //sensitive-operation 2: FileRd1(); } X0 X2 RET X1 Invokes Y CFG(X) RET Expand To X0 X2 Y1 Y3 Y4 Y0 Y2 Y5 Y At risk

13. Summarization All_Path_Secure  No paths to return node(s) w/o security check.  Otherwise, Insecure_Path Good  No paths to unguarded sensitive operations.  Otherwise, Bad We store negations of these two properties Method Summary  A 2-tuple to store  4 kinds of values ENRTEN High-level idea: Analyze methods once; “summarize” and reuse the results

14. Observations to Compute Method Summary A node that Invokes a bad method M 1 A sensitive node equivalent Invokes a All_Path- Secure and Good method M 1 A security check node To compute Bad attribute: equivalent Invokes a Insecure_Path method M 1 a neutral node To compute Insecure_path attribute: equivalent

15. Second Approach – Computing Summaries of Methods Meth. X Meth. Y Step 1: Construct call graph Step 2: Find out Y needs to be analyzed first; The summary of the method computed first can then be reused. In presence of multiple methods, list the methods in reverse topological order Given two methods X & Y. X invokes Y. CFG(Y) Summary(Y) = Step 3: Analyze CFG(Y) to Compute Summary(Y) Y1 Y3 Y4 Y0 Y2 Y5 RET safe

16. Second Approach (Contd.) Step 4: - Compute Summary(X) X0 X2 RET X1 Invokes Y, Summary(Y) = CFG(X) invocation Node X1 is equivalent to a neutral node X0 X2 RET CFG(X) Summary(X) = At risk

17. Second Approach - Algorithm Algorithm in summary: Construct call Graph; Get a list of methods in Reverse Topological Order(RTO); For each method M in the list ordered in RTO Compute Summary(M), reusing the method summaries of those invocation nodes; Runtime complexity: O(ΣM)  ΣM the sum of the sizes of the CFGs of all the methods.

18. Expanded Second Approach - For Recursion Let us say M1 and M2 are mutually recursive methods  Analysis based on reverse topological order will not work Solution:  Repeat the previous algorithm, until all method summaries reach a fix point.  Worst case run time: O(N* ΣM). Quadratic N the number of methods analyzed  Quadratic algorithms still problematic for a large code base such as libraries How can we make this more efficient? Meth. M1 Meth. M2 A cycle exists in the call graph

19. Main Idea:  Compute method summaries for all methods, at the same time  For Insecure_Path summaries, process all the methods at the same time Start with entry nodes in a queue, and keep adding successors to the queue as you explore Linear run time complexity: O( Σ M)  Σ M sum of sizes of the CFGs of all methods public void X(int x) { 0: // do some operations 1: Z(); //sensitive-operation 2: FileRd1(); } Our Approach – A New Efficient Solution (Part 1 – Compute Insecure_Path) private void Z() { 0: SecurityManager sm = System.getSecurityManager(); 1: sm.checkPermission( FilePermission(pathname, ”read”)); //sensitive-operation 2: FileRd2(); } X0 X1 RET X1 Invokes Z Z0 Z1 RET Z1 X0 QStep 1: Add X0, Z0 to Q Z0 Step 2: X0 is a neutral node. Add its successor X1 to Q Z0 Q X1 Step 3: Z0 is a neutral node. Add its successor Z1 to Q X1 Q Z1 Step 4: X1 is a method invocation node. Add X1 to WQ(Z). Z1 Q X1 WQ(Z) Step 5: Z1 is a security check. Stop exploring the path. Q X1 WQ(Z) Step 6: Q is empty. Terminate the algorithm. Both X and Z are All_Path_Secure.

20. Main Idea (Similar to the computing of Insecure_Path )  process all the methods at the same time. Different in processing of each node in the queue Q  for a sensitive operation, label method summary as bad  for a node invoking a method M’, If Insecure_Path(M’) is true, put all successors to the Q Put the node to a waiting queue of M’  Whenever the bad summary of M’ is computed, process the node based on summary of M’ Linear run time complexity: O( Σ M)  O(ΣM) sum of sizes of the CFGs of all methods public void X(int x) { 0: // do some operations 1: Z(); //sensitive-operation 2: FileRd1(); } Our Approach - A New Efficient Solution ( Part 2 – Compute Bad Summaries ) private void Z() { 0: SecurityManager sm = System.getSecurityManager(); 1: sm.checkPermission( FilePermission(pathname, ”read”)); //sensitive-operation 2: FileRd2(); } X0 X1 RET X1 Invokes Z Z0 Z1 RET Z1 X0 QStep 1: Add X0, Z0 to Q Z0 Step 2: X0 is a neutral node. Add its successor X1 to Q Z0 Q X1 Step 3: Z0 is a neutral node. Add its successor Z1 to Q X1 Q Z1 Step 4: X1 is a method invocation node. Insecure_Path(Z) is false. Add Method X to WQ(Z). Z1 Q X WQ(Z) Step 5: Z1 is a security check. Stop exploring the path. Q X WQ(Z) Step 6: Q is empty. Terminate the algorithm. Both X and Z are Good.

21. public void Z() { 0: // do some operations //sensitive-operation 1: FileRd1(); 2: Y(); // Y is a non-public method } Our Improvement to the New Approach Observations  Our goal is to identify paths from public methods to sensitive operations without security checks.  To achieve such a goal, sometimes there is no need to compute some method summaries. Z0 Z1 RET Z2 Invokes Y CFG(Z ) At risk Current solution:  Constructs and explores CFG(Z) and CFG(Y)  Computes Summary(Y) and Summary(Z). However, to identify a risky path in Z,  No need to explore the CFG(Y) and compute Summary(Y)  No need to compute Insecure_Path attribute of Z How to make the algorithm even more efficient?

22. Our Approach: On-the-fly Approach - A Even More Efficient Solution Start to process all public methods first  By putting their entry nodes of CFGs to a queue While exploring the CFGs, if a method invocation node is encountered and is never visited before, add its entry node to the queue to explore its CFG.  Whenever its Bad attribute or Insecure_Path attribute is computed, replace the node with a non-method- invocation node based on its summary For a return node, label the method as Insecure_Path For a sensitive operation, label the method as Bad For a security check, stop exploring that path

23. public void Z() { 0: // do some operations //sensitive-operation 1: FileRd1(); 2: Y(); // Y is a non-public method } An Example to Illustrate the On- the-fly Approach Z0 Z1 RET Z2 Invokes non public method Y CFG(Z ) At risk Step 3: Z1 is sensitive, label method Z as Bad Q Z1 Step 2: Z0 is a neutral node. Add its successors to Q Q Z0 Q Step 1: Add Z0 to Q Step 4: Q is empty. Terminate the algorithm

24. Runtime of On-the-fly Approach In comparison to the non-on-the-fly approach:  No need to construct the Call Graphs to get all methods!  Construct the CFG and compute the method summary of each method on demand. In worst case, linear run time complexity O( Σ M)  Σ M sum of sizes of the CFGs of all methods

25. Witness Generation & Analysis Witness - A call chain to show a counter example, i.e.  An unguarded path to a sensitive operation, or  An insecure path. java.lang.Class: java.lang.Class forName0(String, boolean, ClassLoader) <java.lang.Class: java.lang.Class forName(String) java.lang.Class: java.lang.Class forName(String,boolean, ClassLoader) java.net.URL: java.net.URLStreamHandler getURLStreamHandler(String) java.io.ObjectInputStream: java.lang.Class resolveClass(ObjectStreamClass) java.net.DatagramSocket: void createImpl() Method format: : : Bold highlights risky methods java.net.URL: void (URL, String) java.io.ObjectInputStream: java.lang.Object readObject represents other risky methods

26. Experiment & Results Risky method: A public and bad method. Real Risky: a risky method that has at least one feasible bad path in practice. HotSpot VM from SUN Microsystems, java.io.*; java.net.*; java.lang.Class; Total JVM Class # of Meth. In JVM classConcrete Total Meth.LOBC # of Risky Meth. Real Risky 22775703152023,394610 Harmony VM from Apache Software Foundation, java.io.*; java.net.*; java.lang.Class Total JVM Class # of Meth. In JVM classConcrete Total MethLOBC # of Risky Meth. Real Risky 21748689392865,36200

27. Experiment & Results (Contd.) The average time taken by CMV to analyze each class was 74 seconds  The bulk of the time is spent on Call Graph construction Preloaded method summary database storing summaries Risky methods: Need further analysis In HotSpot VM, only 61 methods to be analyzed, vs. code review on 1520 methods, a reduction in two orders of magnitude! Automated witness analysis: further reduce human review efforts

28. Future Work #1 - Method Overriding A method in base class can be overridden by child classes For a method invocation node, method summaries from base and all child classes should be considered. Our Solution  Transform the code of base class by adding the n- way branch statement before the first statement, to make a call to each overriding method of child classes.  Method summary in each branch is computed

29. Future Work #2 - Exceptions Code can throw exceptions. Control flow is changed Issues  Throw statement If catch block exists, add an edge from throw to catch block Add an edge from throw to return node  Method invocation node Similar to Throw statement  Rethrow statement, i.e. a throw in a catch block Replace the node with a return node Solutions: Add new edges

30. Future Work #3 - Check Non-native Sensitive Operations Sensitive operations are not implemented as native methods. For examples:  Setting a Sensitive Private Data Member  Returning a Sensitive Private Data Member  Returning a Password  Returning Network IP Address Solutions  Identify these sensitive operations, and then apply the same approach (to the handle of sensitive native methods), to compute method summaries.

31. Future Work #4 - Automated Sensitive Operation Detection To automate the detections of sensitive operations in Java libraries. Challenges- Hard to identify them due to:  Some native methods are non-sensitive They do not access sensitive system resources  The parameters passed to a sensitive operation: another factor (in addition to the method name) to determine if a method is a sensitive operation and the sensitive type (e.g. read or write a file)

32. Future Work #5 - Automated Permission Checks Detection Determine the types of permission checks for each sensitive operation. Challenges  Needs to first identify the sensitive operations in JVM.  Should also determine the permission checks for those sensitive operations performed in Java libraries, but without permission checks, due to an undetected defect.

33. Future Work #6 - Certification Given a method say M 0 that is claimed to be Good or All_Path_Secure, verification is needed to ascertain its correctness. Our Solution  We have initiated a novel and scalable verification technique to certify the correctness of method summaries.  Experiments are ongoing, and we will report our work in the future.

34. Related work Model checking  Several general purpose: MOPS, SLAM, Bandera, C-Wolf Ours is specialized for this problem; property-specific customization makes it scalable  Jensen et al [IEEE S&P99] Algorithms for verification of closed systems  Our approach checks open systems (e.g., libraries) Static analysis  Bug-finding [METAL, CQUAL]  Checking complete mediation [Zhang et al [Security’03], Fraser [PLAS06] These techniques require non-trivial transformations to Java-like libraries  E.g. security checks and sensitive ops in different methods Retrofitting code for authorization  Naccio, SASI, Ganapathy et al [IEEE S&P06] One requires retrofitting only when verification reports unsafe methods

35. Tool & Paper Tool: Complete Mediation Verifier (CMV)  Developed. To be released by Dec. 2008 Paper: CMV: Automatic Verification of Complete Mediation for Java Virtual Machines  Accepted by ACM Symposium on Information, Computer and Communications Security (ASIACCS’08), to be held in Tokyo, Japan. March 18-20, 2008  Collaborators: Hilary Branske Prof. Prasad Sistla Prof. V.N.Venkatakrishnan

36. Thank you!