1. Lightweight Modeling of Java Virtual Machine Security Constraints using Alloy Mark Reynolds BU CS511 Final Report May 7, 2008

2. Outline Brief Recap of Model Status at Midterm New Constraints New Results Summary

3. JVM Bytecode Verifier Operates on binary class files –“JVM assembly language” Performs a superset of the set of checks performed by the Java compiler Uses a constraint based approach –Local variable constraint –Stack depth invariance constraint –Stack guard constraint –Opcode constraint –Method argument constraint –Many others Ideal for an Alloy model

4. Midterm Status Model template created –“Instruction” sig models instruction properties –“State” sig models instruction execution within a method –Complete Alloy model except for initialization Class2Alloy converter written and tested –Takes class files, converts them to Alloy relation initializers –Class2Alloy output + model template = complete Alloy model Local Variable Constraint checked successfully –“All local variables must be written before being read” –Normal bytecodes pass –Deliberately erroneous bytecodes fail

5. New Constraints Stack depth invariance constraint –The stack depth at any program point is the same for any path leading to that point Stack guard constraint –The depth of the stack is never negative –JVM is not like the Intel architecture, method arguments are passed and returned in local variables, not on the stack

6. Stack Depth Invariance No matter how you get there, the stack depth must be the same

7. Stack Guard Constraint 0 Time Stack depth on method entry = 0 Stack depth on method exit = 0 Stack depth >= 0 always

8. Stack Constraint Checking Added “smod” relation to “Instruction” sig in order to capture stack modifications carried out by individual instructions Added “Depth” as a relation within “State” sig as a way of modeling stack depth as a function of execution state Modified “nextState” predicate to update stack depth from smod relation value pred nextState[s, s': State] { nextInstruction[s.prog, s'.prog] && nextReader[s.prog, s.readers, s'.readers] && nextWriter[s.prog, s.writers, s'.writers] && (s'.depth = add[s.depth, s.prog.smod]) }

9. Instruction Relations - Example maplenrwtermubtcbtsmod startup10,10 iload_10111 bipush 10121 icmpge 133313-2 iload_16111 iconst_5711 imul81 istore_2912 goto 15103150 iload_113111 istore_21412 iload_215121 ireturn1611

10. Stack Constraint Assertions assert checkit1 { all s, s' : State | (s.prog.map = s'.prog.map) => (s.depth = s'.depth) } assert checkit2 { all s : State | gte[s.depth, 0] }

11. Results Normal bytecodes pass both new constraints as well as the original LV constraint! Deliberately aberrant bytecodes fail at least one of the constraints

12. Summary Alloy is very well suited to modeling JVM execution Three security constraints modeled so far –Many more left to be done Exception handling proved to be difficult –More than one exception handler in a single method was especially tricky –Same for the JVM itself http://cs-people.bu.edu/markreyn