1. Marc Schönefeld Software-Architect schonef@acm.org
Security Aspects in Java Bytecode Engineering (Blackhat Briefings 2002, Las Vegas, Aug 01,02)
Marc Schönefeld Software-Architect

2. Agenda Java Security Architecture: an Overview
The JVM: Structures and Concepts
Bytecode Basics
Bytecode (Reverse) Engineering
Applications of Bytecode (Reverse) Engineering

3. Java Security Architecture:
       
Java Security Architecture:
an overview

4. How the Court defines Java ( NO
How the Court defines Java ( NO. C RMW(PVT) , ORDER RE SUN'S MOTION FOR PRELIMINARY INJUNCTION AGAINST MICROSOFT)
„Sun's JAVATM Technology is a so-called "class-based" language in that its functionality is determined by the Java classes available to the programmer.
[...] Programs written in the Java programming language are compiled into intermediate instructions called bytecodes or Applets. [...]. These [...] are then "interpreted" by another computer program which emulates a hypothetical CPU called the Java Virtual Machine. [...]
The Java Virtual Machine translates the Applets into instructions understood by the specific computer CPU on which the Java Virtual Machine is running. Therefore, a specific interpreter or virtual machine is needed for each computer CPU on which the Java program is run.“

5. How the court defines Java Findings of Judge Thomas Penfield Jackson, Nov. 5, 1999
73. The term "Java" refers to four interlocking elements.
First, there is a Java programming language with which developers can write applications.
Second, there is a set of programs written in Java that expose APIs on which developers writing in Java can rely. These programs are called the "Java class libraries."
The third element is the Java compiler, which translates the code written by the developer into Java "bytecode."
Finally, there are programs called "Java virtual machines," or "JVMs" which translate Java bytecode into instructions comprehensible to the underlying operating system.

6. Java Architectural Stack
EJBeans Servlets, JSP
Applet
Application
J2EE App-Server
Browser JRE
Standalone JRE, J2SE Web Start
Java Virtual Machine
Operating System
Hardware

7. The typical JVM development environment
Runtime
Dog.java
Javac/
jikes
Dog.
class
Class
loader V E R I F
System
Class
loader
String
system
class
Cat.py
jython
Cat.
class
Class
loader
Object
system
class
Bird.j J A S M
Bird.
class
Class
loader
<other>
System
class J V M

8. The basic security architecture
Java security (APIs)
(access): The Security manager
(origin): Signed Codebases
(behalf): Principle-based access control (JAAS)
cryptography
JVM security
Class loaders
Class file verification process
JVM intrinsic security features

9. Java security Security Manager and its API
Central instance for access control as far as code is concerned
Policies define access to outer-domain ressources
SecurityManager objects instances enforce policies, throwing SecurityExceptions
By Default java programs do not have a security manager, therefor it is a good precaution to instantiate one

10. Java security Security Manager and its API
Fine-grained control to Limit access on:
SocketConnections (create, accept, multicast)
Thread Groups
Dynamic Library Loading (JNI)
Files (read, write, delete)
Access to External shared ressources (printjob, clipboard)
Program control (exit, toplevelwindow)
Runtime components (member, package, classloader)

11. Java security Code Base Authentication
Java-Archives (JARs) store codebases
Proof of Origin can be be achieved by signing the jars
JAR
Cat.class
Dog.class
Bird.class
JAR
Signed hash
Cat.class
Dog.class
Bird.class
hash
Private
key
sign

12. Java security JAAS: Security based on principals
Enables login functionality
Username, password
Fingerprint
...
Execution permitted/denied depending on the identity who runs the code
Policy based access to functionality
Fine-grained permission handling possible

13. JVM security intrinsic features
Non-continuous memory model, distinct data areas
Java stack frames (execution state)
Method area (bytecode storage)
Garbage-collected heap (object storage)
Type-safe casting
No self-modifying code
Automated garbage-collecting disallows explicit free operation
Automatic Array bounds-checking prevents off-by-one and buffer overflow scenarios

14. JVM security Class loaders
Classloaders load a classfile as byte array into the JVM
Can load from
file,
network or
dynamically generated byte array
Can even compile on the fly (so Java behaves like Perl)
Security features
Establishing name spaces
Enforcing separation of trusted system library code from user-supplied code via parent-delegation

15. JVM security Verifier Task: check loaded classfile for integrity
4-step process
1st step: structural correctness
2nd step: data type correctness
3rd step: bytecode checks
4th step: symbolical references management (runtime)

16. JVM security Classfile verification
public class Cat {
void bite (int times) {
... } J V M
Verifier P A S 1 2 3 4
JAVAC
Class
loader
CA FE BA BE D
CA FE BA BE D
bytecode assembler
.class public Dog
.method bite I
.invokestatic seekVictim
...
.end method
.end class

17. The Verification Process Pass 1: Basic Structural checks
the classloader delivers byte array
Magic number = 0xCAFEBABE ?
Version id: 1.1=45.3, 1.2=46.0, 1.3=47.0, 1.4=48.0
All recognized attributes need to be in proper length
The class file must not be truncated or have extra bytes at the end
The constant pool must not contain any „superficially unrecognizable information“

18. The Verification Process Pass 2: Check Context-Pool (CP) information
final classes are not subclassed, and final methods are not overridden.
All classes (except java.lang.Object) must have a superclass.
Check constraints for CP-entries: For example, class references in the CP can be resolved via a field to a string reference in the CP.
Checking that all field references and method references in the CP must have legal names, classes, and type signature.

19. The Verification Process Pass 3 : Bytecode verification
Core part of verification
Static constraints
Checking maximal local variable count throughout control flow
Checking control-flow correctness (branch always to start of instruction, not beyond end of code)
all exception-handlers are valid (no partial overlap)
...
Structural constraints
Reachability : subroutines (scope), exception handlers
data-flow : Instances initialization and new objects, stack size

20. The Verification Process Pass 4: delayed checks during runtime
Verifies that currently executing class is allowed to reference the given class.
The first time an instruction calls a method, or accesses or modifies a field, the verifier checks the following:
method or field class
Method or field signature
that the currently executing method has access to the given method or field
insert „quick“ optimized instructions

21. Problems with JDK verifier Not enabled by default
public class IllegalAccess2{
public String str = "The trade secret";
public void test(){
System.out.println("Test2: " + str); }
public class IllegalAccess{
public static void main(String args[]){
IllegalAccess2 t2 = new IllegalAccess2();
System.out.println("Test1: " + t2.str);
t2.test();
t2.str = "an open hint";
System.exit(0); }
D:\entw\java\blackhat>java -classpath . IllegalAccess
Test1: The trade secret
Test2: The trade secret
Test1: an open hint

22. Problems with JDK verifier Not enabled by default
public class IllegalAccess2{
private String str = "The trade secret";
public void test(){
System.out.println("Test2: " + str); }
public class IllegalAccess{
public static void main(String[] args){
IllegalAccess2 t2 = new IllegalAccess2();
System.out.println("Test1: " + t2.str);
t2.test();
t2.str = "an open hint";
System.exit(0); }
Variable str now restricted, JVM should now complain access, but at least on JDK1.3.1 and 1.4.0_ the following happens...

23. Problems with JDK verifier Not enabled by default
D:\entw\java\blackhat>java IllegalAccess
Test1: The trade secret
Test2: The trade secret
Test1: an open hint
D:\entw\java\blackhat>java -verify IllegalAccess
Exception in thread "main" java.lang.IllegalAccessError: try to access field IllegalAccess2.str from class IllegalAccess
at IllegalAccess.main(IllegalAccess.java:7)
Only the explicit -verify flag restricts access to restricted variable „str“ !

24. Problems with Inner classes
Inner classes can access private fields of outer classes
final class Outer {
private String secret = "you will never be able to read this" ;
public void alter_secret(String x) {
secret = x; }
private String reverseSecret() {
StringBuffer b = new StringBuffer(secret);
return b.reverse().toString();
class Inner {
private String innersecret = secret;
private String reverseinner = reverseSecret();
public static void main(String[] args) {
Outer outer = new Outer();
new

25. Problems with Inner classes
You can‘t access these private fields from other classes via java code, but you can with a handcrafted bytecode class
new
new Outer
dup
invokespecial Outer/<init> ()V // new Outer
dup // dup here to avoid local vars, take it directly from stack
invokestatic Outer/access$000 (LOuter;)Ljava/lang/String;
getstatic java/lang/System/out Ljava/io/PrintStream;
swap // correct the positions
invokevirtual java/io/PrintStream/println (Ljava/lang/String;)V
invokestatic Outer/access$100 (LOuter;)Ljava/lang/String;
swap // correct the positions
return

26. Other Problems with JDK verifier
SDK and JRE 1.3.1_01 or earlier
„A vulnerability in the Java(TM) Runtime Environment Bytecode Verifier may be exploited by an untrusted applet to escalate privileges. „
Also some Netscape browser versions were affected

27. Problems with JDK verifier
You can check your classes with a standalone verifier
Open source solution „Justice“ supplied in Apache Jakarta project BCEL
new

28. Problems with Java security What is also still missing
Checks in terms of hard and soft limits on
memory allocation
Thread activation
Excessive memory usage and threading utilization often leads to Denial of Service problems

29. The JVM:
Structures
and Concepts

30. JVM Internals The architecture Java bytecode, the internal language
JVM is an abstract concept
Sun just specified the interface
implementation details depend on specific product (SUN JDK, IBM JDK, Blackdown)
Java bytecode, the internal language
independent from CPU-type (bytecode)
Stackoriented, object-oriented, type-safe

31. Runtime view on a JVM Method Runtime Data storage Area (Classes)
Heap (Objects)
JVM
runtime
Class
loader
PC registers
Stack Frames
Native
methods
Native method
stacks

32. Runtime data
Frame:
Saves runtime state of execution threads, therefore holds information for method execution (program counter)
All frames of a thread are managed in a stack frame

33. Runtime data Method area Runtime information of the class file
Type information
Constant Pool
Method information
Field information
Class static fields
Reference to the classloader of the class
Reference to reflection anchor (Class)

34. The Constant Pool
The "constant pool" is a heterogenous array of data. Each entry in the constant pool can be one of the following:
string , class or interface name , reference to a field or method , numeric value , constant String value
No other part of the class file makes specific references to strings, classes, fields, or methods. All references constants and also for names of methods and fields are via lookup into the constant pool.

35. The Class File StructureD R
CONSTANT-
POOL I N T E R F A C S
ACCESS
FLAGS
(Final,
Native,
Private,
Protected,
...) A T R I B U E S
FIELDS
METHODS
You can use a classdumper lake javap -c or DumpClass to analyze these inner details

36. The Class File Format
Java class files are brought into the JVM via the classloader
The class file is basically just a plain byte array, following the rules of the byte code verifier.
All 16-bit and 32-bit quantities are formed by reading in two or four 8-bit bytes, respectively, and joining them together in big-endian format.

37. Methods and Fields
The type of a field or method is indicated by a string called its signature.
Fields may have an additional attribute giving the field's initial value.
Methods have an additional CODE attribute giving the java bytecode for executing that method.

38. The CODE Attribute maximum stack space
maximum number of local variables
The actual bytecode for executing the method.
A table of exception handlers,
start and end offset into the bytecodes,
an exception type, and
the offset of a handler for the exception

39. Bytecode
Basics

40. The JVM types JVM-Types and their prefixes
Byte  b 
Short  s 
Integer  i  (java booleans are mapped to jvm ints!)
Long  l 
Character  c
Single float f 
double float d 
References  a  to Classes, Interfaces, Arrays
These Prefixes used in opcodes (iadd, astore,...)

41. The JVM Instruction Mnemonics
Shuffling (pop, swap, dup, ...)
Calculating (iadd, isub, imul, idiv, ineg,...)
Conversion (d2i, i2b, d2f, i2z,...)
Local storage operation (iload, istore,...)
Array Operation (arraylength, newarray,...)
Object management (get/putfield, invokevirtual, new)
Push operation (aconst_null, iconst_m1,....)
Control flow (nop, goto, jsr, ret, tableswitch,...)
Threading (monitorenter, monitorexit,...)

42. Bytecode
Java Bytecode (JBC) are followed by zero or more bytes of additional operand information.
Table lookup instructions (tableswitch, lookupswitch) have a flexible length
The wide operation extension allows the base operations to use „large“ operands
No self-modifying code
No branching to arbitrary locations, only to beginning of instructions limited to scope of current method (enforced by verifier!)

43. Bytecode
(Reverse)
Engineering

44. Bytecode Engineering tools
Obfuscators
Remove/Manipulate all information that can be used for reverse engineering
Native compilers
„Real“ compile of java bytecodes to native instructions (x86/sparc)
Build your own bytecode
Programmatic Generation
Manipulate classfiles with an API

45. Obfuscators Techniques used
Identifier Name Mangling 
The JVM does not need useful names for Methods and Fields
They can be renamed to single letter identifiers
Constant Pool Name Mangling
Decrypts constant pool entries on runtime
Control flow obfuscation
Insertion of phantom variables, stack scrambling
And by relying on their default values inserting ghost branch instructions, which never execute

46. Obfuscators Problems with Obfuscation
Constant value Mangling implies overhead processing in extra method call of an „deobfuscatename“ method in each retrieval from constant pool
Dynamic class loading may become broken as classes get new names and reflection calls like class.forName(„Account“) will fail because class „Account“ now known as by it‘s obfuscated name „b16“!
And: Obfuscation breaks patterns that can be recognized by JIT-engines for optimization

47. Obfuscators Design your code for better protection
Try to use the lowest visibility (if possible private, if not package),
because public and protected fields must keep their name after obfuscation and therefore reveal some structure of your intellectual property , where possible remove unneccessary „public“ modifiers
Interaction into the package should be done via a few explicit public accessor classes
to set/get internal fields and call methods
Use resource bundles instead of the classfiles to hold string constants
Harder reverse-engineering thru one more protectable indirection

48. Protecting the Source Code: Native Compilers
Convert Java bytecode to C
Generate executable via normal c-build
fast execution
Additional decompilation effort needed
Long turnaround times
Even for small java programs you get monster size executable files (67mb source for Viva.java) from some commercial products
Transformed program may than be vulnerable to buffer overflows and off-by-ones

49. Bytecode Reverse Engineering
Decompilation
Get Source code from class files
Graphical Analysis
Rebuild the logical control flow
Disassembly
Get symbolic bytecode from class files

50. But only one really works
Decompilers
But only one really works
Several available,
MOCHA , by Hanpeter van Vliet was the first one
JASMINE , patch of Mocha - crashed less often
JAD, the fast Java Decompiler (written in C++) by Pavel Kouznetsov is quite usable, but can also be fooled
DJ JAVA , GUI for JAD
NMI, GUI for JAD

51. Decompilers General method of decompilation
Rebuilding control flow from class file code segment
Matching flow patterns to java language contructs
Associating constants and external references from constant pool entries
Do not redo field/method and constant pool entry mangling
Condensed constant pool entries therefore sometimes result in non-valid java identifiers in the generated source

52. Graphical Analysis Overview
All Java Classfiles (those which are obfuscated, too) have to be compliant to the JVM specification
Although Control flow is interleaved in the obfuscated class file, a graphical flow reveals most of the original control flow
The inner structures and dependencies of the methods can be discovered by graphical analysis

54. Graphical Analysis Tools
BCEL: Byte-Code Engineering Library
Written by to Markus Dahm
Maintained by the Apache Jakarta Project
Exposes internal structure of *.class files as java objects
aiSee: graphical visualisation tool
Cross-platform
Developed by the university of saarland
Uses graphical description language

55. Graphical Analysis Procedure
Walk thru a specific code attribute of methods in a classfile
Retrieve target instructions
Calculate control blocks
Calculate successor relationships
Rebuild control flow
Export control flow graph (CFG) as GDL
Display the GDL file for the control flow in aiSee

56. Disassembling Javap Gnoloo The disassembler from the JDK
disassembler by Joshua Engel from „Programming for the JVM“ (ISBN: )
Can be used in pair with Oolong, the corresponding assembler
Comes with a good DumpClass tool

57. Applications of
Bytecode
(Reverse)
Engineering

58. Build your own bytecode
Tools
How-to
Scenarios

59. Build your own bytecode Tools
Javac & Javap
Included in the JDK
Shows you how to translate high language constructs to bytecode
Hex-Editor for binary patching
Ultraedit or Xemacs
Oolong/Gnoloo
Assembler and disassembler
BCEL

60. Managing the Bytecode with BCEL
BCEL comes with several tools
org.apache.bcel.util.Class2HTML
Generates nice HTML pages for a class
org.apache.bcel.util.JavaWrapper
Replaces standard java interpreter to use own class loader
Lets you modify a loaded class via modifyClass method, that can be overloaded
org.apache.bcel.verifier.GraphicalVerifier
Allows you to verify your class files with verbose output

61. Binary Patching with your favorite editor
Read the output of „javap –c“ or DumpClass
You got to know the opcodes values to find the injection point
Possibilities:
so you can blank out some unwanted code with (nop)..
Adjust some binary checks
For example x9a0016 to x („ifne +22“ to „pop nop nop“)
Remember to keep the right balance on the stack

62. Sample bytecode The java source
public class Viva {
public static void main(String[] args) {
String[] TOTD = {"Viva","Las","Vegas"};
for (int i = 0; i < TOTD.length; i++) {
new Viva(TOTD[i]); }
public Viva(String a) {
System.out.println(a.toUpperCase());

63. Sample bytecode The disassembled Viva bytecode (Gnoloo) –I-
.method public static main ([Ljava/lang/String;)V
.limit stack 4
.limit locals 3
.line 3
iconst_3
anewarray java/lang/String
dup
iconst_0
ldc "Viva"
aastore
iconst_1
ldc "Las"
iconst_2
ldc "Vegas"
astore_1
.line 4
istore_2
goto l39
.line 5
l25:
new Viva
dup
aload_1
iload_2
aaload
invokespecial Viva/<init> (Ljava/lang/String;)V
pop
.line 4
iinc 2 1
l39:
arraylength
if_icmplt l25
.line 7
return

64. Sample bytecode The disassembled Viva bytecode (Gnoloo) –II-
.method public <init> (Ljava/lang/String;)V
.limit stack 2
.limit locals 2
.line 8
aload_0
invokespecial java/lang/Object/<init> ()V
.line 9
getstatic java/lang/System/out Ljava/io/PrintStream;
aload_1
invokevirtual java/lang/String/toUpperCase ()Ljava/lang/String;
invokevirtual java/io/PrintStream/println (Ljava/lang/String;)V
.line 10
return
.end method

65. Analysis of some obfuscator‘s work
DEMO

66. Things you can only do in JBC methods that only differ in return type!
.class public overload
.super java/lang/Object
.field static myint I
.field static mydouble D
.method static jvmoverload ()I
.limit stack 3
.limit locals 1
iconst_1
ireturn
.end method
.method static jvmoverload ()D
dconst_1
dreturn
.method public static main ([Ljava/lang/String;)V
.limit stack 3
.limit locals 1
invokestatic overload/jvmoverload ()I
putstatic overload/myint I
invokestatic overload/jvmoverload ()D
putstatic overload/mydouble D
return
.end method
.end class

67. Things you can only do in JBC methods that only differ in return type!
If the decompiled source is compiled again:
The signatures are considered duplicate in java
But valid in java bytecode, and therefore can be used in obfuscation techniques
>javac -classpath . overload.java
overload.java:15: jvmoverload() is already defined in
overload
static double jvmoverload() ^
1 error

68. DEMO

69. Usage Patterns Scenarios Tools for database access
Adding dynamically Logging, profiling facilities without changing sourcecode
Verifying class files on domain boundaries (add-in to content filtering systems)
Find Frontier bugs, before they may be exploited for DoS-attacks by calling code which crashes the JVM
Port your own language to Java bytecode

70. Java Library Holes
In JNI-based applications checking for null pointers is often given too little regard
Also the routines in the core Java Runtime classes use native code (included in jvm.dll)
But not every entry into these trusted libs is shielded appropriatly
In some cases you can force a JVM crash when pumping „null“ values to that routines

71. Frontier Bugs Sample package crashtest; import sun.dc.pr.PathDasher;
public class CrashTest {
public CrashTest()
PathDasher d = new PathDasher(null); }
public static void main(String args[])
CrashTest crashTest1 = new CrashTest();
Unexpected Signal : EXCEPTION_ACCESS_VIOLATION occurred at PC=0x6d443081
Function name=JVM_DisableCompiler
Library=H:\programme\JB6\jdk1.3.1\jre\bin\hotspot\jvm.dll
Current Java thread:
at sun.dc.pr.PathDasher.cInitialize(Native Method)
at sun.dc.pr.PathDasher.<init>(PathDasher.java:48)
at crashtest.CrashTest.<init>(CrashTest.java:91)
at crashtest.CrashTest.main(CrashTest.java:98)
Dynamic libraries:
0x x H:\programme\JB6\jdk1.3.1\bin\javaw.exe
0x77F x77FF0000 C:\WINDOWS\System32\ntdll.dll
....
0x76BB x76BBB000 C:\WINDOWS\System32\PSAPI.DLL
Local Time = Sun Jun 23 14:00:
Elapsed Time = 0 #
# HotSpot Virtual Machine Error : EXCEPTION_ACCESS_VIOLATION
# Error ID : 4F530E CC
# Please report this error at #
# Java VM: Java HotSpot(TM) Client VM (1.3.1-b24 mixed mode)
# An error report file has been saved as hs_err_pid3888.log.
# Please refer to the file for further information.

72. Frontier Bugs Localization
This possible vulnerability may also be harmful in other tools as if they rely on JNI (native interface) classes calling to some native core (written in some appropriate language, C or C++)
Example JDK: The frontier between
java library code (rt.jar) and the
jvm (jvm.dll, libjvm.so) has official and inofficial entrypoints
The official ones are called by Native Java-routines recommended by Sun (java.*)
The inofficial are called by sun.* routines

73. Frontier Bugs Semiautomatic Detection
Traverse java runtime libraries
Procedure
Detect public classes
In the public classes detect public methods
From that filter native methods
Now you have a list containing the public entry points to the JVM library or another JNI-library
Tool: NativeFinder, which is based on BCEL

74. Frontier Bugs Semiautomatic Detection
On this list:
Start with the static class methods, they are easier to test
With the object methods, you need to know how to build such on object
In the easy case you have an default constructor
In the hard case you will have to delve deep into the class hierarchy to get an object
Then call the method with some nice „null“ values and wait 

75. DEMO

76. Soon to become a problem: Hot Swap
Until JDK 1.3 class definitions were immutable during the lifetime of a JVM
JDK 1.4 now offers a „hot swap“ of class definitions via JVMDI (debugger API)
Now you (or someone else?) can „substitute your class definitions by modified code in a running application through the debugger“

77. Q&A Mail to schonef@acm.org for the tools Or visit our website
for some more examples and links (sept 02)

78. Thank you! To You as a great audience,
To The Blackhat Staff (specially Charel for the tickets),
To Halvar, Johnny and G0dzilla for the helpful discussions,
And to my Girlfriend