Defect Discovery Features for HPUX/IPF Compilers (Vaccine Project) Suneel Jain Xinliang David Li Xiaohua Zhang Carl Burch

page 2September 20, 2005 AADEBUG 2005 Agenda Features Overview Compile time checks +wsecurity +wlint +w64bit Runtime checks +check=all|none|stack|malloc|uninit

page 3September 20, 2005 AADEBUG 2005 Features Overview Compile time security vulnerability checking – Advanced analysis using interprocedural optimization Compile time detection of potential runtime errors – Whole program lint (similar to FlexeLint) – Diagnostics about 32->64 bit porting problems Runtime error detection via compiler instrumentation – Out of bound accesses for stack, heap objects – Memory allocation/deallocation problems Available with the latest compilers on IPF (AR0509)

page 4September 20, 2005 AADEBUG 2005 Static Analysis of Security Vulnerabilities Buffer overflow vulnerabilities occur with stack buffers being overflowed by an attacker so that the return address can be overwritten, unsafe use of strcpy etc … Format string vulnerabilities occur when untrustworthy data is used as a format string argument. Untrustworthy data  data that could potentially be controlled by an attacker Simplest option is to treat all format strings not known at compile time as potentially dangerous, but that approach results in lots of warnings. We treat all program inputs that could be controlled by an attacker as tainted and then track the propagation of tainted values through each program operation.

page 5September 20, 2005 AADEBUG 2005 Static Analysis of Security Vulnerabilities Uses the option +wsecurity Simple local checks for unsafe libc calls, format string problems (similar to RATS, ITS4) – not enough context leads to many false positives Cross-Module analysis to detect potential use of tainted values at critical reference points – much more accurate – provides more information to the user

page 6September 20, 2005 AADEBUG 2005 Cross-Module Static Analysis Detect if tainted data can reach critical reference Examples of critical references – length argument in string copy APIs – size argument in allocators – target buffer for memory copy APIs – array reference indexing – loop exit condition Interprocedural and context-sensitive flow analysis – vastly improves precision of the analysis – show path from the suspect definition to the reference

page 7September 20, 2005 AADEBUG 2005 Example (xchat) xchat1.c #include extern int getlen(); extern void getdata(int len, char* buf); int main() { char buf[100]; int len = getlen(); getdata(len, buf); copydata(len,buf); } xchat2.c #include int getlen() { int buf[10]; recv(1, buf, 1, 0); return buf[0] + 2; } xchat3.c #include void getdata(int len, char *buf) { recv(1, buf, len, 0); // Line 4 } xchat4.c #include void copydata(int len, char *buf) { int i; for (i = 0; i < len; i++) // Line 7 { buf[i] = i; }

page 8September 20, 2005 AADEBUG 2005 Call Graph main getlen getdata copydata

page 9September 20, 2005 AADEBUG 2005 Data flow main getlen getdata copydata Tainted data detected Critical reference detected Critical reference defected Tainted data Flow back to main Tainted data flow into critical reference

page 10September 20, 2005 AADEBUG 2005 Compiler Diagnostics for xchat "xchat4.c", line 5: warning #20114-D: (SECURITY) Tainted value may be used in loop exit condition computation (procedure "copydata") ++ 'copydata' is called by 'main' at line 9 in file xchat1.c ++++ tainted value is returned from 'getlen' called by 'main' at line 7 in file xchat1.c "xchat3.c", line 4: warning #20111-D: (SECURITY) Tainted data may be used in data length computation (procedure "getdata") ++ 'getdata' is called by 'main' at line 8 in file xchat1.c ++++ tainted value is returned from 'getlen' called by 'main' at line 7 in file xchat1.c

page 11September 20, 2005 AADEBUG wlint option Enables several warnings in the compiler that provide lint like functionality. These compile time diagnostics can be very useful in detecting potential problems in the source code. To disable a specific warning introduced by +wlint, a +Wnnnn option can be used after the +wlint option.

page 12September 20, 2005 AADEBUG w64bit option Enable warnings that help detection of potential problems in converting 32 bit applications to 64 bit. The +w64bit option applies only to a 64 bit compile (using +DD64).

page 13September 20, 2005 AADEBUG 2005 Runtime Security Checks Prefer preventive checking via static analysis over runtime detection – runtime checks do not find the root cause of the problem – execution time overhead due to instrumentation

page 14September 20, 2005 AADEBUG 2005 New runtime check options +check=all|none|bounds|malloc|stack|uninit The +check=xxx options provide runtime checks to detect potential errors. These options introduce additional instructions and may significantly slow down the program. A failed check will result in the program aborting at runtime. The +check options need to be specified at both compile time and link time since they may require additional libraries to be linked into the user program.

page 15September 20, 2005 AADEBUG check=bounds Enables array bounds checking. This introduces additional runtime overhead for instrumentation of array references. The +check=bounds option applies to local and global array variables. It also applies to references to array fields of structs. It does not apply to arrays allocated dynamically using malloc or alloca. It also does not cover pointer dereferences.

page 16September 20, 2005 AADEBUG check=bounds #include int x[200]; int main () { int inp; scanf("%d", &inp); return x[inp]; /* Bounds check fails if inp 199. */ }

page 17September 20, 2005 AADEBUG check=bounds cc –v +check=bounds 1.c./a.out 200 Runtime Error: out of bounds in file 1.c at line 6 (0) 0x04001aa0 _rtc_raise_fault + 0x620 [a.out] (1) 0x _rtc_check_bounds + 0x490 [a.out] (2) 0x main + 0x1a0 [a.out] Bus error (core dumped)

page 18September 20, 2005 AADEBUG check=stack +check=stack[:frame|:variables|:none] This option enables runtime checks to detect illegal overwriting of the stack. This can be done for the whole stack frame or around individual stack variables. +check=stack:frame – The option enables runtime checks for illegal overwriting of the stack frame. +check=stack:variables This option enables runtime checks around variables on the stack.

page 19September 20, 2005 AADEBUG check=stack +check=stack:none – This option disables all runtime stack checking. +check=stack This default form of the option +check=stack is equivalent to +check=stack:variables at optimization levels 0 and 1. It is equivalent to +check=stack:frame for optimization level 2 and above. When a stack check fails at runtime, the stack trace is written to stderr and the program is aborted.

page 20September 20, 2005 AADEBUG check=stack:frame $ cat stack_frame.c int main() { int a[2]; int i=0; for (;i<=12; i++) a[i] = i; return a[1]; }

page 21September 20, 2005 AADEBUG check=stack:frame $ cc +check=stack:frame stack_frame.c $ a.out Runtime Error: stack overrun at 0x7fffee9c (0) 0x _rtc_raise_fault + 0x120 [./a.out] (1) 0x040010b0 main + 0x130 [./a.out] Abort(coredump)

page 22September 20, 2005 AADEBUG check=stack:variables $ cat stack_variables.c int main() { int a[12]; int i=0; for (;i<=12; i++) a[i] = i; return a[1]; }

page 23September 20, 2005 AADEBUG check=stack:variables $ cc +check=stack:variables stack_variables.c $ a.out Runtime Error: storage around end of variable "a" corrupted (0) 0x040013a0 _rtc_raise_fault + 0x120 [./a.out] (1) 0x04001da0 _rtc_check_guards + 0x460 [./a.out] (2) 0x main + 0x130 [./a.out] Abort(coredump)

page 24September 20, 2005 AADEBUG check=stack:variables The +check=stack:variables guards any array, struct/class/union, or object whose address is taken by putting head and foot guard values before and after these variables. If any part of the guards are changed the function will abort when it returns. The runtime error message for +check=stack:variables provides detailed info including the variable name, whether the begin or the end of the variable is corrupted, and the stack trace of the aborting function. This option also protects the stack frame, so there is no need to use both +check=stack:variables and +check=stack:frame,

page 25September 20, 2005 AADEBUG 2005 Detecting use of uninitialized variables Programming errors result in certain stack variables being used before they are initialized. The detection of the use of stack variables before they have been initialized can occur both at compile time and at runtime. Compile time warnings may occur for code paths which may not be taken at runtime. It is difficult for the programmer to separate out the true issues (warnings from code paths which will actually be taken) from the large number of warnings produced. Having a runtime check in the compiler which will trigger a runtime assertion only when there is actually a use of an uninitialized variable in the course of program execution will help during program development and testing. Use the option +check=uninit for this.

page 26September 20, 2005 AADEBUG check=uninit void Test6(int value) { int uninitialized; int var; switch (value) { case 3: uninitialized = 4; case 2: var = 5 * uninitialized; break; case 1: int *var2; var2= &uninitialized; var = 5 * uninitialized; break; }} int main(void) { Test6(1); }

page 27September 20, 2005 AADEBUG check=uninit $ cc -v -o./uninit +check=uninit test1.c $./uninit Runtime Error: use of uninitialized variable in file uninit.c line 15 variable uninitialized (0) 0x040015c0 _rtc_raise_fault + 0x120 [./uninit] (1) 0x _rtc_init_check + 0x120 [./uninit] (2) 0x Test6 + 0x1f0 [./uninit] (3) 0x main + 0x30 [./uninit] (4) 0x6c028de0 main_opd_entry + 0x50 [/usr/lib/hpux32/dld.so] ABORT instruction This will result in the program getting aborted due to the use of uninitialized variable "uninitialized" in line 15 in file test1.c

page 28September 20, 2005 AADEBUG check=uninit #include void Test6(int value) { int uninitialized; int var; int var6; switch (value) { case 3: uninitialized = 4; case 2: var = 5 * uninitialized; break; case 1: break; } var6 = 3 * uninitialized; printf("%d is the value of var6\n", var6); } int main(void) { Test6(1); }

page 29September 20, 2005 AADEBUG check=uninit $ cc -v -o./uninit1 +check=uninit test2.c $./uninit1 We will get the following runtime assertion: Runtime Error: use of uninitialized variable in file test2.c line 15 variable uninitialized (0) 0x _rtc_raise_fault + 0x120 [./uninit1] (1) 0x a0 _rtc_init_check + 0x120 [./uninit1] (2) 0x Test6 + 0x180 [./uninit1] (3) 0x c0 main + 0x30 [./uninit1] (4) 0x c0028de0 main_opd_entry + 0x50 [/usr/lib/hpux32/dld.so] ABORT instruction

page 30September 20, 2005 AADEBUG check=malloc Detects several kinds of memory problems: memory leaks heap memory bounds overrun and underrun freed memory read bad free and duplicate free and provide memory heap info

page 31September 20, 2005 AADEBUG check=malloc requirements The +check=malloc option is a link time option. It doesn't require recompiling the source files with this option, but the user should use the +check=malloc cc or aCC driver option to link the program. It requires the latest /opt/langtools/lib/hpux[32|64]/librtc.so, with latest wdb installed. malloc/free/realloc should be in a shared library so they can be overridden by librtc.so. In order to get the line number in the error message, the program should not stripped.

page 32September 20, 2005 AADEBUG check=malloc Uses librtc.so to do memory management and checking, the librtc.so is linked into the program as the first shared library. All the references to malloc/free functions are through the functions in librtc.so. The malloc() function records the allocation info and does appropriate initialization specified by rtc config (either default config or config file/config env variable ). The free() function checks the freed pointer against the allocation info to find bad frees, and does the memory bounds violation check if specified. It may also scramble the freed memory block to catch freed memory read. All actual malloc and free are done by the malloc and free in libc.so, so even if the program has its own version of malloc/free, they are not used for memory allocation and deallocation.

page 33September 20, 2005 AADEBUG check=malloc If a memory problem (such as bounds violation, bad free, out of memory) is found when doing allocation/de- allocation, the program will print out error message and abort the execution. User can change the rtc config to not abort on particular events and continue execution. At program exit time, librtc.so will check the memory info to find memory leaks. The leak info and memory corruption info are dumped to log files, the log file names are written to stderr.

page 34September 20, 2005 AADEBUG check=malloc $ cat memory_bug1.c #include void foo() { char *p; int i; p = (char *)malloc(10240); p += 2; free(p); // bad free and memory leak } int main() { foo(); return 0; }

page 35September 20, 2005 AADEBUG check=malloc $ cc +check=malloc malloc-bug1.c $ a.out warning: Memory corruption info is written to "./a.out mem". Abort on memory event RTC_BAD_FREE, see log file./a.out mem for detail (0) 0x6ca0e120 __rtc_free + 0x640 [/opt/langtools/lib/hpux32/librtc.so] (1) 0x04000a40 foo + 0xa0 [./a.out] (2) 0x04000a80 main + 0x20 [./a.out] (3) 0x6c13cde0 main_opd_entry + 0x50 [/usr/lib/hpux32/dld.so] Abort(coredump)

page 36September 20, 2005 AADEBUG check=malloc $ cat./a.out mem Attempt to free unallocated or already freed object at 0x404978a2 (0) 0x6ca14860 print_stack_trace_to_log_file + 0x220 [/opt/langtools/lib/hpux32/librtc.so] (1) 0x6ca15cf0 __rtc_event + 0x130 [/opt/langtools/lib/hpux32/librtc.so] (2) 0x6ca1a8e0 rtc_record_free + 0xdb0 [/opt/langtools/lib/hpux32/librtc.so] (3) 0x6ca0dee0 __rtc_free + 0x400 [/opt/langtools/lib/hpux32/librtc.so] (4) 0x04000a40 foo + 0xa0 [./a.out] (5) 0x04000a80 main + 0x20 [./a.out] (6) 0x6c53cde0 main_opd_entry + 0x50 [/usr/lib/hpux32/dld.so]

page 37September 20, 2005 AADEBUG check=malloc limitations The runtime memory check library maintains data about each allocated memory block, and does checks when the memory is freed, so there is some memory space and runtime overhead to the program using the service. The optimized program may have functions inlined into other functions, so the reported stack trace may not reflect the actual function which has memory leaked or corrupted. To get an accurate stack trace, the user needs to disable inlining with the "+d" option. Since the runtime memory checker uses gdb to print its report at exit time, if the user interrupts the program at that time the program may stop in gdb. To get the log files the program can not be traced, that means it can not be run under a debugger or be traced by tusc or caliper.

page 38September 20, 2005 AADEBUG 2005 For Further Information …