Hybrid Concolic Testing Rupak Majumdar Koushik Sen UC Los Angeles UC Berkeley

Automated Test Generation Studied since 70’s  King 76, Myers years have passed, and yet no effective solution What Happened???

Automated Test Generation Studied since 70’s  King 76, Myers years have passed, and yet no effective solution What Happened???  Program-analysis techniques were expensive  Automated theorem proving and constraint solving techniques were not efficient

Automated Test Generation Studied since 70’s  King 76, Myers years have passed, and yet no effective solution What Happened???  Program-analysis techniques were expensive  Automated theorem proving and constraint solving techniques were not efficient In the recent years we have seen remarkable progress in static program- analysis and constraint solving  SLAM, BLAST, ESP, Bandera, Saturn, MAGIC

Automated Test Generation Studied since 70’s  King 76, Myers years have passed, and yet no effective solution What Happened???  Program-analysis techniques were expensive  Automated theorem proving and constraint solving techniques were not efficient In the recent years we have seen remarkable progress in static program- analysis and constraint solving  SLAM, BLAST, ESP, Bandera, Saturn, MAGIC Question: Can we combine program analysis with classical testing techniques to Scale Automated Test Generation?

Our Approach Concolic Testing: 1.Combines Dynamic and Static Program Analysis 2.Exhaustive 3.Fails to scale Random Testing: 1.Fast 2.Non-exhaustive 3.Redundant Executions and poor coverage + = Hybrid Concolic Testing

Goals of Test Generation (Simplified) ‏ Generate test inputs Execute program on generated test inputs Catch assertion violations Problem: how to ensure that all reachable statements are executed Solution:  Explore all feasible execution paths

Execution of Programs All Possible Execution Paths  Binary tree Computation tree  Internal node ! conditional statement execution  Edge ! execution of a sequence of non- conditional statements  Each path in the tree represents an equivalence class of inputs F T FF F F T T T T T T Condition al Statement s Non- Condition al Statement s

Fuzz (Random) Testing Random testing  Random Testing [Bird and Munoz 83]  Fuzz testing Windows NT [Forrester and Miller 00]  QuickCheck [Claessen & Hughes 01]  JCrasher [Csallner and Smaragdakis 04]  RUTE-J [Andrews et al. 06]  Randoop [Pacheco et al. 07] Very low probability of reaching an error Problematic for complex data structures

Fuzz (Random) Testing Random testing  Random Testing [Bird and Munoz 83]  Fuzz testing Windows NT [Forrester and Miller 00]  QuickCheck [Claessen & Hughes 01]  JCrasher [Csallner and Smaragdakis 04]  RUTE-J [Andrews et al. 06]  Randoop [Pacheco et al. 07] Very low probability of reaching an error Problematic for complex data structures Example ( ) { s = readString(); if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && s[4]==‘2’ && s[5]==‘0’ && s[6]==‘0’ && s[7]==‘7’) { printf(“Am I here?”); } Example ( ) { s = readString(); if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && s[4]==‘2’ && s[5]==‘0’ && s[6]==‘0’ && s[7]==‘7’) { printf(“Am I here?”); } Input domain = {‘0’, ‘2’, ‘7’, ‘C’, ‘E’, ‘I’, ‘S’} Probability of reaching printf = 7 -8 » 10 -7

Fuzz (Random) Testing Random testing  Random Testing [Bird and Munoz 83]  Fuzz testing Windows NT [Forrester and Miller 00]  QuickCheck [Claessen & Hughes 01]  JCrasher [Csallner and Smaragdakis 04]  RUTE-J [Andrews et al. 06]  Randoop [Pacheco et al. 07] Very low probability of reaching an error Problematic for complex data structures Example ( ) { s = readString(); if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && s[4]==‘2’ && s[5]==‘0’ && s[6]==‘0’ && s[7]==‘7’) { printf(“Am I here?”); } Example ( ) { s = readString(); if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && s[4]==‘2’ && s[5]==‘0’ && s[6]==‘0’ && s[7]==‘7’) { printf(“Am I here?”); } Input domain = {‘0’, ‘2’, ‘7’, ‘C’, ‘E’, ‘I’, ‘S’} Probability of reaching printf = 7 -8 » Fast and Inexpensive

Concolic Testing Combine concrete testing (concrete execution) and symbolic testing (symbolic execution) [PLDI’05, FSE’05, FASE’06, CAV’06, HVC’06] Concrete + Symbolic = Concolic

Example int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; }

Example ERROR 2*y == x x > y+10 Y Y N N int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; }

Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 22, y = 7x = x 0, y = y 0

Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 22, y = 7, z = 14 x = x 0, y = y 0, z = 2*y 0

Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 22, y = 7, z = 14 x = x 0, y = y 0, z = 2*y 0 2*y 0 != x 0

Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } Concrete Execution Symbolic Execution concrete state symbolic state path condition 2*y 0 != x 0 Solve: 2*y 0 == x 0 Solution: x 0 = 2, y 0 = 1 x = 22, y = 7, z = 14 x = x 0, y = y 0, z = 2*y 0

Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 2, y = 1x = x 0, y = y 0

Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 2, y = 1, z = 2 x = x 0, y = y 0, z = 2*y 0

Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 2, y = 1, z = 2 x = x 0, y = y 0, z = 2*y 0 2*y 0 == x 0

Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 2, y = 1, z = 2 x = x 0, y = y 0, z = 2*y 0 2*y 0 == x 0 x 0 · y 0 +10

Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 2, y = 1, z = 2 x = x 0, y = y 0, z = 2*y 0 Solve: (2*y 0 == x 0 ) Æ (x 0 > y )‏ Solution: x 0 = 30, y 0 = 15 2*y 0 == x 0 x 0 · y 0 +10

Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 30, y = 15x = x 0, y = y 0

Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 30, y = 15x = x 0, y = y 0 2*y 0 == x 0 x 0 > y Program Error

Explicit Path (not State) Model Checking Traverse all execution paths one by one to detect errors  assertion violations  program crash  uncaught exceptions combine with valgrind to discover memory errors F T FF F F T T T T T T

Explicit Path (not State) Model Checking Traverse all execution paths one by one to detect errors  assertion violations  program crash  uncaught exceptions combine with valgrind to discover memory errors F T FF F F T T T T T T

Explicit Path (not State) Model Checking Traverse all execution paths one by one to detect errors  assertion violations  program crash  uncaught exceptions combine with valgrind to discover memory errors F T FF F F T T T T T T

Explicit Path (not State) Model Checking Traverse all execution paths one by one to detect errors  assertion violations  program crash  uncaught exceptions combine with valgrind to discover memory errors F T FF F F T T T T T T

Explicit Path (not State) Model Checking Traverse all execution paths one by one to detect errors  assertion violations  program crash  uncaught exceptions combine with valgrind to discover memory errors F T FF F F T T T T T T

Explicit Path (not State) Model Checking Traverse all execution paths one by one to detect errors  assertion violations  program crash  uncaught exceptions combine with valgrind to discover memory errors F T FF F F T T T T T T

Limitations Path Space of a Large Program is Huge  Path Explosion Problem Entire Computation Tree

Limitations Path Space of a Large Program is Huge  Path Explosion Problem Explored by Concolic Testing Entire Computation Tree

Limitations: A Comparative View Concolic: Broad, shallowRandom: Narrow, deep

Limitations: Example Example ( ) { 1: state = 0; 2: while(1) { 3: s = input(); 4: c = input(); 5: if(c==‘:’ && state==0) state=1; 6: else if(c==‘\n’ && state==1) state=2; 7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) { COVER_ME:; } Example ( ) { 1: state = 0; 2: while(1) { 3: s = input(); 4: c = input(); 5: if(c==‘:’ && state==0) state=1; 6: else if(c==‘\n’ && state==1) state=2; 7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) { COVER_ME:; } Similar code in Text editors (vi) ‏ Parsers (lexer) ‏ Event-driven programs (GUI) ‏ Want to hit COVER_ME input() denotes external input Can be hit on an input sequence s = “ICSE” c : ‘:’ ‘\n’

Limitations: Example Example ( ) { 1: state = 0; 2: while(1) { 3: s = input(); 4: c = input(); 5: if(c==‘:’ && state==0) state=1; 6: else if(c==‘\n’ && state==1) state=2; 7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) { COVER_ME:; } Example ( ) { 1: state = 0; 2: while(1) { 3: s = input(); 4: c = input(); 5: if(c==‘:’ && state==0) state=1; 6: else if(c==‘\n’ && state==1) state=2; 7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) { COVER_ME:; } Pure random testing can get to state = 2 But difficult to get ‘ICSE’ as a Sequence Probability 1/(2 8 ) 6 » 3X Conversely, concolic testing can generate ‘ICSE’ but explores many paths to get to state = 2

Hybrid Concolic Testing Interleave Random Testing and Concolic Testing to increase coverage Motivated by similar idea used in VLSI design validation: Ganai et al. 1999, Ho et al. 2000

Hybrid Concolic Testing Interleave Random Testing and Concolic Testing to increase coverage while (not required coverage) { while (not saturation) ‏ perform random testing; Checkpoint; while (not increase in coverage) ‏ perform concolic testing; Restore; }

Hybrid Concolic Testing Interleave Random Testing and Concolic Testing to increase coverage while (not required coverage) { while (not saturation) ‏ perform random testing; Checkpoint; while (not increase in coverage) ‏ perform concolic testing; Restore; } Deep, broad search Hybrid Search

Hybrid Concolic Testing Random Phase  ‘$’, ‘&’, ‘-’, ‘6’, ‘:’, ‘%’, ‘^’, ‘\n’, ‘x’, ‘~’ … Saturates after many (~10000) iterations In less than 1 second COVER_ME is not reached Example ( ) { 1: state = 0; 2: while(1) { 3: s = input(); 4: c = input(); 5: if(c==‘:’ && state==0) state=1; 6: else if(c==‘\n’ && state==1) state=2; 7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) { COVER_ME:; } Example ( ) { 1: state = 0; 2: while(1) { 3: s = input(); 4: c = input(); 5: if(c==‘:’ && state==0) state=1; 6: else if(c==‘\n’ && state==1) state=2; 7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) { COVER_ME:; }

Hybrid Concolic Testing Random Phase  ‘$’, ‘&’, ‘-’, ‘6’, ‘:’, ‘%’, ‘^’, ‘\n’, ‘x’, ‘~’ … Saturates after many (~10000) iterations In less than 1 second COVER_ME is not reached Concolic Phase  s[0]=‘I’, s[1]=‘C’, s[2]=‘S’, s[3]=‘E’ Reaches COVER_ME Example ( ) { 1: state = 0; 2: while(1) { 3: s = input(); 4: c = input(); 5: if(c==‘:’ && state==0) state=1; 6: else if(c==‘\n’ && state==1) state=2; 7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) { COVER_ME:; } Example ( ) { 1: state = 0; 2: while(1) { 3: s = input(); 4: c = input(); 5: if(c==‘:’ && state==0) state=1; 6: else if(c==‘\n’ && state==1) state=2; 7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) { COVER_ME:; }

Hybrid Concolic Testing 4x more coverage than random testing 2x more coverage than concolic testing

Results

Results: Red Black Tree test_driver() ‏ RedBlackTree rb = new RedBlackTree(); while(1) { choice = input(); data = input(); switch(choice) { case 1: rb.add(data); break; case 2: rb.remove(data); break; case 3: rb.find(data); break; default: rb.add_if_not_member(data); break; }

Results

Summary Concolic Testing Random Testing

Summary Concolic Testing Random Testing Hybrid Concolic Testing

Thank You!