1. Introduction to White-box Testing
Frank Xu, Ph.D.
Department of Computer and Information Science
Gannon University

2. Motivation Software problems and related faults: Airbus A380 (2006)
Source: The top 10 IT disasters of all time:
Software problems and related faults:
Airbus A380 (2006)
suffers from incompatible software issues
At least two years behind schedule
Ariane 5 rocket (1996)
Data conversion (64-bit to 16-bit format)
Cost $500M
Therac 25 - Radiation therapy machine (1993)
Overdosing patients during radiation
Cost 6 lives

3. Testing
One of the practical methods commonly used to detect the presence of errors (failures) in a computer program is to test it for a set of inputs.
The output
is correct?
Our program
I1, I2, I3, …, In, …
Expected results
= =?
Actual results
Black-box testing
“Inputs”
White-box testing

4. Exercise
Implementing a method to solve Triangle problem

5. What is Triangle Problem?

6. How to test it? String getTriangleType (int a, int b, int c){
if((a<b+c) && (b<a+c) && (c<a+b)){
if (a==b && b==c)
return “Equilateral ”;
else if (a!=b && a!=c &&b!=c)
return “Scalene ”;
else
return “Isosceles” ; }
return “NotATriangle “;

7. Steps for White-box Testing
assertEquals(“equalateral”, triangle.getTriangleType(7,7,7))
Step 1
Step 2
assertEquals(“Isosceles ”, triangle.getTriangleType(6,6,8))
…..
Source Code
Control Flow
Diagram
Paths (based on coverage)
Junit Test cases

8. if ... else Control Flow if (condition) statement1; else statement2;
if (a==b && b==c)
return “Equilateral ”;
else if (a!=b && a!=c &&b!=c)
return “Scalene ”;
else
return “Isosceles” ;
if (condition) statement1;
else statement2;

9. Control Flow Diagram

10. Branch Coverage Method
Identify all the decisions
Count all the branches from each of the decisions
Pick the minimum number of paths that will cover all the branches from the decisions.

11. How many paths? For decision coverage McCabe’s Cyclomatic number
# of binary decisions + 1
Input for path Equilateral: (7,7,7)
assertEquals (“Equilateral”, triangle.getTriType(7,7,7))

12. Related to Cyber Security
Include
Software security
Network security
Data security
Common approach for software security
Testing for security
E.g., testing for web security
Test cases are needed for all testing

13. Auto-Generate Test Cases is Challenging
Challenging 1:inputs generation
Challenging 2: Oracle Problem
assertEquals (“Equilateral”, triangle.getTriType(7,7,7)) ?
assertEquals (“Isosceles ”, triangle.getTriType(6,6,8))
…..

14. Mining Decision Trees as Test Oracles for Java Bytecode
Xu, W., Ding, T., Xu, D., Rule-based Test Input Generation From Bytecode, Proc. of the 8th International Conference on Software Security and Reliability, pp , San Francisco, CA, June 2014.
Xu, W., Ding, T., Wang, H., Xu. D., Mining Test Oracles for Test Inputs Generated from Java Bytecode, Proc. of the 37th Annual International Computer Software & Applications Conference, pp , Kyoto, Japan, July 2013

15. Our Approach Rule-based search method to generate inputs
Seed value adjust seed values based on rules
(5,7,8) for Isosceles
Adjust input values
a==b
(7,7,8) (5,5,8)
Using heuristic model for test oracle
A new data mining approach to build a heuristic behavioral model (in the form of decision tree)

16. Revisit: Triangle Problem

17. Java is Complex Statement Condition contains comparison and expression
a <b+c (Java)
Condition
(a<b+c) && (b<a+c) && (c<a+b)

18. Jimple Java simpler version Simplify Statement Simplify condition
Intermediate representation of bytecode
Simplify Statement
a <b+c (Java)
[1] $d=b+c and [2] a <$d (Jimple)
Simplify condition
(a<b+c) && (b<a+c) && (c<a+b) (Java)
Jimple
if (a<b+c) {
if (b<a+c) {
if(c<a+b) …
}}}

19. Training data Validation data Input Generator Input Generator (7,7,7)
Path 1
[1]→[2]→[3]→[4]→[5] f >[6]→[7] f >[8]→[9] f >[10] f >[11] f >[12] 2
[1]→[2]→[3]→[4]→[5] f >[6]→[7] f >[8]→[9] f >[10] t >[13] f >[14] f >[15] t >[17] 3
,,, .. 8
[1]→[2]→[3]→[4]→[5] f >[6]→[7] f >[8]→[9] f >[10] t >[13] f >[14] f >[15] f >[16] 9
[1]→[2]→[3]→[4]→[5] f >[6]→[7] f >[8]→[9] f >[10] f >[11] t >[13] f >[14] f >[15] f >[16] 10
[1]→[2]→[3]→[4]→[5] f >[6]→[7] f >[8]→[9] t >[18] 11
[1]→[2]→[3]→[4]→[5] f >[6]→[7] t >[18] 12
[1]→[2]→[3]→[4]→[5] t >[18]
CFG
Generator
Path generator
Input Generator
Input Generator ID a b c 1 7 2 11 3 8 19 4 5 22 9 … 13 33 45 14 52 30 15 31 47 16 27 28
Validation data
(7,7,7)
Model miner
Equilateral
Training data

20. Input Generator

21. Path: [1]→[2]→[3]→[4]→[5] t >[18]
Search an input that make predicate [5]:i0>=$i3 to true
a>=b+c (NotATriangle)

22. Apply Rules to a Predicate Tree for Generating Test Inputs10 4 7

23. Expected Evaluation OutcomesID
Predicate
Expected Evaluation Outcomes
Advising Rules
i0 > i1
(i0 > i1) = true
(i0 ↑, i1) (i0, i1↓)
(i0 > i1) = false
(i0 ↓,b) (i0, i1↑)
i0 == i1
(i0==i1) = true
(i0 ↓D, i1)( i0, i1↑D)
(i0== i1)= false
(i0 ↑,i1) (i0, i1↑) (i0↓,i1) (i0, i1↓)
i2 = i0 + i1
i2 ↑
(i0 ↑, i1) (i0, i1↑)
i2 ↓
(i0 ↓, i1) (i0, i1↓)
i2 = i0 - i1
(i0 ↑, i1) (i0, i1↓)
(i0 ↓, i1)( i0, i1↑)
i2 = i0 * i1
(i0>0, i1>0)
(i0↑, i1) (i0, i1↑)
(i0↓, i1) (i0, i1↓)
i2 = i0 / i1
(i0>0, i0 > 0)
(i0↑, i1) (i0, i1↓)
(i0-↓ i1)( i0, i1↑) ..
s0>s1
(s0 >s1) = true
(s0[k] ↑, s1) (s0, s1[l] ↓)
(s0 > s1) = false
(s0[k] ↓,s2) (s0, s1[l] ↑)

24. Model Miner

25. Jimple Predicates and Attributes of Triangle Program
Source Code
i0 > = $i3
a > = b + c
i1 > = $i4
b > = a + c
i2 > = $i5
c > = a + b
i0 != i1
a != b
i1 != i2
b != c
i0 == i1
a == b
i0 == i2
a == c
i1 == i2
b == c
Input
(a=7,b=7,c=7) f t
For a given input, predicates produce a set of T or F values

26. Covert Inputs Using Attributes
Test input ID a b c
a>=b+c
b>=a+c
c>=a+b
a!=b
b!=c
a==b
a==c
b==c
Output 1 7 f t 2 11 3 4 8 19 … 13 33 45 14 52 30 15 31 47 16 27 28 22

27. The key idea of the algorithm is to Tool
Test input ID a b c
a>=b+c
b>=a+c
c>=a+b
a!=b
b!=c
a==b
a==c
b==c
Output 1 7 f t 2 11 3 4 8 19 … 13 33 45 14 52 30 15 31 47 16 27 28 22
C4.5 mining algorithm
The key idea of the algorithm is to
calculate the highest normalized information gain of attributes and then build a decision node that splits on the attributes
Tool
Weka 3:

28. Empirical Study

29. Three Study Subjects Unit Under Testing Line of Code
Number of Predicates
Java
Jimple
(Allow duplications)
Attributes
(No duplication)
Triangle 22 27 3 8
Next Date 48 51 9 19
Vending Machine 82 68 21 41 10

30. Goal of Empirical Studies
Measure fault detection capability
# mutants killed /#mutants *100%

31. Measure fault detection capability: Process
Mutation Operator
Examples
Category ID
Type
Original
Replaced
Arithmetic Operations 1
Arithmetic Operator Replacement
a + b
a - b 2
Arithmetic Operator Insertion
b + c
-b + c
Relations 3
Relational Operator Replacement
a != b
a == b
Conditions 4
Conditional Operator Replacement
(a==b) && (b==c)
(a==b) || (b==c)
Constants 5
Constant Value Modification
s = a
s = b
Return Values 6
Return Value Modification
return s
return s’
Step 1: Implant mutants
Step 2: Build a decision tree model
Step 3: Find mismatches
Find possible causes
Step 4: Calculate fault detectability
Insert bug
Create mutant and insert mutants
Two possible causes
Found bugs
assertEquals(“Equilateral”, new Trianlge(7,7,7).getTriType())
Model is not correct
assertEquals (“Isosceles”, new Trianlge(7,7,7).getTriType())
Faulty version
Find mismatches

32. Results of fault detection evaluation
Mutation ID
# of Mutants
# of Tests Executed
Oracle Results
# Mutants Discovered
Faults in Models S D U
Triangle 51
123 48 3
NextDate 54 94 95
109 15 16 18 2
Vending 45
465
Total
150
886
142 8
statement coverage (S), decision coverage (D), and un-reduced decision coverage (U),

33. Conclusions The first attempt to mine decision tree models
from auto-generated test inputs based on static analysis of Java bytecode
Our empirical study indicates that
using the mined test oracles, average 94.67% mutants are killed by the generated test inputs.
Future plan
Empirical study

34. Thanks

35. Related Work
Lo et al. (Lo, Cheng, Han, Khoo, & Sun, 2009), Milea et al. (Milea, Khoo, Lo, & Pop, 2012) mines a set of discriminative features capturing repetitive series of events from program execution traces. These features are then used to train a classier to detect failures.
Bowring et al. (Bowring, Rehg, & Harrold, 2004) models program executions as Markov models, and a clustering method for Markov models that aggregates multiple program executions into effective behavior classifiers.
(Pacheco & Ernst, 2005) Pacheco and Ernst build an operational model from observations of the software running properly. The operation model includes object invariants and properties. The object invariants are the conditions hold on entry and exit of all public methods.
Our approach generates and classifies inputs based on the internal structure of the UUT.
Briand (Briand, 2008) has proposed the use of machine learning techniques - including decision trees - for the test oracle problem.
The decision tree model he has proposed is manually built from software requirements.

36. Future Research Direction
Requirements Engineering & Natural language Process
Generating UML diagrams, e.g., Use case, Class diagram
Validating SRS
Deriving test cases from SRS
Software Design & Social Networks Analysis
Utilizing SSA for analyzing communication diagram, class diagram, and sequence diagram for improving the quality of the software
Software Implementation & Big Data
Mining repository for software quality assurance using Hadoop
Software Testing & Mobile/Cloud Application
Testing mobile applications and distributed applications

37. Test Oracle Overview

38. Control Flow Diagram

39. Build Variable Dependency Tree (VDT)

40. Bio – Frank Xu Education Working Experience
Ph.D. in Software Engineering North Dakota State University
M.S. in Computer Science Towson University
B.S. in Computer Science Southeast Missouri State University
Working Experience
GE Transportation, present, Consultant of Locomotive Remote Diagnostics Service Center
Gannon University, Present, Assistant/Associate Professor of Software Engineering, Director of Keystone Software Development Institute
University VA –Wise, , Assistant Professor of Software Engineering
Swanson Health Products, 2005 ~ 2007, Sr. Programmer Analyst
Volt Information Science Inc., 2004 ~ 2005, Software Engineer

41. Teaching Software engineer in UML Requirement engineering
Software Design
Software Quality Assurance
Software Maintenance
Object-Oriented Modeling
Personal Software Process
Advance Programming in Java
Mobile Application Programming
Multi-Tiered Systems
Dynamic Web Programming
Operating System
Discrete Math

42. Source: Student Evaluation Report
Teaching Evaluation
Source: Student Evaluation Report

43. Research

44. Publications and Citations
Year
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
Total
Conference 3 2 1 4 24
Journal 8 5 6 32
Source: Google scholar: