1. CS5123 Software Validation and Quality Assurance
Lecture 7
Non-Functional Testing

2. Non-Functional Testing
Performance Testing
Test whether the efficiency (time and space) of a software meets requirements
Security Testing
Test whether the software is vulnerable to attacks (special invalid inputs designed to control the software or reveal info from the software) 2

3. Performance Testing Load Testing Soak Testing Stress Testing
Spike Testing 3

4. Input loads 4 Size of inputs Number of inputs provided
An extremely long SQL query
An extremely large html file for a browser
Number of inputs provided
Number of students supported in a school management system
Number of web pages opened in a browser
Frequency of inputs provided
Number of SQL queries made per second
Number of Http requests made per second 4

5. Performance Measures
Input Lag
Response Time
Throughput 5

6. Performance Measures 6

7. Load Testing
Provide input under the maximal designed load of software and observe behavior
Purpose:
See whether the software works normally
Find potential bottlenecks of performance: Profiling
Instrument each major component (e.g., method) to see how much time / memory is spent on it
Sampling is sometimes used to reduce overhead 7

8. Load Testing 8 Test steps Determine the content of inputs
Usually can be a large amount of identical or similar inputs
The input can be simple or very complex (to check the performance of software when handling complex input)
Determine the frequency of input feeding
Determine how long the input feeding lasts
Design Load
Input Load 8
time

9. Input feeding
A multi thread program to feed inputs randomly in a given period of time
Sometimes require multiple machines to feed inputs
Usually only consider valid inputs 9

10. Stress Testing
Provide input OVER the maximal designed load of software and observe behavior
The software is expected to fail
Purpose:
Observe when (how much load) the software is going to fail
Observe the how the failure looks like: crash? CPU or memory used up? Can be recovered or not?
Observe whether the system can partially work when failure happens 10

11. Stress Testing
Illustration
Design Load
Input Load
time 11

12. Soak Testing
Provide heavy input load (slightly under designed maximal load) for a long time
Purpose:
Testing for how long time the software can work normally under heavy input load
Usually memory and disk oriented
Observe the memory / disk usage trend (abnormal increase in the usage) 12

13. Soak Testing
Illustration
Design Load
Input Load
time 13

14. Spike Testing
Provide extremely heavy input load (OVER designed maximal load) for a very short time
Purpose:
Test how the software can handle a input load burst
Probable expected behavior:
Temporarily refuse inputs that cannot be handled
Provide some temporary services for the inputs to wait until the burst ends 14

15. Spike Testing
Illustration
Design Load
Input Load
time 15

16. Performance Diagnosis
Find out why performance problems happen
Figure out how to optimize software to achieve higher performance 16

17. Profilers 17

18. Memory Profilers 18

19. Performance Testing: Review
Load Testing
Stress Testing
Soak Testing
Spike Testing 19

20. Tools for Test Measurement
EclEmma
Update site:
Install in eclipse
Setup coverage configuration: do not check the test folder
Run test using Emma coverage as…
Read test coverage
Enhance test coverage 20

21. Tools for Test Measurement
VisualVM
Start VisualVM
Start a Java software
Open the tab for the process
Do profiler for memory and computation 21

22. Demo
Usage of JMeter to perform load testing for web application and databases 22

23. Security Testing 23 Major security concerns Vulnerabilities
Penetration Testing 23

24. Major Security Concerns
Undermine usability
DOS attacks
Peculiar inputs causing crashes, bloats, …
Information Leaking
SQL Injection, Cross-site Scripting, unencrypted data, side channels, …
Command and Control
OS Injection, Cross-site Scripting, Return Oriented Programming, … 24

25. Vulnerabilities 25 Avoid common vulnerabilities Buffer Overflow
Injection
Cross-Site Scripting 25

26. Buffer Overflow 26 Quite many languages (C, C++) are memory unsafe
You define a buffer, and it is your responsibility to keep your data in the buffer
If you read or write to the place out of a buffer
Semantic errors
Crashes
What else? Anything related to security?
char buffer[12]; 26

27. Review of OS course: call stacks
Function calls are traced by call stacks
int main(int argc, char args**){
int result;
if(argc >= 1){f(args[0]);} }
void f(char* data){
char buffer[12];
strcpy(buffer, data)
if(g()){return;}
else{…}
bool g(){
... g f f f
main
main
main
main

28. Call stack of the function f
The local variable buffer
The parameter data
The return address to go back
to the call-site at main function
char[12] buffer

29. Feed in a valid input
Example “username”
char[12] buffer

30. Feed in an invalid input
Example “usernameusername”
The parameter data is covered
So it is no longer usable
The return value is covered
So can not return normally
Still just a bug
Minor security problem
Undermines usability
char[12] buffer

31. Feed in a malicious input
Idea to do the trick
Feed in an input with 20 chars
Cover the return address
f() will return to the code we
Specify
Consider the program is on a
server, accessing user requests
How to make it possible?
Where to put the code?
How to specify the return
value to our code?
char[12] buffer

32. Feed in a malicious input
Use the buffer itself to
store the code
Set the return value
to the buffer address
Example
Run exec(“/bin/sh”) to open
a shell
Translate to machine code
char[12] buffer
mov $a0 15
mov $a1 data
syscall
data: /, b, i, n, /, s, h
0x20, 0x42, 0x00, ...

33. Feed in a malicious input
Other issues
How to know the address
of buffer[]:
Programs are executed
in virtual memory, so install
the software and check
memory state
Buffer is too small to hold
your code?
Jump through return value to
the stack frame of parent function
char[12] buffer

34. The state of practice
Buffer overflow is very common in C / C++ programs
About 50% of new attacks are related to buffer overflow
Known bugs are being exploited from time to time 34

35. How to deal with buffer overflow
Boundary check for input-reachable buffers
Not so easy in practice
Check too many places: slow the software down
Check too few places: buffer overflow risk
Automatic supports
Buffer Overflow Detection: libsafe, stackguard, …
Runtime protection: weak memory safe
Runtime protection: split stack 35

36. A real-world example
If you are interested
Here is a real world example: 36

37. Injection 37 Directly inject user input into code to be executed
SQL Injection
Inject code to SQL queries
OS Injection
Inject code to OS commands 37

38. SQL Injection 38 An example A student information system
You can query your grade for certain course, year, …
You login to your session, and say you are going to search for the grade of “CS5123”
What does the server do? 38

39. SQL Injection 39 The malicious Input
We want to inject code into the SQL query
Say we want it to be “select * from Grade”
It is the same with “select * from Grade where username = ‘you’ and course = ‘CS5123’ or ‘a’ = ‘a’” 39

40. OS Injection 40 Quite Similar
Consider a server is going to make a dir for you as a new user, and it will execute exec(“mkdir path/to/” + username)
What username you should make up?
An example:
HahaGotyou | \bin\sh 40

41. Injection Protection
Injection works by passing user inputs into back-end engines
Can we simply cut off the path?
Definitely NO
We have to do some filtering
We are going to work on the example:
select * from Grade where username = ‘you’ and course = ‘CS4723’/**/or/**/‘a’=‘a’ 41

42. User Input Filtering What to filter? 42
or ? => “oorr” can bypass it
Space? => use /**/ can bypass it
Quotes? A little bit difficult, we can search by year, and use year = 2009 or 1=1
Want more?
See
select * from Grade where username = ‘you’ and course = ‘CS4723’ or ‘a’ = ‘a’ 42

43. User Input Filtering: Other Issues
Functioning of the software
Filter ‘or’ will affect username ‘George’
Cannot filter space when space is allowed for the field
Other string manipulations
In web applications, there are HTML/URL escape characters
&quot for “, &#39 for ‘, &nbsp / %20 for space, …
Replacing escaping characters are common
So &#39 may be used if quotes are disabled…
Always be clear about how user inputs flows in your code 43

44. Cross Site Scripting
One of the most popular attacks to web applications
Everything is about where the input goes to
This time it goes to a web page
This becomes more popular with so-called web 2.0 (let users do the work, e.g., wiki, youtube, blogs) 44

45. XSS: Scenario 45

46. Example
Bob wants to get all the login information of his friends in a social network
So Bob writes a blog, in the blog, he writes:
xxxxxxx, xxxx, getcookie())</script>
Now Mary reads the blog, so the script runs, Bob will get the cookie, and will be able to login with Mary’s cookie… 46

47. Protection against XSS
Limit the usage of cookies: may result in much inconvenience
Quite similar to SQL Injection
Try to filter dangerous things such as “<script>” from user’s input
Also quite similar to SQL Injection
There are a lot of ways to bypass the filtering, so always hard to do a correct filtering
Even harder because HTML is more complex than SQL, and much more web page generations than database query points… 47

48. Core idea: Devil inputs
Software Security is almost all about the malicious inputs
Buffer Overflow, Injection, and XSS accounts for 70% to 80% of attacks each year…
Also for DOS (Denial of Service) attacks
An example: you may want to filter ‘\’ for security reasons, but if so, handling a input like ‘\\\....\\\\’ with n ‘\’s will take n2 CPU time…
Consider all possible values of user inputs
Never make assumptions to user inputs 48

49. Penetration Testing
Random testing (or fuzzing) is often useful for security testing, because it can generate inputs that you cannot imagine
Have security checks during the testing
Buffer Overflow: whether any “out of boundary” happens
Use boundary checker in testing, and disable them in distribution
SQL Injection & XSS: whether user inputs reach syntax tree part of the HTML or SQL code
Use taints during testing to track the user inputs along the execution 49

50. Review of Non-Functional Testing
Performance Testing
Test whether the efficiency (time and space) of a software meets requirements
Security Testing
Test whether the software is vulnerable to attacks (special invalid inputs designed to control the software or reveal info from the software) 50