1. Supannika Koolmanojwong CSCI 577 Ref: Qi Li’s lectures
Software Testing
Supannika Koolmanojwong
CSCI 577
Ref: Qi Li’s lectures

2. Continuous Integration
Personas
CRUD Analysis
MoSCoW
Software Test Automation
TDD
Agile Testing
SW design pattern
Walkthrough
V&V standard
Defect Tracking Tool
Code Inspection
“Bug” is found
Software Reliability
Software Metrics
HCI

3. Positive and Negative Testing
Positive Testing
Do “normal” user actions
Find cases where the program does not do what it is supposed to do
Test with valid input
Negative Testing
Do “abnormal” user actions
Find cases where the program does things it is not supposed to do
Test with invalid input

4. Outline
Software Test in General
Value-based Software Test

5. Most Common Software problems
Incorrect calculation
Incorrect data edits & ineffective data edits
Incorrect matching and merging of data
Data searches that yields incorrect results
Incorrect processing of data relationship
Incorrect coding / implementation of business rules
Inadequate software performance

6. Confusing or misleading data
Software usability by end users & Obsolete Software
Inconsistent processing
Unreliable results or performance
Inadequate support of business needs
Incorrect or inadequate interfaces
with other systems
Inadequate performance and security controls
Incorrect file handling

7. Cost to fix faults 60* to 100* 1.5* to 6* 1* Cost Definition
Development
Post Release

8. Objectives of testing
Executing a program with the intent of finding an error.
To check if the system meets the requirements and be executed successfully in the Intended environment.
To check if the system is “ Fit for purpose”.
To check if the system does what it is expected to do.

9. A good test :
A good test case is one that has a probability of finding an as yet undiscovered error.
A successful test is one that uncovers a yet undiscovered error.
A good test is not redundant.
A good test should be “best of breed”.
A good test should neither be too simple nor too complex.

10. Objective of a Software Tester
Find bugs as early as possible and make sure they get fixed.
To understand the application well.
Study the functionality in detail to find where the bugs are likely to occur.
Study the code to ensure that each and every line of code is tested.
Create test cases in such a way that testing is done to uncover the hidden bugs and also ensure that the software is usable and reliable

11. How do you know you are a good tester?b.
sources: Google images

12. How do you know you are a good tester?
Signs that you are dating a tester
Your love letters get returned to you marked up with
red ink, highlighting your grammar and spelling
mistakes.
When you ask him how you look in a dress, he’ll actually tell you.
He won’t help you change a broken light bulb because his job is simply to report and not to fix.
He’ll keep bringing up old problems that you’ve since resolved just to make sure that they’re truly gone

13. Static and Dynamic Verification
Software reviews, inspections and walkthroughs - Concerned with analysis of the static system representation to discover problems (static verification)
Software testing with test cases - Concerned with exercising and observing product behaviour (dynamic verification)
The system is executed with test data and its operational behaviour is observed

14. Inspection Fagan Inspection Process
Planning  Overview meeting  Preparation  Inspection meeting  Rework  Follow-up
Inspection roles
Author, Moderator, Reader, Recorder, Inspector
Inspection types
Code review, peer review
Inspect
Req Spec, Sys Arch, Programming, Test scripts

15. Inspections and testing
Inspections and testing are complementary and not opposing verification techniques
Both should be used during the V & V process
Inspections can check conformance with a specification but not conformance with the customer’s real requirements
Inspections cannot check non-functional characteristics such as performance, usability, etc.

16. Test data and test cases
Test data Inputs which have been devised to test the system
Test cases Inputs to test the system and the predicted outputs from these inputs if the system operates according to its specification

17. Methods of testing Test to specification: Test to code: Black box
Data driven
Functional testing
Code is ignored: only use specification document to develop test cases
Test to code:
Glass box/White box
Logic driven testing
Ignore specification and only examine the code.

18. Types of Testing – (Jokes)
Aggression Testing: If this doesn’t work, I’m gonna kill somebody.
Compression Testing: []
Confession Testing: Okay, Okay, I did program that bug.
Congressional Testing:Are you now, or have you ever been a bug?
Depression Testing:If this doesn’t work, I’m gonna kill myself.
Egression Testing: Uh-oh, a bug… I’m outta here.
Digression Testing: Well, it works, but can I tell you about my truck…
Expression Testing: a bug.
Obsession Testing: I’ll find this bug if it’s the last thing I do.
Oppression Testing: Test this now!
Poission Testing: Alors! Regardez le poission!
Repression Testing: It’s not a bug, it’s a feature.
Secession Testing: The bug is dead! Long lives the bug!
Suggestion Testing: Well, it works but wouldn’t it be better if…
Ref: netfunny.com

19. Testing Levels Unit testing Integration testing System testing
Acceptance testing

20. Unit testing The most ‘micro’ scale of testing.
Tests done on particular functions or code modules.
Requires knowledge of the internal program design and code.
Done by Programmers (not by testers).
Unit testing tool

21. Integration Testing Types of Integration Testing
Testing of combined parts of an application to determine their functional correctness.
‘Parts’ can be
code modules
individual applications
client/server applications on a network.
Types of Integration Testing
Top-down
Bottom-up
Sandwich
Big-bang

22. Top-down Integration Testing

26. Systems Testing
To test the co-existence of products and applications that are required to perform together in the production-like operational environment (hardware, software, network)
To ensure that the system functions together with all the components of its environment as a total system
To ensure that the system releases can be deployed in the current environment

27. Acceptance Testing Objectives
To verify that the system meets the user requirements
When
After System Testing
Input
Business Needs & Detailed Requirements
Master Test Plan
User Acceptance Test Plan
Output
User Acceptance Test report

28. Testing an application under heavy loads.
Load testing
Testing an application under heavy loads.
Eg. Testing of a web site under a range of loads to determine, when the system response time degraded or fails.

29. Stress Testing
Testing under unusually heavy loads, heavy repetition of certain actions or inputs, input of large numerical values, large complex queries to a database etc.
Term often used interchangeably with ‘load’ and ‘performance’ testing.
Performance testing
Testing how well an application complies to performance requirements.

30. Alpha testing Beta-testing
Testing done when development is nearing completion; minor design changes may still be made as a result of such testing.
Beta-testing
Testing when development and testing are essentially completed and final bugs and problems need to be found before release.

31. Good Test Plans (1/2) Developed and Reviewed early.
Clear, Complete and Specific
Specifies tangible deliverables that can be inspected.
Staff knows what to expect and when to expect it.

32. Good Test Plans (2/2) Realistic quality levels for goals
Includes time for planning
Can be monitored and updated
Includes user responsibilities
Based on past experience
Recognizes learning curves

33. Test Cases Test plan reference id Test case Test condition
Contents
Test plan reference id
Test case
Test condition
Expected behavior

34. Good Test Cases Find Defects
Have high probability of finding a new defect.
Unambiguous tangible result that can be inspected.
Repeatable and predictable.

35. Good Test Cases Traceable to requirements or design documents
Push systems to its limits
Execution and tracking can be automated
Do not mislead
Feasible

37. Bugs prioritization

38. Outline
Software Test in General
Value-based Software Test

39. Tester’s Attitude and Mindset
“The job of tests, and the people that develop and run tests, is to prevent defects, not to find them.”
- Mary Poppendieck

40. Pareto 80-20 distribution of test case value [Bullock, 2000]
Actual business value
100
100 80 80
% of
% of 60 60
Value
Value
Automated test
Automated test
for
for
generation tool
generation tool
Correct
Correct 40 40 - -
all tests have equal value*
Customer
Customer
Billing
Billing 20 20 5 5 10 10 15 15
Customer Type
Customer Type
*Usual SwE assumption for all requirements, objects, defects, …

41. Business Case for Value-Based Testing

42. How can we compare the value of test cases ?
How to prioritize test cases ?
How to measure the value of test cases?

43. Value-based Software Testing Framework- Feature Prioritization

44. How much test is enough?
Li, Q., Yang, Y., Li, M., Wang, Q., Boehm, B. W. and Hu, C., Improving software testing process: feature prioritization to make winners of success-critical stakeholders. Journal of Software Maintenance and Evolution: Research and Practice, n/a. doi:  /smr.512

45. Value-based Test Case Prioritization

46. Value-based Test Order Logic
Value First: Test the one with the highest value.
Dependency Second: If the test case with the highest value is not “Ready-to-Test”, which means at least one of the test cases in its Dependencies Set is “Not-Tested-Yet”. In such situation, prioritize the “Not-Tested-Yet” test cases according to “Value First” in this Dependencies Set and start to test until all test cases in the Dependencies Set are “Passed”. Then the test case with the highest value is “Ready-to-Test”.
Shrink the prioritization set ASAP: Exclude the tested one out of the prioritization set.

47. Value-based Test Order Logic

48. Test Case Dependency Tree

49. Accumulated Cost-Effectiveness (ACE) of Test

50. Testing in 577 EP-17 EP-18 EP-19 EP-20 Value-Based Testing Process
Value-Based Software Testing Guideline
EP-19
Test Plan and Cases Template
EP-20
VB Test Procedure and Result Template

51. Test Plan What, when, where, how, by whom? Traceability Matrix
Type of testing
Timeline
Developers’ machine / server
Tools, HW, SW
Responsible person
Traceability Matrix

52. Test Cases

54. Test Procedure

55. Found bugs, then what?

56. Developer: There is no I in TEAM
Tester: We cannot spell BUGS without U