1. Test Requirements: The Basis of Testing
David Capocci, CQA, CSTE
Test Requirements: The Basis of Testing
David Capocci, CQA, CSTE
Sr. QA Systems Analyst
SAFECO Corporation

2. Agenda Defining Test Requirements (TR)
David Capocci, CQA, CSTE
Agenda
Defining Test Requirements (TR)
What, Why, Where
Fitting TR’s into the testing picture
What’s within our testing process
Generating TR’s
Organizing & Decomposing TR’s
Test Requirements Hierarchy Samples
Setting the stage for measuring test coverage

3. David Capocci, CQA, CSTE
Background
“Test Requirements Hierarchy” is a term coined from Rational’s SQA Team Test software.
The principle of identifying, organizing, and measuring test requirements is universal to many test processes and methodologies
Much of this in-service is distilled from:
Rational methodologies (we are an SQA Team Test house after all)
QAI Workbench model
Seminar topics from
19th annual International Software Testing Conference
Star ‘98 West Conference
Ed Kit’s “Software Testing in the Real World”

4. Test Requirements What exactly is a Test Requirement?
David Capocci, CQA, CSTE
Test Requirements
What exactly is a Test Requirement?
Why identify Test Requirements?
Where does a Test Requirement come from?
Defining TR’s: What, Why, Where

5. What exactly is a Test Requirement?
David Capocci, CQA, CSTE
What exactly is a Test Requirement?
Identifies the “WHAT” of testing
What needs to be tested, AND
What are you going to validate about it
Includes both normal and error conditions
Covers business rules, functionality, non-functional standards
Do NOT have case specific data values assigned to them yet (data appears in test cases, the “How” of testing) examples…
Defining TR’s: What, Why, Where

6. Example 1: Testing the inserting of a record to a table
David Capocci, CQA, CSTE
Example 1: Testing the inserting of a record to a table
Test Requirements Identified (among others):
“Validate that you can insert an entry”
“Validate that insertion fails if entry already present”
“Validate that insertion fails if table already full”
“Validate that you can insert an entry to an empty table (initial)”
These are test requirements NOT tests because they do not describe the data element being inserted
The data is irrelevant at this level, it will appear in the test cases used to cover these test requirements
“Validate you can insert ‘John Doe’” is a test case not a test requirement
Defining TR’s: What, Why, Where

7. Why identify Test Requirements?
David Capocci, CQA, CSTE
Why identify Test Requirements?
It’s what QC “owns” in our workbench: Requirements-based or Function-based testing
It’s the basis for establishing the completion of testing
Helps determine the scale of the testing effort
Governs the types of resources you will need
Serves to identify automation strategies you can use
Becomes a roadmap for your testing effort
Can be a tool for leverage and dialog with developers and business analysts
Dev Team can sign off on them (Verification!)
Defining TR’s: What, Why, Where

8. Where does a TR come from?
David Capocci, CQA, CSTE
Where does a TR come from?
Traditional: Business Requirements, functionality, internal logic…
Marketing specs, Functional specs, Technical specs
Reality:
“Interview Analysis”, Non-Functional Checklists (standards & compliance), Use Cases (from business scenarios and users), Discovery during testing, any other deliverables from previous workbenches (diagrams, modeling, flowcharts, etc.)
Defining TR’s: What, Why, Where

9. How do Test Requirements relate to the Test Plan?
David Capocci, CQA, CSTE
How do Test Requirements relate to the Test Plan?
Traditionally, the Test Plan has represented “what” was going to be tested, even including test cases.
Paradigm is shifting: Test Plan should relate what your testing process (and deliverables) will be for a particular project.
A Test Plan should build confidence in the testing process for a project, making approaches clear.
A Test Plan can include the Test Requirements
However, if the Test Requirements are too lengthy, they should become their own document: A Test Requirements Hierarchy
Defining TR’s: What, Why, Where

10. Agenda Defining TR’s: What, Why, Where
David Capocci, CQA, CSTE
Agenda
Defining TR’s: What, Why, Where
Fitting TR’s into the testing picture
What’s within our testing process
Generating TR’s
Organizing & Decomposing TR’s
Test Requirements Hierarchy Samples
Setting the stage for measuring test coverage

11. Drilling down: Where test requirements fit into the picture
David Capocci, CQA, CSTE
Drilling down: Where test requirements fit into the picture
Business
Requirement 1
Test
Generates
Requirement M 1
Test
Scenarios/
Executes/Runs M
Generates M
Cases 1
Test
Procedure/
Script
Fitting TR’s into the testing picture

12. Drilling Down Business Requirement Test Requirement Test Scenarios/
David Capocci, CQA, CSTE
Drilling Down
First, Let’s look
at this relationship:
What’s within our
testing process
Business
Requirement
Test
Requirement
Test
Scenarios/
Then we’ll look
at this relationship:
Gernerating TR’s
from what feeds into
our testing process
Cases
Test
Procedure/
Script
Fitting TR’s into the testing picture

13. ATM Example: Practice Writing Test Requirements
David Capocci, CQA, CSTE
ATM Example: Practice Writing Test Requirements
Business Requirements:
- “ATM must do withdrawals”
- “Withdrawals are between $20-$300”
- “Withdrawals are in $20 multiples”
Group Exercise!
1. Limit the scope to these 3 requirements.
2. What will you validate (test for)?
3. Are there any implied requirements that may not be written out?
What’s within our testing process

14. Example 2: Testing Withdrawals on an ATM
David Capocci, CQA, CSTE
Example 2: Testing Withdrawals on an ATM
Test Requirements Identified (among others):
“Validate that a withdrawal option is available”
"Validate that a withdrawal of a multiple of $20, between $20-$300 can be done"
"Validate that <$20 is not allowed"
"Validate that >$300 is not allowed"
"Validate that $20 multiples >$300 is not allowed"
"Validate that non-$20 multiples between $20-$300 not allowed"
"Validate strange numeric amounts/combinations not allowed (all zero's, all 9's, )"
“Validate that the withdrawal received is equal to the amount requested”
"Validate that a valid withdrawal amount must be below the account balance”
These are test requirements NOT tests because they do not describe the data element being used (like $20, $40, $60, $1)
The data is irrelevant at this level, it will appear in the test cases used to cover these test requirements
What’s within our testing process

15. David Capocci, CQA, CSTE
Test Scenarios/Cases for - “Validate that a withdrawal of a multiple of $20, between $20-$300 can be done”
What’s within our testing process

16. Test Procedure & Script for previous example
David Capocci, CQA, CSTE
Test Procedure & Script for previous example
Procedure:
Script: (in pseudo-code )
Step 1: Insert Card
Step 2: Enter PIN
Step 3: Select Withdraw option
Step 4: Enter dollar amount
Step 5: Validate amount received
Do until EOF ‘until end of data file
Input data record
Senddata CARDINFO to “Cardfield”
Senddata “Enter”
Senddata PIN to “PINFfield”
Senddata “W” to “SelectionField”
Senddata AMOUNT to “DollarField”
If ErrorMsg > 0 then print ErrorMsg
Print “DollarAMTgiven”
Loop
Think
Manual !
Automated !
Think
What’s within our testing process

17. Agenda Defining TR’s: What, Why, Where
David Capocci, CQA, CSTE
Agenda
Defining TR’s: What, Why, Where
Fitting TR’s into the testing picture
What’s within our testing process
Generating TR’s
Organizing & Decomposing TR’s
Test Requirements Hierarchy Samples
Setting the stage for measuring test coverage

18. Our workbench is called “Generating Test Requirements”
David Capocci, CQA, CSTE
The Workbench Concept DO
Check
Standards
Tools
Rework
Entrance Criteria
Exit Criteria
Product Input
Product Output
Our workbench is called “Generating Test Requirements”
Generating TR’s

19. David Capocci, CQA, CSTE
Entrance Criteria for Business Requirements to generate Test Requirements
Visible ?
Clear? (unambiguous)
Complete?
Consistent? (conflicting requirements must be prioritized)
Reasonable? (achievable)
Measurable? (quantifiable)
Modifiable? (will it change or is it stable?)
Traceable? (the source is known)
Dependent requirements identified?
Testable? (given current environment, resources, skills)
Generating TR’s

20. Exit Criteria for Test Requirements
David Capocci, CQA, CSTE
Exit Criteria for Test Requirements
Can another tester create test cases/scenarios from these?
Does a Test Requirement specify what is being tested and what about it we are validating? (Clear?)
Are the Test Requirements…
Complete?
Consistent? (conflicting requirements must be prioritized)
Reasonable? (achievable)
Measurable? (quantifiable for measuring test coverage)
Modifiable? (will it change or is it stable?)
Traceable? (the source is known)
Testable? (given current environment, resources, skills)
Do the test requirements cover the complete scope of the project?
Are all the test requirements verified and signed off by the Development Team?
Generating TR’s

21. When creating Test Requirements (“Do”)...
David Capocci, CQA, CSTE
When creating Test Requirements (“Do”)...
Use “action” verbs & words
“Validate that…”
“Check for…”
“Test that…”
Trace them back to the source
Remember that different applications arrange in different ways
Think of MF, batch, C/S, web, e-commerce, GUI, etc.
Use “testing considerations” checklists that generally cover what kinds of things should be considered when testing your specific situation
Make your Test Requirements document a “living document”
Generating TR’s

22. Also... Maintain a level of balance between too much & too little
David Capocci, CQA, CSTE
Also...
Maintain a level of balance between too much & too little
Too High level: won’t be useful, vague, can’t generate test cases from it.
Too low level: Over-process, over documentation, no productivity
General rule: 5-7 levels deep in an outline format
Organize them!
Outline/Hierarchy format recommended
Look at how the BA breaks down the project into areas
Look at how the PA breaks down the project into areas
Organize by functional areas
Organize by “types” of testing (Function vs. System vs. Non-Functional)
Generating TR’s

23. Agenda Organizing & Decomposing TR’s Defining TR’s: What, Why, Where
David Capocci, CQA, CSTE
Agenda
Defining TR’s: What, Why, Where
Fitting TR’s into the testing picture
What’s within our testing process
Generating TR’s
Organizing & Decomposing TR’s
Test Requirements Hierarchy Samples
Setting the stage for measuring test coverage

24. Distinguishing the types of testing….
David Capocci, CQA, CSTE
Distinguishing the types of testing….
I. Function-Based Tests
II. User Interface Tests
III. Security Tests
IV. Installation Tests
V. Configuration Tests
VI. Performance Tests (Response)
VII. Load Tests (simultaneous users, lots of small transactions)
VIII. Volume Tests (Big transactions)
IX. Stress Tests (breaking point: memory, resources)
X. Resource Usage Tests
XI. Documentation Tests
XII. Compatibility Tests
XIII. Recovery Tests
XIV. Serviceability Tests
and others…
*III - XIV are all “Systems-based tests”
Organizing TR’s

25. Organizing by Functional areas….
David Capocci, CQA, CSTE
Organizing by Functional areas….
Most testers perform function-based, or requirements-based tests
Tests on functionality that the system will provide
Usually scenario/case based
includes normal and alternate (invalid) cases
Organizing TR’s

26. Organizing by Functional areas….
David Capocci, CQA, CSTE
Organizing by Functional areas….
Most testers also perform User Interface Style Tests
These are generally separate from the functionality that the software will provide
Usually encompass the architectural standards & compliance (like Windows Design Standards)
Also includes tests of navigation, menus, admin functions (like printing, saving)
Organizing TR’s

27. Agenda Defining TR’s: What, Why, Where
David Capocci, CQA, CSTE
Agenda
Defining TR’s: What, Why, Where
Fitting TR’s into the testing picture
What’s within our testing process
Generating TR’s
Organizing & Decomposing TR’s
Test Requirements Hierarchy Samples
Setting the stage for measuring test coverage

28. Remember this?…Drilling down
David Capocci, CQA, CSTE
Remember this?…Drilling down
Fitting TR’s into the testing picture

29. Decomposing: Drilling down within a Test Requirement
David Capocci, CQA, CSTE
Decomposing: Drilling down within a Test Requirement
Business
Function
Tasks within the
Data Entry Types
for transactions
Transactions to
perform a task
Field Validation
Business
Requirement
Test
Requirement
Keep the
function-based
perspective
in mind!
Test
Scenarios/
Cases
Test
Procedure/
Script
Fitting TR’s into the testing picture

30. Test Requirement Decomposition
David Capocci, CQA, CSTE
Test Requirement Decomposition
Decomposing TR’s

31. Test Requirement Decomposition
David Capocci, CQA, CSTE
Test Requirement Decomposition
High level Functional Areas:
usually from
“Functional Spec” type
docs, or BA work
Business
Function
Tasks within the
Function
Transactions to
perform a task
Lower level
Functional Areas:
usually from
“Technical Spec” type
docs regarding
internal logic,
or PA work
Data Entry Types
for transactions
Field Validation
Decomposing TR’s

32. Test Requirement Decomposition
David Capocci, CQA, CSTE
Test Requirement Decomposition
You can start on the high level functional areas early into the project & build in lower level areas as they become available
Any level can contain a test requirement which can also be made up of (or broken down into) lower level test requirements
Decomposing TR’s

33. Business Function Level
David Capocci, CQA, CSTE
Business Function Level
Try to identify “groups” of functions or functions connected by similar themes
file management functions, printing functions, help functions, car rental functions, reservation functions, ticket purchase functions, claim reporting functions
Be sure all areas of the system are covered. If something is left out or doesn’t fit into a group, it becomes its own group.
It may be easier to identify functional areas by “window” instead of by function.
Function
Trans Data
Type
Field
Validation
Task
Transaction
Decomposing TR’s

34. Business Function Level
David Capocci, CQA, CSTE
Business Function Level
At this level, the idea is seeing the connections, integration, and interactions of the system’s functionality.
May not necessarily be identifying a test requirement at this level as much as just identifying the functional area.
Function
Trans Data
Type
Field
Validation
Task
Transaction
Decomposing TR’s

35. David Capocci, CQA, CSTE
Task Level
Break down each Function area into the tasks within the function
For complex tasks, it is possible to break them down further into sub-tasks
Some Business Functions can not decompose into further tasks (example: Check Writing function)
Trans Data
Type
Field
Validation
Function
Task
Transaction
Decomposing TR’s

36. David Capocci, CQA, CSTE
Transaction Level
From this level down, we start to address the internal things that occur to make the functions and tasks happen
Identify any logical transactions that ties the task to the database or any other transactions necessary to perform the task.
Identify any data processing, calculation, data formatting type transactions
Note: A screen or window may cause the execution of several different logical transactions
Transaction
Trans Data
Type
Field
Validation
Function
Task
Decomposing TR’s

37. Transaction Data Type Level
David Capocci, CQA, CSTE
Transaction Data Type Level
Identify which of the four types the transaction can become: Add, Change, Delete, Inquire
It is entirely possible that a transaction can be multiple types.
If a transaction is only one type, you can wrap this level up into the higher level.
Trans Data
Type
Field
Validation
Function
Task
Transaction
Decomposing TR’s

38. Field Validation Level
David Capocci, CQA, CSTE
Field Validation Level
Most testers like to jump directly to this level. It’s the most obvious at times.
Field validation covers all edits & cross edits on fields and data.
Be careful of the detail you document at this level. Remember the separation between test requirement and test case.
Field
Validation
Trans Data
Type
Function
Task
Transaction
Decomposing TR’s

39. Field Validation Level
David Capocci, CQA, CSTE
Field Validation Level
Not all test requirements (especially at this level) fit well to be documented in this manner.
Example: Testing all the stated properties of windows objects
Use “Validate that the properties of all the objects in this window match the properties listed on the Object Properties Reference Table in Appendix B upon initialization of the window”
Don’t list each property check as a separate test requirement if it can be summarize under one test requirement
This is a judgement call YOU make for your given project.
Field
Validation
Trans Data
Type
Function
Task
Transaction
Decomposing TR’s

40. Example 3: Rental Car Application
David Capocci, CQA, CSTE
Example 3: Rental Car Application
1. Validate that a Rental can occur.
1.1 Check Customer policy coverage
1.2 Query Car availability
1.3 Query Car rates
1.4 Open a Rental ticket
1.4.1 Validate that a customer record can be entered
1.4.2 Validate that credit card approval is obtained
1.4.3 Validate that status on the car record is changed
from “waiting” to “rented”
2. Billing Function
3. Reservation Function
Let’s look at
the lower levels
for this one
Then we’ll try it
on this one
Decomposing TR’s

41. Example 3: Rental Car Application
David Capocci, CQA, CSTE
Example 3: Rental Car Application
1. Validate that a Rental can occur.
1.4 Open a Rental ticket
1.4.1 Validate that a customer record can be entered
Validate that a new customer can be added to the customer table
Validate that the first name is all alpha
Validate that the age is > 21.
Validate that the phone number is numeric
Validate area code is an existing area code number.
Validate changing an existing customer
Decomposing TR’s

42. Example 3: Rental Car Application
David Capocci, CQA, CSTE
Example 3: Rental Car Application
1. Validate that a Rental can occur.
1.4 Open a Rental ticket
1.4.2 Validate that credit card approval is obtained
…fill in the lower level test requirements!
First, Identify any sub-areas (further tasks, or even
separate transactions within this)
Then, Identify the lowest level field validation test
requirements (think about what is typically
involved with credit card authorizations)
Decomposing TR’s

43. What did you come up with?
David Capocci, CQA, CSTE
What did you come up with?
1. Validate that a Rental can occur.
1.4 Open a Rental ticket
1.4.2 Validate that credit card approval is obtained
_________________________________________
Decomposing TR’s

44. Possible Test Requirements...
David Capocci, CQA, CSTE
Possible Test Requirements...
Function
1. Validate that a Rental can occur.
1.4 Open a Rental ticket
1.4.2 Validate that credit card approval is obtained
Validate the expiration date is a valid future date
Validate the expiration date is not within 1 month of
expiring.
Validate that the CC# is 12 digits
Validate that the $ amount is <= credit balance available
Validate that an authorization # is received.
Task
Transaction
Decomposing TR’s

45. Agenda Defining TR’s: What, Why, Where
David Capocci, CQA, CSTE
Agenda
Defining TR’s: What, Why, Where
Fitting TR’s into the testing picture
What’s within our testing process
Generating TR’s
Organizing & Decomposing TR’s
Test Requirements Hierarchy Samples
Setting the stage for measuring test coverage

46. David Capocci, CQA, CSTE
TRH Samples
Let’s look at a few examples of how test requirements can be organized:
Sample 1: Excerpt from a Personal Finance app
Sample 2: actually organizing Function requirements, but good representative of the concept
Sample 3: QBS, from Rational’s sample project (adds types of testing into the hierarchy)
Sample 4: another view of the Personal Finance app
Sample 5: A mainframe batch system

47. Test Coverage Measures
David Capocci, CQA, CSTE
Test Coverage Measures
Test Requirements are the “what” of testing & are the basis for establishing the completion of testing
TR’s give us the point of measurement for test coverage
Each TR should receive a Priority, Risk, and Weight
Each TR should be tracked for Verification () and Validation (%)
Test Coverage Measures

48. Summary & Recap Defined: What, Why, Where
David Capocci, CQA, CSTE
Summary & Recap
Defined: What, Why, Where
Described: How TR’s “fit” into the big picture
Decomposed: Organizing TRs & Generate TR’s into a measurable format
Demonstrated: Sampling some TRH
Determined: The Measurement Connection