Csci565: System testing Chapter 14 H. Reza.

Csci565: System testing Chapter 14 H. Reza

objectives System testing Thread? Thread-based system testing
All about threads at system level How long a thread should be? How to identify threads? The role of threads in Structural/functional testing Example SATM Test Threads

System level testing System testing
Can be considered a final step in integration testing A check of consistency between the software system and customer expectations Encompasses both system wide properties and capabilities against a system specification and/or customer expecations System, acceptance, and regression testing are all concerned with the behavior of a entire software system

System, acceptance, and regression testing
System test Acceptance test Regression test Check against requirements specifications Checks suitability for user Rechecks test cases passed by previous production versions Performed by development test group Performed by test group with user involvement Performed by development group Verifies correctness and completion of the product Validates usefulness and satisfaction with the product Guards against unintended changes

System test design It is considered a good practice to design system test cases as early as possible Why? Exposes critical behaviors Avoids unpleasant surprises (e.g., fault of interactions/integrations and coordination's) Testing global properties may require extensive simulation of the execution environment

Thread Based Testing: some interpretations of Threads
A form of scenario (or usage) based testing Has different meaning at different levels E.g., sequences of events in Sequence, sequences of ASFs Threads have different meaning at different levels of abstractions System level behavior corresponds to a sequence of Atomic System Functions (ASFs) Observable actions at input/output ports Integration level behavior corresponds to a sequence of MM-path Unit level behavior corresponds to a sequence of statements

Threads: some generic examples
Example of threads may include A scenario of normal usage A stimulus/response pair A sequence of source instructions A sequence of atomic system functions (ASFs) A sequences of a state machine description of the system A sequences of transitions in FSM A sequences of transitions in Petri Nets …

Threads at different levels: some specific examples (SATM)
Candidate threads for SATM Entry of a digit (minimal atomic system function) an ASF works with only one stimuli/response PIN Entry (upper limit to integration testing) ASFs work with a set of stimuli/response max integration, min system level A simple transaction (very abstract) A system level thread because of end-to-end completion <Card entry, menu selection, perform operation, report the results> An ATM session having two or more transactions (very abstract) A sequence of threads OK for system level Good for interactions among threads

Thread Definitions An Atomic System functions (ASF)
An action that is observable at the system level in terms of port input and output events (GUI) Represents the connection point between integration and system testing Largest items to be tested by Integration testing Smallest item for system testing In event-based system, they behave as a sequences of ASF followed by event quiescences An ASF begins with a port input event, traverses parts of one or more MM-Paths, and terminates with a port output event E.g., Digit entry (KeySenor (Inport) , GetPIN, screen 2 or 4 displayed (outport)) Card entry(Inport,…, outport) Cash dispensing(inport,…, outport)

Definition: ASF Graph ASF graph
The directed graph in which nodes are ASFs, and Edges are sequential flow The graph start with Source ASF and terminates with Sink ASF System Thread Refers to a path from a source ASF to a Sink ASF in the ASF graph Thread graph The directed graph in which nodes are systems threads and edges represent sequential execution of individual threads

System testing and requirement specification
System testing w.r.t. requirement specifications Any system can be specified using Data (ERD/UML) Actions ( I/O view, transfer of control/data, etc.) Ports devices (source/destination of input/output level system) Events (system level input occur at inport/outport) Threads (scenarios (UML) or paths in control models)

Data-centric models When the focus is on the persistent data, data-centric models must be used (e.g., ERD) developed using CRUD (create, retrieve, Update, Delete) In SATM, Data PAN (different accounts) PIN (personal identification number)

Ports Ports (devices) The source and destination of system level inputs and outputs the points where I/O devices are connected to the system Examples in SATM Keypad The digits Functions keys (buttons), and cancel Key Display screen Slots

Events refer to system level input or output
the translation point between physical (external) and logical (internal) Event can be Discrete (e.g., keystroke) Continuous (e.g., pressure, temperature)

actions Process, or transformation point work with DFD
Inputs/outputs for both control and/or data

Relationships among basis concepts

Modeling relationship among basis constructs (fig. 14.2)

Examples of Models Models can be Structural models Contextual models
Used for development Express the functional decomposition Contextual models Emphasize system ports (or external environment) Behavioral models Emphasize the control aspects of the system Identification of models? Right models for right problems

Model selection: Event partitioning view of function F

Model selection: FSM of Function F

Finding Threads (scenarios) from behavioral models
FSM models of the SATM is the appropriate staring point for the system testing threads The model can be interpreted as follows States processing steps or the abstraction of the history Transitions internal/external events

Top level SATM in FSM

Low level SATM in FSM (PIN Entry state)

What happens when no data/event entered or missed? Should it change the state? The system can behave by either posting an error msg? use a default value/state, prompt user for missing data, or void/cancel the transaction. What happens if there is any failure due to power or NTW? What happens if invalid data or event entered? Should the state changes or should be ignored? What happens if our user does not do anything?

Table 1: Events in the PIN entry FSM

Very low level SATM ( PIN TRY) in SFM

Table 2: Port Events in the PIN try FSM

Table 3: An example of a path (or thread) with a correct PIN (1234)
Table 3 shows a possible path leading to correct PIN Logical event

Table 4: An example of a thread (1235, 123C, 1234)
Logical events

Structural Strategies (or path testing) for thread testing
Generating thread test cases is easy Too many threads (explosion problem) 156 distinct paths from the FIRST PIN Try state to the Await Choice or Card Entry (see fig.14.6) Strategies Bottom-up threads 6 paths can be tested in the PIN Try FSM to test (see table 5)

Table 5: 6 thread in the PIN Try FSM
correct incorrect cancellation

Bottom-up (next upper level)
Once the lower level is tested, move to the upper level (the PIN Entry FSM) Four paths tested to achieve these objectives Correct recognition of PIN Incorrect PIN Response to the cancel key Matching expected and actual PIN entered `See table 6’

Thread Paths in the PIN Entry FSM

Coverage Metrics for FSM
FSM are directed graphs use a graph-based coverage metrics Hierarchical relationship means Super node (upper level node) Sub-node (lower level) Super nodes treat sub-nodes as a procedure with enter/exist semantics Practicality of FSM (Good) Limitation is # of states (bad) Driving FSM model of the software(bad)

States and transitions Coverage matrices
Two Coverage (metrics) can be applied State coverage (node coverage) States correspond to nodes Transition coverage (edge coverage) Transitions correspond to edges Path Coverage? ‘See table 14.7’

Table 7: State/Transition coverage of the 3-try thread

State-Based Coverage

Table 9: Transition-Based Coverage

Problems with path testing (FSM)
The main assumption revolves around path traversing to uncover faults Can be counter-productive E.g., CCC in table 6 causes a three-try to fail Some path cannot be executed by themselves because of dependency between lower levels and upper levels E.g., 1235 in table 5 causes a transition to Second PIN try ( i.e., state 2.2) What happens when no key pressed when the machine is in the state 2.2?

Functional (black-box) Strategies for thread testing
FSM represents behavioral models What if no behavioral models exist? Use functional (or computational) models for functional testing Use information from the input/output spaces and the function

Functional models and functional testing
Test cases for functional testing Use input/output information to perform: Testing legal and illegal input Use inputs that force all error messages to occur Use inputs that force the software to establish default values Testing legal and illegal output Force different outputs to be created for the same input Force illegal outputs to be generated

Coverage metrics for functional models
Coverage metrics for functional models can be based on Events Ports (device) Data Threads?

input port Coverage: There are five port input thread coverage metrics
PI1: each port input event occurs Minimum requirement PI2: each common sequences of port input events (or sequence of transitions) occurs Normal usage; difficult to quantify PI3: each port input event occurs in every “relevant data context” Physical events with different logical meaning e.g. B1 with different meaning because of the context”) PI4: for a given context, all “inappropriate” input events occurs Negative scenarios/unexpected input events (e.g., B1pressed for digit in the PIN entry) PI5: for a given context, all possible input events occurs Testers have difficulty to anticipate the expected responds to all stimulus This problem can be resolve by using complete, consistent, and correct SPEC Need to know things that should happened and things that should not happened E.g., use UML Activity graph (diagram) to model both expected and unexpected things that may happened

Output port Coverage There are two port output thread coverage metrics
PO1: each port output event occurs (output messages) Set of output msgs for each error conditions PO2: each port output event occurs for each cause one respond may occur to many stimulus or causes E.g., Out of order screen may display for mechanical failures or lack of sufficient funds or door is not working

Port-based (device) Testing (GUI)
Works for systems having port devices coming from external supplies Works with event-based systems Need to identify for every port and its corresponding events Generate scenarios that exercise input/output ports w.r.t. to the event lists

Data-centric thread based testing
Works for data driven system Not appropriate for event-driven or reactive systems

Thread Identification using data-centric models (ERD and Schema testing)
scenarios corresponding to the syntactical elements of DBASE can be identified from the ERD or schema diagrams Use Relationships Cardinality (e.g., one-to-one) E.g., bank customer may have several accounts, each with different PIN Participations (e.g., total vs. partial) Functional dependencies (X->Y) Test for dependency preservation and non-additive join property Test Constraints Referential Integrity, Foreign Key, etc Constraints Violations E.g., Insert/delete/update, etc. Test all triggers, stored procedures/actions executed when the table with which it is associated has an insert/delete, or update …

Coverage matrices for data-centric system
Uses ERD (or similar models as starting points) The coverage includes DM1: exercise the cardinality of every relationship in ERD DM2: exercise the participation of every relationship in ERD DM3:Exercise the functional dependencies among relationships in ERD …

Example: Transactions for Library system
Some typical transactions in the library system Add a book to the library Delete a book from the library Add a borrower to the library Delete a borrower from the library Loan a book to a borrower Process the return of a book from a borrower

Example of Data-based thread testing (Library system)

SATM TEST threads How to use path testing to generate threads at system level? Use an overall state model (FSM) as starting point Treat super states as a key atomic functions (i.e., ASF) Follow the order among super states according to the model Specify the precondition (if it is needed) Specify the postcondition (if it is needed)

An example: SATM Test Threads (table 10)
Example of preconditions for SATM can be 1.The total cash available to the machine is $500 2.The terminal displays screen 1 3. Database is populated with initial information (see table 10)

Three basic threads for balance inquiry, deposit, and withdrawal
stimula respond Note: B1: Balance B2: Deposit to checking B3: Withdraw

Thread 4: invalid card

Threads (test cases) for sub-states corresponding to Balance (Thread 5-8)

Macro-states for thread 1: withdrawal (Thread 9-11)
B1: Balance B2: Deposit to checking B3: Withdraw

What happens if our user does not do anything?

Tentative Presentation Schedule
Group Dec.4 Dec.9 Dec.11 Adhikari (1) 9:55 Arora (2) 10:20 Aguilar (3) 9:30 Bajayee (4) Bhatia (5) Palmer (6) Pal (7) Course Wrap up and Final remarks Note: Students participation are required during the presentations.

Unexpected: need to model unexpected behaviors that make sense
An unacceptable behavior which should be documented and avoided in the final implementation What happens if a stolen card is used? What happens when no data/event entered or missed? Should it change the state? The system can behave by either posting an error msg? use a default value/state, prompt user for missing data, or void/cancel the transaction. What happens if there is a failure due to power or NTW? What happens if invalid data or event entered? Should the state changes or should it be ignored?

Withdraw Deposit ATM USER Balance Inquiry ATM actor and use case

USE Case Name: Withdraw Funds
Summary: Customer uses a valid card to withdraw funds from a valid bank account Actor: ATM Customer Precondition: ATM is displaying Screen 1 (welcome) customer inserts an ATM card into card reader If the card recognized, the system reads the card number System prompts customer for a PIN (screen 2) Customer enters PIN System checks the card for validate If card is valid, the system checks the PIN entered against the existing If the PIN match, the system finds out what accounts the card can accessed System display customer accounts and prompts to choice transaction Customer select withdraw, select account and enter amount SATM checks that the account is valid, fund is enough, and daily is OK and available SATM dispenses the money SATM prints a receipt with a transaction number, the type of transaction SATAM Eject the card System display Screen 1

Unexpected usages (negative /failure scenarios)
What if the system cannot recognize the card. System ejects the card What if the current date is past the card’s expiration date Card should be confiscated What if the card reported as lost/stolen What if PIN does not matches Allow three attempts and then confiscate the card What if the customer enter C Cancel the transaction What if the customer leave in the middle of transaction Use system time out to cancel transaction and keep the card What if the request for withdraw exceeds Display non-sufficient and msg and eject the card …

What happens when no data/event entered or missed
What happens when no data/event entered or missed? Should it change the state? The system can behave by either posting an error msg? use a default value/state, prompt user for missing data, or void/cancel the transaction. What happens if there is any failure due to power or NTW? What happens if invalid data or event entered? Should the state changes or should be ignored?

Super transition What happens when no data/event entered or missed? Should it change the state? The system can behave by either posting an error msg? use a default value/state, prompt user for missing data, or void/cancel the transaction. What happens if there is any failure due to power or NTW? What happens if invalid data or event entered? Should the state changes or should be ignored?

Use Case Scenarios A use case scenario is an instance of a complete path through a use-case Scenario should be meaningful from user point of view If the PN modeling of use-case is COMPLETE and FINITE, then it is possible to list all test cases corresponding to the scenarios Need to reduce the size of the set T of test cases domain knowledge may be needed.

Strategies to deal with the thread or path explosions
435 distinct threads per valid account in SATM Huge number! Possible alternatives Operational profiles Zipf’s law (80% vs. 20%): 80% of the execution traverse only 20% of the threads e.g., 20% of students send 80% of s. Assuming that fault distribution indirectly related to system reliability decision tree can be used to determine the operational profile (execution frequencies of various paths corresponding to scenarios)

Transition Probability for the SATM system
1. Card entry P(card-not-ok)=.05 P(card-ok)=.95 2.1 first pin try P(pin-not-ok)=.10 P(pin-not-ok)=.10 P(pin-ok)=.90 2.2 second try entry P(pin-not-ok)=.10 P(pin--ok)=.90 2.3 3rd try 3. Select transaction P(pin-ok)=.90 P(B1)=.05 P(B3)=.85 P(B2)=.10 4. balance 5. deposit 6. withdrawal P(Not-OK)=.10 P(exitB2)=1 P(exitB1)=1 P(OK)=.85 7. Print receipt P(lowBalance)=.05

Thread probabilities Common thread Probabilities Un-common Thread
Card-OK .95 PIN-OK .90 PIN-not-ok-1st-try .10 Withdraw-OK .85 PIN-not-ok-2nd-try Normal PIN-ok-3rd -try Withdraw Low Cash

Other approaches: Requirement Prioritizations
Prioritize the requirements Decide on the importance of the requirements What would be the consequences of some of the features if they are not tested? Use risk driven approach to prioritize the important requirements Risk? Probability of occurrence an event and the severity of its impacts Use requirement traceability matrix to establish the relationship between requirements and test cases

Requirement traceability matrix
Prioritizations of main features and test cases utilizes traceability matrix The matrix relate test cases to the requirements Use risk driven approach to select/prioritize use cases Critical test cases also can be related to architecturally significant requirements The problem then would be how to identify the architecturally or design significant requirements?

Traceability Matrix (MAPS REQS TO TEST CASES)
 REQ2 REQ3 REG3 REG4 REG5 The matrix documents : the connection between features and test cases; Find requirements which have no corresponding test cases Which features still need to be tested Which test cases are affected by changes in the requirements

Risk and Test cases PROBLEM ID POSSIBLE PROBLEM
PROBABILTY OF OCCURANCE SEVERITY OF IMPACT RISK EXPOSURE TEST CASE ID1 LOSS OF POWER .1 10 1 TC-17 ID2 CORRUPTED FILE HANDELR .2 TC-220 ID3 UNAUTHROIZED ACCESS .6 8 4.8 TC-99; TC-87 ID4 NTW FAILURE .3 2.4 TC-17; TC-896 ID5 SLUGGISH THROUGHPUT 6 3.6 TC-123 ID6 OUT OF SYNC DB .9 9 TC-654; TC-324

Progression vs. regression
Regression testing (RT) Recheck the integrity of the modified (corrective, perfective, adaptive, and preventive) application s has lower operational frequencies uses longer threads that can fail in several ways with Requires s automation (Manuel regression testing means no regression testing ) Tools: Capture/Replay used for programs that uses GUI Selective RT techniques limits the amount of time needed to execute RT Inclusive: includes tests that are “modification-revealing” Efficient: identify the optimal subset of RT that is less computationally demanding General: applies to a wide variety of practical situations

Progression testing Progression testing Shorter path longer path
Check the integrity of the modified application with respect to the new features has higher operational frequencies uses shorter threads that can fail in a few ways Longer threads can be built from shorter threads Shorter path new features (i.e, higher fault rate ) longer path old features (i.e., lower fault rate)

System testing categories
Some of the major categories of system testing include Compatibility testing (how well the software performs in particular environment Configuration testing ( how software support different SW/HW configuration) Load testing (test application under a range of loads in some order to find out at what point the system’s response time degrades or fails) Stress testing (force the system to operate under an unreasonable load while denying it the resources needed to process that load) Security testing Safety testing, performance testing, etc.

Summary 1: Specification based testing
Using model/specification to generate test cases has several advantages: Tests can be created earlier in the software life cycle and can be executed before the program is finished Testing activities shifting to the front-end engineering allows for more effective planning and hence utilization of resources The process of generating test from the SPEC will often help the testers to find inconsistencies/ambiguities/incompelete-ness in the specification The essential part of the test data can be platform independent …

Summary 2: The combination of graph-based and functional based metrics can be used at system level 10 functional based 7 event-based 3 data-based 2 graph-based State-based Transition-based In graph-based system testing Need additional coverage, perhaps End-to-end (or path) coverage that corresponds to Initial and final configurations? Strategy to debug the FSM? Strategies to select important paths? Strategies to generate test cases using FMS? Strategies to anticipate unexpected behaviors by users? How to model these bahaviors ? Need another model? Or existing model?

Note about Testing The studies indicate that testing is responsible for more that 50% of the total cost of software development Testing support quality assurance but can be used to show the presence of certain qualities Testing requires the execution of the software (aka dynamic verification)

Advantages of testing The relative ease with which many of the testing activities can be performed Test-case requirements can be generated from various forms of the software and precode artifacts such as implementations, models, etc. The software being developed can be executed in its expected environment Much of the process can be automated and test cases can be REUSED Testing can be used to increase confidence

More on disadvantages of testing
Testing cannot show the absence of faults Testing cannot show the presence of qualities Test execution results for specific test cases cannot usually be generalized

Fundamental Testing Research Areas
How to use models (e.g., Statechart , Petri nets LTS, etc.) to test: GUI/Web-based systems Model? Coverage? The effectiveness of converges (or cost)? experimentation Related to GUI testing is usability testing which must be performed if GUI is an integral part of the system (e.g., Displays in safety critical systems), Regression Testing of GUI, Web, Database system Reusability of test cases Integration of RT with GUI/Web testing/usability testing?

Research 2: COTS Testing distributed systems built from component-based systems Need more efficient and more effective ways to test these systems Many issues (COTS) component provider vs. component users The lack of availability of source code for COTs is a major issue Testing COTs for safety, integration, security, etc. Need methods and techniques for representing and computing the types of testing information that a component user needs E.g., coupling and coverage information for use in integration testing can be embed with the component or techniques developed by COTs provider for generating that information to COTs users

Research: testing Based on Precode Artifacts
Testing techniques can be based on precode artifacts, such as Architectural specification based-on formal notations These techniques can provide test-case requirements for assessing various aspects of the architecture Allows various aspects of a system be assessed at early stage of software development Design Requirements

Other research areas: 1 Testing Evolving Software
Regression testing is used to test software that is being developed under constant evolution as the market or technology changes Regression testing is very expensive (cost 1/3 of total cost) Need efficient regression testing methods Research spans a wide variety of topics E.g., given an existing test suite and information about a previous testing, select a subset of the test suite for use in testing the modified software E.g., how to manage the size of a test suite How to use precode artifacts in selective retest of the software

Other research areas: 2 Demonstrating effectiveness of Testing Techniques How to evaluate criteria for showing adequacy of test suits Selections of test cases Based on software intended and acceptable behaviors Based on software unacceptable behaviors Techniques to identify classes of fault for which specific criteria are effective Analytical, statistical or empirical evidence of the effectiveness of criteria Need to determine the interaction among the various test selection criteria

Other research areas: 3 Establishing effective processes for testing
An important aspect of testing is the process that we use for planning/implementing it Tools and methods that can be used by practitioners to test their software Ways to automatically create the tools (e.g., compilers) Automated test data generations (i.e., inputs for test cases) Generating test data is often labor-intensive process: how can we automate this process?

More Empirical studies to show the scalability and usefulness
Need to design controlled experiments to show the techniques we develop Need to collect sets of experimental subjects, and, if possible, make them available to the researchers Need to collaborate with industrial people/partner to perform benchmark

Acknowledge/References
Some Slides borrowed from Mode-Based testing by Harry Robinson Introduction to software testing by Paul Amman and Jeff Offutt Mary Jean Harrold. Testing: A Roadmap. 22nd International Conference on Software Eng. (ICSE00), June 2000. David Lee, and Mihalis Kakis. Principles and Methods of testing Finite State Machines -Survey IEEE, vol.84, No.8., August 1996

Csci565: System testing Chapter 14 H. Reza.

Similar presentations

Presentation on theme: "Csci565: System testing Chapter 14 H. Reza."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Csci565: System testing Chapter 14 H. Reza.

Similar presentations

Presentation on theme: "Csci565: System testing Chapter 14 H. Reza."— Presentation transcript:

Similar presentations

About project

Feedback