October 29, 2004SAVCBS04 Presented by: Gaoyan Xie CTL Model-checking for Systems with Unspecified Components Gaoyan Xie and Zhe Dang School of Electrical Engineering and Computer Science, Washington State University, Pullman, WA 99164, USA

SAVCBS04 Motivation Challenges to the quality assurance of component-based systems: Externally obtained components may be new sources of system failures Safely upgrading a deployed component is extremely difficult Example: The Ariane 5 disaster was caused by reusing certain software module from Ariane 4 without sufficiently testing it in the new environment

SAVCBS04 Two Dimensions of The Possible Solutions Methodologies Compositional approaches conventional, used most often Bottom-up, from components to systems Decompositional approaches not many can be automated Top-down, from systems to components Techniques Software Testing A trial and error technique Integration/system testing is often needed at the system level Model-checking: A formal and usually complete technique The assume-guarantee paradigm is often used for system-level verification

SAVCBS04 Testing a Component in Isolation Before it is released By component developers By third-party labs component certification Difficulty: A component may target a wide scope of deployment environments, which may not be tried out When it is integrated into a system By system developers Expensive: A component is usually built with multiple sets of functionalities Infeasible: The state space of a components interface may be too large to be thoroughly tested

SAVCBS04 Testing A Component-based System (CBS) Integration/system testing Difficulties : Components within a CBS may run concurrently Even inside a component may have multi-threads running concurrently, and Concurrency is notoriously difficult to test Locating the source of a bug is not easy! Feasibility : A component may be used for dynamic upgrading a running system, which may not be stopped for testing

SAVCBS04 Model-checking Checks whether a finite-state system is a model of a given temporal formula Requires a complete and formal specification of the finite-state system Not always possible: design details or source codes of components in a CBS are generally unavailable The assume-guarantee paradigm is often used for system-level verification A compositional approach Requires an assumption for the environment of a module

SAVCBS04 The Research Problem A different perspective How to ensure that a component functions correctly in a specific deployment environment (which is called a host system ) A model-checking problem Given a host system M with an unspecified component X and a CTL formula f, check whether f is a model of (M, X): (M, X) = f

SAVCBS04 Basic Ideas Goal Seek a definite answer (no approximation) to the model-checking problem, i.e., (M, X) = f Solution Automatically deduce a sufficient and necessary condition over the unspecified component Check the condition through testing the unspecified component

SAVCBS04 The System Model A system Sys= (M, X) consists of a host system M and an unspecified component X The host system M is a well-specified finite state transition system The unspecified component X can be viewed as a deterministic Mealy machine Internal detail of X is unknown Interface ( input/output symbols) and an upper bound for the number of states in X are known An implementation of X is available for testing M and X run concurrently and communicate with each other by synchronizing over Xs input/output symbols

SAVCBS04 An Example s0s0 s1s1 s2s2 s4s4 msg? send/yes ack/yes send/no ack/yes msg? Comm data ack yes no Host system M Check: starting from state s 0, is state s 3 reachable? (M, Comm), s 0 = E[true U s 3 ] s3s3 s3s3

SAVCBS04 Check: starting from state s 0, is state s 3 reachable? M, s 0 = E[true U s 3 ] s0s0 s1s1 s2s2 s4s4 s3s3 Transition system M An Example contd.

SAVCBS04 Check: starting from state s 0, is state s 3 reachable? M, s 0 = E[true U s 3 ] Search the graph to see whether there is a path between s 0 and s 3 s 0 s 1 s 4 s 3 is such a witness path s0s0 s1s1 s2s2 s4s4 s3s3 Transition system M An Example contd.

SAVCBS04 s0s0 s1s1 s2s2 s4s4 s3s3 msg? send/yes ack/yes send/no ack/yes msg? Comm send ack yes no Test Comm to see whether the communication trace along the path can be accepted by Comm: Will send/no, ack/yes be a successful test case for Comm? Host system M Check: starting from state s 0, is state s 3 reachable? (M, Comm), s 0 = E[true U s 3 ] Search the graph to see whether there is a path between s 0 and s 3 Is s 0 s 1 s 4 s 3 still a witness path? An Example contd.

SAVCBS04 s0s0 s1s1 s2s2 s4s4 s3s3 msg? send/yes ack/yes send/no ack/yes msg? Infinite number of paths could be witness path: s 0 s 1 s 2 s 1 s 4 s 3, s 0 s 1 s 2 s 3, s 0 s 1 s 2 s 1 s 2 …s 3,… Infinite number of communication traces need to be tested: send/yes send/no ack/yes, send/yes ack/yes, send/yes send/yes…ack/yes, … Comm send ack yes no Host system M An Example contd.

SAVCBS04 s0s0 s1s1 s2s2 s4s4 s3s3 msg? send/yes ack/yes send/no ack/yes msg? Use an annotated directed graph G h (called witness graph) to represent all those communication traces The original model-checking problem is: True, if one of the communication traces in G h is a successful test case for Comm False, if none of the communication traces in G h is a successful test case for Comm s1s1 s2s2 s4s4 s3s3 send/yes ack/yes send/no ack/yes Host system M Witness graph G h An Example contd.

SAVCBS04 An Example contd. How to handle communication traces with infinite length? A result from traditional black-box testing: An equivalent internal structure of a black-box can be recovered through testing if only an upper bound for its number of states is known This result implies that Only communication traces with bounded length need to be tested The bound can be calculated from the number of states in M and the upper bound for the number of states in Comm

SAVCBS04 Our Algorithm The original CTL model-checking algorithm for checking M = f: Goes through a series of stages. A subformula h of f is processed at each stage: h is added to the labels of a state s if h is true at s Returns true if s 0 is labeled with f, or false otherwise Our algorithm adapts the original algorithm to handle systems with unspecified components. It follows a similar structure, except that during each stage for subformula h, the labeling of a state s is far more complicated

SAVCBS04 Our Algorithmcontd. A state s could be labeled with 1 if h is true at s and the truth has nothing to do with communications A witness graph if h is an CTL operator and the truth of h at s depends on communication traces in the graph A logical combination of witness graphs if h is a logic combination of other formulas and the truth of h at s depends on communication traces in the graph For each CTL operator in h, a witness graph is constructed and associated with an ID (which begins from 2) A state is labeled with an ID expression (constructed from IDs and logic connectives, ) A labeling function L h is returned upon the completion of processing h

SAVCBS04 Our Algorithmcontd. The algorithm outline: ProcessCTL is the lableling process, it recursively handles the five cases of f:,, EX, EU, and EG by calling the procedures Negation, Union, ProcessEX, ProcessEU, and ProcessEG respectively Procedure CheckCTL(M, X, s 0, f) L f := ProcessCTL(M, f); If s 0 is labeled by L f Then If L f (s 0 ) = 1 Then Return true; Else Return TestWG(X, reset, s 0, L f (s 0 )); Endif Else Return false; Endif End Procedure The maximal length of communication traces to be tested is bounded by (k·n·m 2 ), where k is the number of CTL operators in the formula f, m is the upper bound for the number of states in the unspecified component X, and n is the number of states in the host system M

SAVCBS04 Another Example Check: starting from the initial state s 0, whenever the system reaches state s 2, it would eventually reach s 3 ; i.e., check (M, X), s 0 = AG(s 2 AF s 3 ) to be true s0s0 s1s1 s2s2 s4s4 s3s3 msg? send/yes ack/yes send/no ack/yes msg? Comm send ack yes no Host system M Taking a negation of the original problem, its equivalent to checking (M, X), s 0 = E[true U (s 2 EG s 3 )] to be false

SAVCBS04 Check (M, X), s 0 =E[true U(s 2 EG s 3 )] Step 1. Atomic subformula s 2 is processed by ProcessCTL, which returns a labeling function L 1 = {(s 2, 1)} s0s0 s1s1 s2s2 s4s4 s3s3 msg? send/yes ack/yes send/no ack/yes msg? Comm send ack yes no Host system M

SAVCBS04 Step 1. Atomic subformula s 2 is processed by ProcessCTL, which returns a labeling function L 1 = {(s 2, 1)} s0s0 s1s1 s2s2 s4s4 s3s3 msg? send/yes ack/yes send/no ack/yes msg? Comm send ack yes no Host system M Step 2. Atomic subformula s 3 is processed by ProcessCTL, which returns a labeling function L 2 = {(s 3, 1)} Check (M, X), s 0 =E[true U(s 2 EG s 3 )]

SAVCBS04 Step 1. Atomic subformula s 2 is processed by ProcessCTL, which returns a labeling function L 1 = {(s 2, 1)} s0s0 s1s1 s2s2 s4s4 s3s3 msg? send/yes ack/yes send/no ack/yes msg? Comm send ack yes no Host system M Step 2. Atomic subformula s 3 is processed by ProcessCTL, which returns a labeling function L 2 = {(s 3, 1)} Step 3. Subformula s 3 is processed by Negation, which returns a labeling function L 3 = {(s 0, 1), (s 1, 1), (s 2, 1), (s 4, 1)} Check (M, X), s 0 =E[true U(s 2 EG s 3 )]

SAVCBS04 Step 4. Subformula EG s 3 is processed by ProcessEG, which constructs a witness graph G 1 = with an ID 2 and returns a labeling function L 4 = {(s 0, 2), (s 1, 2), (s 2, 2)} s0s0 s1s1 s2s2 s4s4 s3s3 msg? send/yes ack/yes send/no ack/yes msg? The host system M s0s0 s1s1 s2s2 send/yes Witness graph G 1 with ID 2 Check (M, X), s 0 =E[true U(s 2 EG s 3 )]

SAVCBS04 Step 4. Subformula EG s 3 is processed by ProcessEG, which constructs a witness graph G 1 = with an ID 2 and returns a labeling function L 4 = {(s 0, 2), (s 1, 2), (s 2, 2)} s0s0 s1s1 s2s2 s4s4 s3s3 msg? send/yes ack/yes send/no ack/yes msg? The host system M s0s0 s1s1 s2s2 send/yes Witness graph G 1 with ID 2 Step 5. Subformula s 2 EG s 3 is processed by Negation and Union, which finally return a labeling function L 5 = {(s 2, 2)} Check (M, X), s 0 =E[true U(s 2 EG s 3 )]

SAVCBS04 Step 6. The formula E[true U (s 2 EG s 3 )] is processed by procedure ProcessEU, which constructs a witness graph G 2 = with an ID 3 and returns a labeling function L f = {(s 0, 3), (s 1, 3), (s 2, 3), (s 3, 3), (s 4, 3)} s0s0 s1s1 s2s2 s4s4 s3s3 msg? send/yes ack/yes send/no ack/yes msg? The host system M s0s0 s1s1 s2s2 s4s4 s3s3 send/yes ack/yes send/no ack/yes Witness graph G 2 with ID 3 Check (M, X), s 0 =E[true U(s 2 EG s 3 )] L f (s 0 ) 1, this indicates that the truth of the original problem depends on communications. Then we shall see whether L f (s 0 ) can be evaluated to be true

SAVCBS04 Evaluate ID Expression L f (s 0 ) Evaluating L f (s 0 ) is a testing process with test-cases generated from the two witness graphs (details can be seen in the paper) Essentially, we are testing whether some communication trace (of bounded length) with m consecutive symbol pairs send yes is a successful test-case for the unspecified component X. s0s0 s1s1 s2s2 s4s4 s3s3 msg? send/yes ack/yes send/no ack/yes msg? Comm send ack yes no The host system M Errata: on page 6, in the paragraph above Note., … two consecutive symbol pairs … should be … m consecutive symbol pairs…

SAVCBS04 Summary A decompositional approach Reduces a system verification problem to a testing problem over the unspecified component A hybrid approach Combines model-checking with software testing techniques Advantages Stronger confidence about the reliability of the system Customization of testing over a component with respect to particular system requirement Reuse of intermediate verification results Automatic carrying out of the whole process Complete and sound algorithm with an appropriate bound

SAVCBS04 Related Work Specification-based testing [Heitmeyer et. al. 1999], generate test-cases from counter-examples produced by a model-checker Black-box checking [Peled et. al. 1999], test a single black-box against a given LTL formula Automatic assume-guarantee [Giannakopoulou et. al. 2004], automatic generation of assumptions of a component, uses a less expressive formalism (LTS) for properties Automatic deduction of model-checking conditions [Fisler et. al. 2001], has false negative [Xie & Dang 2004] LTL algorithms for systems with only one, finite-state unspecified component (FATES04) An automata-theoretic and decompositional approach to testing a system of concurrent black-boxes (submitted)

SAVCBS04 Ongoing Work Decompositional LTL model-checking Less-restricted system model Asynchronous communications Multiple unspecified components Unspecified components with infinite state space Case studies CORBA applications