CSC2108: Automated Verification or Everything you Wanted to Know about Model-Checking Ü Instructor: Marsha Chechik Ü Bahen 3248, Ü TA: Shiva Nejati Ü Web page:

Software Engineering and Formal Methods

Motivation “To err is human but to really foul things up requires a computer” Farmer’s Almanac for 1978 “Capsules of Wisdom” “ 60% of all major industrial disasters from 1921 to 1989 occurred after 1975” Nancy Leveson “Safeware: Computers and Technology”

Example: auto-pilot Problem: “Design a part in auto-pilot that avoids collision with other planes.” Solution: “When distance is 1km, give warning to other plane and notify pilot. When distance is 300m, and no changes in the course of other plane were noticed, go up to avoid collision”

Problem with solution Ü Both planes have the same software. Both go up...

This happens in real software! Ü Some famous bugs  NASA Space Rover, Intel floating point processor, etc. Ü Hard to predict all behaviours!  US aircraft went to southern hemisphere and … flipped when crossing the equator  Air traffic controller: US to Britain.  It never dealt with problem of 0 degrees longitude.  Result: software “folded” Britain along Greenwich Meridian  Software written for US F-16  accidents when reused in Israeli aircraft flown over the Dear Sea (altitude < sea level)  Year 2000 problem

Yet more such examples Ü NASA Space Shuttle software (in use since 1980)  16 severity-level 1 software errors  8 remained in code that was used in flights  none encountered during flights  total size - only 400,000 words

So, why not test? Testing only shows presence of bugs not their absence!

Formal Methods Mathematically-based techniques for describing properties of systems Ü Provide framework for  Specifying systems (and thus notion of correctness)  Developing systems  Verifying correctness  Of implementation w.r.t. the specification  Equivalence of different implementations Ü Reasoning is based on logic  Amenable to machine analysis and manipulation  In principle, can verify everything that is true in the system!  Given enough time, skill and patience

Why aren’t FMs used more? “Formal methods can revolutionize development!” “Formal methods are difficult, expensive, not widely useful and for safety-critical systems only”

… and one more problem Ü Need to know what to build (specification) before you start building System engineering Analysis Design Coding Testing Maintenance “water-fall” model Ü Unrealistic!  May need to discover what to build iteratively  Software changes all the time

Formal Methods “Light” Ü Partial application of formal methods  only parts of systems are specified Ü Emphasis on analysis of some properties  security, fairness, deadlock freedom, rather than complete verification Ü Debugging rather than assurance Ü Automation Most successful lightweight technique: Model-Checking

Overview of Automated Verification Answer + Counter-example Answer + Counter-example SW/HW artifact SW/HW artifact Correctness properties Correctness properties Temporal logic Temporal logic Model of System Model of System Model Extraction Model Extraction Translation Checker Engine Checker Engine abstraction Correct?

CTL Model-Checking Ü CTL: Branching-time propositional temporal logic Ü Model - a tree of computation paths Ü Example: Kripke Structure Tree of computation S1 S2 S3 S2 S1 S3 S1 S3 S2 S1 S3 S1 S3

p qrp qr  p q r pq rpq r s0s0 s2s2 s1s1 Models: Kripke Structures Ü Conventional state machines  M =  S is a (finite) set of states  A is a (finite) set of propositional variables  s 0 is a unique initial state (s 0  S)  I: S  2 A is a labeling function that maps each state to the set of propositional variables that hold in it  R  S  S is a (total) transition relation

Propositional Variables  Fixed set of atomic propositions {p, q, r}  Atomic descriptions of a system  “Printer is busy”  “There are currently no requested jobs for the printer”  “Conveyer belt is stopped”  How to choose them?  Should not involve time!

CTL: Computation Tree Logic propositional temporal logic. allows explicit quantification over possible futures Ü Syntax: True (  ) and False (  ) are CTL formulae; propositional variables are CTL formulae; if  and  are CTL formulae, then so are:  ,   ,    EX  ---  holds in some next states; EF  --- along some path,  is true in a future state; E[  U  ] --- along some path,  holds until  holds; EG  --- along some path,  holds in every state  Universal quantification: AX , AF , A[  U  ], AG 

Examples  EX (exists next)  AX (all next)      EG (exists global)           AG (all global) 

Examples, Cont’d  EF (exists future)    AF (all future)     EU (exists until)      AU (all until) 

CTL (Cont’d) Ü Examples:  Properties that hold:  (EX p)(s 0 )  (A[p U q])(s 0 )  ( EX AF p )(s 0 )  Properties that fail:  (A[  p U q])(s 0 ) p qrp qr  p q r pq rpq r s0s0 s2s2 s1s1

Some Statements To Express  It is possible to get to a state where started holds, but ready does not hold  EF (started   ready)  When a request occurs, it will eventually be acknowledged  AG (request  AF acknowledge)

CTL Model-Checking Ü Receive:  Kripke structure K  Temporal logic formula  Ü Assumptions:  Finite number of processes  Each having a finite number of finite-valued variables  Finite length of a CTL formula Ü Algorithm:  Label states of K with subformulas of that  are satisfied there and working outwards towards .  Output states labeled with  Example: EX AG (p  E[p U q])

CTL Model-Checking (Cont’d) EX   Label any state with EX  if any of its successors are labeled with  AF   If any state s is labeled with , label it with AF   Repeat: label any state with AF  if all of its successors are labeled with AF  until there is no change  EX     , AF  , AF  , AF  , AF  , AF  , AF 

Counterexamples Ü Explain:  Why the property fails to hold  to disprove that  holds on all elements of S, produce a single element s  S s.t.  holds on s.  counterexamples restricted to universally-quantified formulas  counterexamples are paths (trees) from initial state illustrating the failure of property pqrpqr  p q r p qrp qr s0s0 s2s2 s1s1  AG p pqrpqr s0s0  p q r s1s1  AX p  AX q pqrpqr s0s0 p qrp qr s2s2  p q r s1s1

Ü Idea of model-checking  recursively go through the structure of the CTL property…  associating each subformula with a set of states where each subproperty is true Ü Symbolic model-checking  effective cure for state explosion problem  use symbolic representation for sets of states  use symbolic representation for transition relation  use binary decision diagrams (BDDs) to encode these Ü Example:  x  y in classical logic Symbolic model-checking x y FT F T T F

Questions: Ü Where does the Kripke structure come from?  Computing it from several concurrent processes  Models of concurrency: maximum parallelism, interleaving  What happens if Kripke structure is infinite?  Abstraction!  How to represent large Kripke structures?  Abstraction!  Symbolic encoding of transition relation  SAT-based encoding (BMC)  Symmetry (not part of the course)  Compositional verification Ü What other methods are there?

2108: Automated Verification A course on model-checking technology Ü Theory:  Different temporal logics (CTL, LTL, mu-calculus)  Foundations:  Automata theory (automata on infinite strings, tree automata, alternating tree automata), lattice theory, fixpoint computations  Introduced as needed!  Different model-checking approaches:  Explicit-state, Symbolic, Bounded (BMC)  Game-theoretic  Automata-theoretic Ü Tools:  SMV, SPIN, Concurrency workbench

Automated Verification (Cont’d) Ü Making this work:  Decision diagrams, SAT solvers  Compositional verification  Partial order reduction  Abstraction and refinement Ü Some applications  Debugging  Requirements monitoring  Test case generation Ü Extensions  Query-checking  Model-checking over partial and inconsistent systems  Etc.

2108: Pragmatics Ü Time: Wednesdays, 2-5 p.m. Ü Readings:  Book:  Clarke, Grumberg, Peled. Model-Checking. MIT Press, 1999  Many research papers Ü Course work:  2-3 small modeling exercises  Regular (small) paper-and-pencil homeworks  Research project  Possibly presentation of a topic related to model-checking Ü Prerequisites:  Discrete mathematics, general inclination towards theory  Prior experience with concurrent systems a plus

Pros and Cons of Model-Checking Ü Often cannot express full requirements  Instead check several smaller properties Ü Few systems can be checked directly  Must generally abstract Ü Work better for certain types of problems  Very useful for control-centered concurrent systems  Avionics software  Hardware  Communication protocols  Not very good at data-centered systems  User interfaces, databases

Pros and Cons, Cont’d Ü Largely automatic and fast Ü Better use for debugging  … rather than assurance Ü Testing vs model-checking  Usually, find more problems by exploring all behaviours of a downscaled system than by testing some behaviours of the full system

Readings Ü List of readings on the web site  Chapter 1 of the textbook  Several papers on formal methods in general. Links from the reading list on the course web page Ü For next week:  Chapter 3 of the textbook