PINCETTE project: Validation of changes and upgrades in large software systems Unique challenges and suggested solutions Hana Chockler

2 Motivation: Challenges of validation of evolving software  Large software systems are usually built incrementally:  Maintenance (fixing errors and flaws, hardware changes, etc.)  Enhancements (new functionality, improved efficiency, extension, new regulations, etc.)  Changes are done frequently during the lifetime of most systems and can introduce new software errors or expose old errors  Upgrades are done gradually, so the old and new versions have to co- exist in the same system  Changes often require re-certification of the system, especially for mission-critical systems

3 "Upgrading a networked system is similar to upgrading software of a car while the car's engine is running, and the car is moving on a highway. Unfortunately, in networked systems we don't have the option of shutting the whole system down while we upgrade and verify a part of it.“ source: software update

4 PINCETTE Project – Validating Changes and Upgrades in Networked Software Checking for crushes Static Analysis Component Black box testing Dynamic Analysis Component Front end Methodology book Using function summaries Verifying only the change White box testing

5 What does it mean to validate a change in a software system? Equivalence checking – when the new version should be equivalent to the previous version in terms of functionality Changes in the underlying hardware Optimizations No crashes – when several versions need to co-exist in the same system, and we need to ensure that the update will not crash the system When there is no correctness specification, this is often the only thing we can check Checking that a specific bug was fixed A counterexample trace can be viewed as a specification of a behavior that needs to be eliminated in the new version Validation of the new functionality If a correctness specification for the change exists, we can check whether the new (or changed) behaviors satisfy this specification

6 Why is it validation of evolving software different from standard software validation? Software systems are too large to be formally verified or exhaustively tested at once Even if it is feasible to validate the whole system, often the process is too long and expensive and does not fit into the schedule of small frequent changes When validating the whole system, there is a danger of overlooking the change How can we use the fact that we are validating evolving software? If the previous version was validated in some way, we can assume that it is correct and not re-validate the parts that were not changed If the results of previous validation exist, we can use them as a basis for the current validation – especially useful when there are many versions that differ from each other only slightly The previous version can be used as a specification

7 After more than two years of PINCETTE: 1.How to verify an evolving system that does not have assertions? 2.How to verify only the change without relying on results of previous verification?

8 1.How to verify an evolving system that does not have assertions?

9 Dynamically Discovering Assertions to Support Formal Verification Motivation: “Gray-box” components (such as OTS components) – poor specifications, partial view of internal details Lack of specification complicates validation and debugging Lack of description of the correct behavior complicates integration Idea: Analyze gray-box components by dynamic analysis techniques: Monitor system executions by observing interactions at the component interface level and inside components Derive models of the expected behavior from the observed events Mark the model violations as symptoms of faults ©Leonardo Mariani, UniMiB

10 Dynamically Discovering Assertions at BCT (UniMiB tool) Combining dynamic analysis and model-based monitoring Combining classic dynamic analysis techniques (Daikon) with incremental finite state generation techniques (kBehavior) to produce I/O models and interaction models FSA are produced and refined based on subsequent executions Extracting information about likely causes of failures by automatically relating the detected anomalies Filtering false positives in two steps: Identify and eliminate false positives by comparing failing and successful executions with heuristics already experienced in other contexts Rank the remaining anomalies according to their mutual correlation and use this information to push the related likely false positives far from the top anomalies ©Leonardo Mariani, UniMiB

11 Extracting a set of valid assertions for regression Program V.0Program V.1 Testing set of traces Analysis Set of candidate assertions Formal methods Set of assertions that hold in V.0 Set of invalid assertions in V.1 Formal methods Set of assertions that hold in V.0 + Efficient and automatic regression verification user UniMiB+Oxford+USI

12 Parameters of the assertions discovery Scope of monitoring All lines Function invocations Loops Particular variables Output refinement Many violations – maybe leave only those in the slice of the change Look for a minimal set How to design the search algorithm? Can we check properties of the type “if P_1, P_2,…P_n are true then Q_1, Q_2,..Q_m” where each property may hold at a different code location?

13 1.how 2.How to verify only the change without relying on results of previous verification? The main idea: Verify only the change ignoring the rest of the program Using ExpliSAT Joint work with Sitvanit Ruah

14 What is ExpliSAT? A model-checker that combines explicit traversal of CFG with symbolic representation of data: Traverses CFG Ensure that we traverse only real paths by computing a representative path with real data values Builds a SAT formula for each path on the control flow graph and invokes a SAT solver Bugs-hunting heuristics can be added to change the order of traversal of the graph, so that the places where bugs are more likely to occur are traversed first The concolic approach is widely used in testing concrete+symbolic Heuristics direct the search towards areas in which bugs are more likely to occur

15 The path representative – checking that the symbolic path is real ExpliSAT traverse the CFG and create all possible control paths Symbolically represent data variables of all executions that follow the same control path A path representative for a certain control path is a valuation of every variable in the control path that holds the path constraint. For every legal control path there exists a path representative The representative is computed iteratively A Update the representative until A and look at command in A Use the representative prefix until A to decide which branch to take a control path with a representative is a legal control path The other branch may be infeasible – checked by trying to construct a representative

16 Ordering - Heuristically deciding in which order to traverse the control paths 1 1: x=input(); 2: if (x>N) then else 3:y=x; 4:y=N; 1: x=input(); 2: if (x>N) then else 4:y=N; 5: assert(y>=N); The particular heuristic depends on the verification goals – for updates, we use the update- checking heuristic

17 Extracting the part of the control- flow graph affected by a change change subgraph induced by the changed node initial state Backward reachability of the initial state from the changed node

18 ExpliSAT with update-checking ExpliSAT with update checking goto-ccSAT solver syntactic marking of changes Old version New version only the new version with marking is passed Checks paths that go through a changed node first result: goto- program with the change marking Parser + translation to a goto-program Cprover framework

19 Update-checking heuristics in ExpliSAT Turns the model- checker to a bug hunter Two options: Check only the change Check the change first, then continue checking the whole program Very efficient New bugs will be discovered quickly The whole process is complete

20 Effect of a change on functionality of the program Claim: in sequential programs, the algorithm from the previous slide finds all paths affected by a change straightforward Observation: in concurrent programs, a change can also affect other paths – future work Changes in global variables

21 Experimental results Compared performance of ExpliSAT on the whole program with the performance of ExpliSAT on the change only Not surprisingly, ExpliSAT on changes only is much faster Real-life example: a C++ program supplied by VTT Technical Research Center of Finland, computing the velocity and acceleration of a robot used in the European ITER project – a new type of a reactor based on energy of nuclear fusion ExpliSAT found a bug in update in several seconds On the whole program ExpliSAT doesn’t terminate

22 VTT (Finland) - the ITER EU Project: the thermo-nuclear reactor ITER is an international magnetic confinement fusion experimental project participated by the EU, India, Japan, China, Russia, South Korea, and the USA. The expected budget of ITER is €5B with 50% of funding provided by the EU. The reactor will be constructed in Cadarache, France, around ITER is one of the most important and complex research projects in the history of the EU. divertor: goes at the bottom of the reactor; should be changed periodically Robot that changes the divertor cassette

23