1. Department of Computer Engineering PROPANE An Environment for Examining the Propagation of Errors in Software Martin Hiller, Arshad Jhumka, Neeraj Suri Chalmers University of Technology Göteborg, Sweden {hiller, arshad, suri}@ce.chalmers.se

2. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 2 Overall Objectives A D C E B F Software reliability can be provided/increased by adding mechanisms that detect and correct data errors  wrappers, assertions, etc. (especially for black-box software) Given a software system and limited resources one would like to concentrate work on the most vulnerable/exposed parts of the software, i.e. … Where do upcoming errors propagate?  Where do upcoming errors propagate?

3. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 3 Outline Error Propagation & Basic Approach Error Propagation & Basic Approach Examples of results obtained by using PROPANE Examples of results obtained by using PROPANE –Aircraft arrestment system Overview of PROPANE Overview of PROPANE –Tool suite –Requirements & limitations Conclusions & some future directions Conclusions & some future directions

4. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 4 Error Propagation B F Error = A system state which is different from the state in a ”correct” execution of the system (i.e, not mutations or software defects) Modules in a software system have different levels of exposure and different ”ability” to break error propagation (i.e., different levels of error containment) – examining error propagation gives the developer a picture/profile of these levels. D C E A

5. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 5 Basic Approach  Error Injection A D C E B F 1.Generate Golden Run, i.e., an error free reference run 2.Generate Injection Run, i.e., a run in which an error (i.e. erroneous system state) is injected 3.Compare Injection Run with Golden Run to see which parts of the system were corrupted by the injected error PROPANE – Propagation Analysis Environment  By instrumenting the target software, PROPANE can, during execution, log individual variables and events and inject errors into individual variables. = probe = injection location

6. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 6 Example: Aircraft Arrestment System CLOCK CALC DIST_S PRES_A V_REG PRES_S ms_slot_nbr i mscnt pulscnt slow_speed stopped IsValue OutValueTOC2ADC TCNT TIC1 PACNT SetValue Rotation sensor Pressure sensor Pressure valve Computer Pressure valve Tape drum (original) Tape drum (mirror) Cable Target system overview Target software overview

7. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 7 Results Generated by PROPANE From low level to high level information From low level to high level information –Signal/Variable Plots –Propagation Signatures –Propagation Graphs –Propagation Summaries Data compilations that can be used for further analysis Data compilations that can be used for further analysis

8. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 8 Signal/Variable Plots One-cycle single-bit upset at t = 1500 ms SetValue erroneous at t = 1539 ms

9. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 9 Propagation Signatures Each injected error generates a Propagation Signature showing when and where the error propagated Each injected error generates a Propagation Signature showing when and where the error propagated

10. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 10 Propagation Graphs The Propagation Graph is generated in three different formats –PROPANE native format –GML (Graph Markup Language) – dot (part of the graphviz tool suite from AT&T Research) Probed location Incoming errors from locations earlier in the propagation path Outgoing errors to locations later in the propagation path error count t min / t avg / t max

11. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 11 Propagation Graph for PACNT Starting point Propagation path Each arc carries information regarding number of propagated errors and propagation time Variables along the propagation trajectory

12. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 12 Propagation Summary for PACNT Signal Error count Error rate t min t avg t max PACNT18401.000000 pulscnt18401.0000020 i1811.9840420 SetValue1275.69316134159 OutValue1275.69316134159 TOC21275.69336154161 ADC1265.688106294168 IsValue1202.6531556823467 slow_speed769.418020045890 mscnt1184.64347629826201 ms_slot_nbr1184.64347629826201 TCNT1184.64347629826201 TIC11184.64347629826201 Probed location (variable) For all incoming arcs to one node: Total error count and error rate Total error count and error rate Combined t min / t avg / t max Combined t min / t avg / t max Propagation Graphs and Propagation Summaries indicate which parts of the observed system state that are most vulnerable to the injected errors  perhaps a good location for error detection/recovery mechanisms?

13. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 13 Error Propagation from PACNT Highest Error Rate Lowest Error Rate ms_slot_nbri mscnt pulscnt slow_speed stopped IsValue OutValue TOC2ADC TCNT TIC1 PACNT SetValue CLOCK PRES_S V_REG PRES_A CALC DIST_S

14. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 14 Overview

15. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 15 PL PROPANE Library Static C Library providing the injection/logging API Static C Library providing the injection/logging API –propane_inject() /* Injects an error */ –propane_log_var() /* Variable probe */ –propane_log_event() /* Event probe */ Target must be instrumented with these functions and linked with the library Target must be instrumented with these functions and linked with the library Experiments are then set up using description files Experiments are then set up using description files – Error types (what errors to inject, e.g., bit-flips) – Error triggers (when to inject errors, e.g., once/periodically) – Active probes (perhaps not all instrumented variables/events have to be logged)

16. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 16 Instrumentation Example int spherical_volume( double radius ) { double volume; volume = 4.0 * (PI * pow(radius, 3.0)) / 3.0; return volume; } int spherical_volume( double radius ) { double volume; /* Injection location for radius */ propane_inject( IL_SPHERE_VOL, &radius, PROPANE_DOUBLE ); /* Probe the value of radius */ propane_log_var( P_RADIUS, &radius ); volume = 4.0 * (PI * pow(radius, 3.0)) / 3.0; /* Probe the value of volume */ propane_log_var( P_VOLUME, &volume ); return volume; } Original code Instrumented code At this point, instrumentation is unfortunately still a manual task. However, automation is the next step in the development of PROPANE.

17. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 17 PSC PROPANE Setup Creator Once the target is instrumented, experiments can be set up Once the target is instrumented, experiments can be set up Given information on injection locations, probes, error types and occurrences, PSC will create Given information on injection locations, probes, error types and occurrences, PSC will create –description files for running experiments –description files for data extraction and analysis

18. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 18 PCD PROPANE Campaign Driver After instrumentation and setup, the PCD runs the experiments After instrumentation and setup, the PCD runs the experiments –Invokes instrumented target executable containing the PL and all links to external modules, e.g., environment simulator Provides continuous information on experiment status and remaining work Provides continuous information on experiment status and remaining work User control User control –Pause –Abort –Skip

19. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 19 PDE PROPANE Data Extractor Extracts and analyses data from the raw readouts produced during experiments Extracts and analyses data from the raw readouts produced during experiments –Perform Golden Run Comparisons, i.e., compare an injection run with a ”clean” reference run to trace errors, generating a Propagation Signature –Collapse multiple propagation signatures into Propagation Graphs and Propagation Summaries –Prepare/compile data for further external analysis (using e.g. MatLab)

20. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 20 Requirements & Limitations Programming language Programming language –The library itself is written in C  the language in the target system must be able to interface with C-libraries Environment simulator Environment simulator –Stimuli to the target system must be provided by an environment simulator –PROPANE provides basic interfacing functionality such that control of the EnvSim can be made part of the experiment setup (~plug-in’s) Target instrumentation Target instrumentation –Difficulties with real-time applications that cannot be made to run in simulated time Current implementation focused towards single-node SW Current implementation focused towards single-node SW –Distributed SW not considered at this point

21. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 21 Conclusions PROPANE enables experimental evaluation of error propagation in SW PROPANE enables experimental evaluation of error propagation in SW –Propagation profiles indicate which parts of system state that are vulnerable  aid for equipping SW with error detection and recovery mechanisms We envision that PROPANE can be used as a design stage aid for analysis of single node SW We envision that PROPANE can be used as a design stage aid for analysis of single node SW Limitations apply for some apsects of real-time and distributed SW Limitations apply for some apsects of real-time and distributed SW

22. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 22 Some Future Directions Increased automation Increased automation –E.g. automatic instrumentation of target software Extended analysis capabilities Extended analysis capabilities –E.g. provide your own analysis plug-ins Open source Open source –Possibly in the near future

23. Department of Computer Engineering July 22, 2002 Martin Hiller PROPANE: An Environment for Examining the Propagation of Errors in Software ISSTA 2002 23 Further Information (and future download area) www.ce.chalmers.se/LDC/DEEDS