Presentation is loading. Please wait.

Presentation is loading. Please wait.

8/6/2002Safeware Engineering Corporation1 The Safety Risk of Requirements Incompleteness Jeffrey Howard Patrick Anderson.

Similar presentations

Presentation on theme: "8/6/2002Safeware Engineering Corporation1 The Safety Risk of Requirements Incompleteness Jeffrey Howard Patrick Anderson."— Presentation transcript:

1 8/6/2002Safeware Engineering Corporation1 The Safety Risk of Requirements Incompleteness Jeffrey Howard Patrick Anderson

2 8/6/2002 Safeware Engineering Corporation 2 Requirements Incompleteness Many incidents and accidents have been linked to flaws in real- time embedded system software Software-related errors are most often requirements errors, particularly incompleteness A specification is incomplete if required behavior is omitted or subject to more than one interpretation

3 8/6/2002 Safeware Engineering Corporation 3 Completeness Criteria Professor Nancy Leveson has compiled over 60 completeness criteria to address this problem, covering: –Human-Computer Interface –Trigger Events –Robustness –Nondeterminism –Values and Timing –Data Age –Feedback –And More Validated at JPL and used at Safeware SpecTRM-RL (SpecTRM Requirements Language) enforces these criteria

4 8/6/2002 Safeware Engineering Corporation 4 Todays Example Accident The importance of the criteria is easily demonstrated when they are ignored No one wants their embarrassing stories told in a conference session Everything you see here is false Everything you see here is true The ElectroShear 2000 Accident

5 8/6/2002 Safeware Engineering Corporation 5 ElectroShear 2000 Schematic

6 8/6/2002 Safeware Engineering Corporation 6 ElectroShear 2000 Shearing Pen Shearing pen, where shearing is done Entry and exit gates –Gate position sensors –Gate actuators –Gate locks Four mechanical arms mounted with electric trimmers Three sheep detection sensors –Digital camera –Weight plate –Thermal sensor Trimmer head sensors –Wool sensor –Skin flush-fit sensor

7 8/6/2002 Safeware Engineering Corporation 7 Normal Operation The system begins with entry gate open and exit gate closed Workers load a sheep and close the entry gate At least two of the three sheep detection sensors agree on the sheeps presence The system shears, adjusting trimmer position using the skin flush-fit sensor The wool detection sensor is ignored - the software detects its own completion After shearing, the exit gate opens Collect wool and repeat

8 8/6/2002 Safeware Engineering Corporation 8 The Accident A technician replaced the trimmer blades in a pen, then greased the entry gate While manually moving the gate, he lowered it to the point of closing it The system exited standby mode and began a shearing cycle The technician was caught in the pen and sheared The system behaved erratically during shearing, and three of the four mechanical arms were damaged

9 8/6/2002 Safeware Engineering Corporation 9 Technicians Statement My next work order was pen #22. The guys working with it had complained that the entrance gate was moving slowly and making some noise. As long as I was there, I was supposed to replace the trimmer heads. They were overdue. I got there and the guys unloaded the sheep they were putting into the pen. They put the pen into standby, so I lifted the exit gate, disconnected the weight plate, and went in to replace the trimmer heads. After that, I sprayed some grease on the gate tracks and worked it by hand a little to get the grease spread out. The machine just went crazy on me. It was a close shave.

10 8/6/2002 Safeware Engineering Corporation 10 The Investigation ElectroShears documentation jumbled requirements and design Accident investigators used SpecTRM-RL to explore the systems behavior SpecTRM-RL uses text attributes and AND/OR tables to represent software behavior

11 8/6/2002 Safeware Engineering Corporation 11 SpecTRM-RL

12 8/6/2002 Safeware Engineering Corporation 12 SpecTRM-RL (2)

13 8/6/2002 Safeware Engineering Corporation 13 Why did the system leave Standby Mode? Gates do not require frequent maintenance Maintenance procedures require the gates to stay open during maintenance Designers didnt anticipate entrance gate closings during standby mode Entrance gate closing during standby mode moves the pen into loaded mode

14 8/6/2002 Safeware Engineering Corporation 14 Shearing Pen Mode Logic

15 8/6/2002 Safeware Engineering Corporation 15 Criterion: Nondeterminism The behavior of the state machine should be deterministic (only one possible transition out of a state is applicable at any time. Automated tools can check this

16 8/6/2002 Safeware Engineering Corporation 16 Was the technician a ram? The system classified the technician as a sheep Two of the three sensors must agree –Digital Camera –Thermal Sensor –Weight Plate The camera mistook the human on all fours as a sheep The software still had obsolete input data queued from the disconnected weight plate

17 8/6/2002 Safeware Engineering Corporation 17 Weight Plate Input

18 8/6/2002 Safeware Engineering Corporation 18 Criterion: Data Age All inputs used in specifying output events must be properly limited in the time they can be used (data age). In SpecTRM-RL, all inputs have an Obsolete value

19 8/6/2002 Safeware Engineering Corporation 19 Why was the exit gate open? If the exit gate is open, the shearing cycle shouldnt start During the accident, it was open No escape for the technician When the system went into standby mode, exit gate position sensors were ignored The system came out of standby mode with an incorrect system model

20 8/6/2002 Safeware Engineering Corporation 20 Exit Gate Position Logic

21 8/6/2002 Safeware Engineering Corporation 21 Criterion: State Completeness The internal software model of the process must be updated to reflect the actual process state at initial startup and after temporary shutdown. SpecTRM-RL requires states to have an Unknown state value

22 8/6/2002 Safeware Engineering Corporation 22 What about the wool sensor? The wool sensor didnt detect wool being sheared That didnt stop the shearing cycle System engineers provided a wool sensor to detect the end of shearing The software keeps track of shearing completion as progress along the planned shearing path The software ignores the sensor, because its easier to detect the end of shearing as running out of planned shearing path

23 8/6/2002 Safeware Engineering Corporation 23 Criterion: Input Variable Completeness All information from the sensors should be used somewhere in the specification. SpecTRM-RL has an Appears In: attribute to identify orphaned inputs

24 8/6/2002 Safeware Engineering Corporation 24 Why were the arms flailing? Mechanical shearing arm motion became increasingly erratic By the end of the accident, three of the four arms were damaged by the controllers commands The shearing arm fine-adjustment sensor doesnt handle struggling humans well The data bus was flooded with commands and telemetry

25 8/6/2002 Safeware Engineering Corporation 25 Criterion: Environmental Capacity For the largest interval in which both input and output loads are assumed and specified, the absorption rate of the output environment must equal or exceed the input arrival rate. SpecTRM-RLs attributes address timing behavior

26 8/6/2002 Safeware Engineering Corporation 26 Why couldnt the operator help? An operator finally noticed the calamity The operator issued a stop command to the shearing pen The shearing pen didnt stop The designers didnt anticipate high communication load The stop command is just another order on the bus The operator had no way to know the order was lost

27 8/6/2002 Safeware Engineering Corporation 27 Criterion: Output Feedback This problem actually touches on a number of criteria –Inadequate display of state to operators –Inability to preempt lower priority tasks –Lack of feedback For the moment, focus on the lack of feedback to the operators SpecTRM attributes on outputs make feedback paths easy to check

28 8/6/2002 Safeware Engineering Corporation 28 Why didnt the entry gate open? When the operators realized the system wouldnt shut down, they commanded the gate open It didnt open Keeping gates closed during shearing is a safety feature The command that closes the gate isnt reversible. No notice was given to the operator.

29 8/6/2002 Safeware Engineering Corporation 29 Criterion: Reversibility Output commands should usually be reversible. SpecTRM-RL outputs have an attribute linking to the output that reverses their command

30 8/6/2002 Safeware Engineering Corporation 30 Investigation Findings There was no operator error in this accident. There were no component failures in this accident. Even the software didnt fail. It met its requirements, such as they were. The Electroshear 2000 was found to be unsafe. The culprit cited was the shearing pen control software. Software problems stemmed from incomplete requirements.

31 8/6/2002 Safeware Engineering Corporation 31 Completeness Criteria (2) The ElectroShear accident demonstrates several completeness critera –Nondeterminism –Data Age –State Completeness –Input Variable Completeness –Environmental Capacity –Output Feedback –Reversibility Consideration of these criteria could have prevented and/or reduced the severity of the accident

32 8/6/2002 Safeware Engineering Corporation 32 Summary The example may be fanciful, but the problems illustrated are quite real The completeness criteria were compiled from decades of research, accident and incident reports, and specification review SpecTRM-RL builds the criteria into a state of the art, analyzable, and executable requirements language

33 8/6/2002Safeware Engineering Corporation33 Discussion And/Or Questions

34 8/6/2002Safeware Engineering Corporation34 The End

Download ppt "8/6/2002Safeware Engineering Corporation1 The Safety Risk of Requirements Incompleteness Jeffrey Howard Patrick Anderson."

Similar presentations

Ads by Google