1. Engineer’s Perceptions of Risk Lynne P. Cooper, Ph.D. Jet Propulsion Laboratory California Institute of Technology - and - Daniel J. Epstein Department of Industrial and Systems Engineering University of Southern California March 6, 2012 Presented at the USC CSSE Annual Research Review 2012 lynne.p.cooper@jpl.nasa.govlynne.p.cooper@jpl.nasa.gov or lynnecoo@usc.edulynnecoo@usc.edu National Aeronautics and Space Administration ©2012 L.P.Cooper

2. Project Risk Scenario 2 Experts System Design That doesn’t make sense InputsModelOutput x1x1 x2x2 x3x3 x4x4 xnxn Y What is this judgment based on? Team

3. “Risk” from a different perspective: One that includes: Emotions Intuition Relationships Collective belief 3 “Pre-Quantitative Risk”

4. Research Approach 4 Real world teams Working on high risk, high technology projects Over time Discussion Analysis: Language used Flow of conversation through topics Development of topics throughout project Actions taken

5. Observed Behavior 5 Very little discussion “about” risk Circular conversations Déjà vu discussions Very few formal decisions Quantification for justification – after decision was made General perception of decreasing risk

6. Language of Risk Rarely talked about risk explicitly But language of risk and uncertainty permeated team discussions 6

7. Components of Risk Discussions Goals Design Environment How things interact Bad things that could happen What they don’t know What they could influence/control Acceptable levels of risk 7

8. How Components Fit Together 8 Goal Design Environment Interactions Uncertainty Bad things that could happen Risk Action Ability to Influence Judgment of Acceptability = interactions

9. Visualizing Risk: Sliders 9 LowMediumHigh Too LittleAcceptable RiskToo Much Acceptable Risk Thresholds Degree of Risk

10. Aggregating Risk 10 Power System BatteriesSolar Arrays Site Selection Thermal Environment Energy Demands O&M Costs Not linear Highly dynamic Strange couplings

11. Results & Practical Implications (continued) 11 Very little discussion “about” risk Circular conversations Déjà vu discussions

12. Circular Conversations 12 Verbal Simulation of System Builds Intuition Identify Interactions Test Changes Insight into Nature of Relationships Diagnostic value Scope problem areas Intervention Value Accelerated cycles Model development

13. Déjà vu Discussions 13 Developing a Story Integrate ideas Refine messages Synchronization Improved external communications Diagnostic value Assess progress based on clarity Intervention Value Accelerate story development On-boarding

14. Discussion-based Evolution of Risk 14 Déjà vu Discussions Story High Confidence Risk Assessments Circular Conversations Evidence

15. In Summary Engineers base critical decisions on pre- quantitative conceptions of risk Pre-Quantitative Risk includes Interactions among goals, design elements, environments, risks, assessments of riskiness Broad view of uncertainty Ability to influence Judgments about the acceptability of risk Team discussion can be used as a diagnostic, as well as for interventions Visualization techniques can help develop shared understanding of risk 15

16. 16 “ The revolutionary idea that defines the boundary between modern times and the past is the mastery of risk: the notion that the future is more than a whim of the gods and that men and women are not passive before nature ” Peter L. Bernstein, 1996, p.1

17. Thank You 17