Presentation is loading. Please wait.

Presentation is loading. Please wait.

Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C.

Similar presentations


Presentation on theme: "Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C."— Presentation transcript:

1 Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C. Ariel Pinto and Resit Unal Old Dominion University James T. Luxhøj Rutgers University Hampton Roads Area INCOSE Conference on Decision Analysis and Its Applications to Systems Engineering, Newport News, VA (November 17-18, 2009) National Aeronautics and Space Administration

2 National Aeronautics and Space Administration Jones,, et al. (2009) 2 Outline Background/Definitions Data Collection Process Results Concluding Remarks

3 National Aeronautics and Space Administration Problem Definition Aerospace technology managers need: To predict future technologies To assess progress toward R&D goals Aerospace technology portfolio decisions are difficult because: Very little time to acquire background data Uncertainty factors (e.g., politics, global economy, environment, funding) Prescriptive decision analysis methods Have been used for financial portfolio assessment Value for policy related decisions has been questioned Jones, et al (2009) 3

4 National Aeronautics and Space Administration Ralph Keeneys Suggestions for Making Better Decision Makers* Develop concepts, tools and procedures to help decision makers My experience is that many people, including well-educated people, have a very difficult time in structuring their decisions. They can get mixed up about the difference between fundamental concepts such as alternatives and objectives. Use real decisions, not just laboratory problems in decision research We have learned a great deal from all the laboratory settings where decision experiments have been conducted. There have also been some research studies of real decision problems. I feel there is much more to be gained by having more of this type of research. Teach people what they can and will learn and use …hundreds and thousands of people have had at least a course that included a substantial part on decision analysis and very few have probably ever conducted a formal decision analysis. Once we find out what people can and will learn and use, that should constitute the basis for much of our teaching of decision analysis. *Source: Ralph L. Keeney, Making Better Decision Makers, Decision Analysis, 1:4 (2004) Jones,, et al. (2009) 4

5 National Aeronautics and Space Administration Decision Analysis Usage in Aerospace Portfolio Development Aerospace managers have investigated the use of decision analysis methods for portfolio investment decisions: Commercial Aviation Safety Team (CAST) NASA Aviation Safety Program Future Aviation Safety Team (FAST) These technology assessments involved: Resource commitments (e.g., employee time, travel money, software acquisition, training) Assumption that decision analysis methods would improve aerospace technology assessment process Jones,, et al. (2009) 5

6 National Aeronautics and Space Administration Technology Assessment A process for measuring the impact of established or new technologies * Hans Mohr, Technology Assessment in Theory and Practice, Society for Philosophy and Technology, 4:4 (Summer 1999) Jones, et al (2009) 6

7 National Aeronautics and Space Administration Aerospace Technology Assessment Three different processes for examining impact of a set of technologies Technology assessment Technology forecasting Technology foresight Aerospace Technology Assessment Jones, et al (2009) 7

8 National Aeronautics and Space Administration Conceptual Model * 1. Identify Context & Understand Objectives 2. Identify Alternatives Aerospace Technology Assessment Decision Analysis Methods 3. Decompose & Model the Problem a. Model of Problem Structure b. Model of Uncertainty c. Model of Preferences 4. Choose the Best Alternative 5. Sensitivity Analysis 6. Implement the Chosen Alternative *Adapted from Robert T. Clemen, Making Hard Decisions, 2 nd Edition (1995) Jones, et al (2009) 8

9 National Aeronautics and Space Administration Purpose of Study What is known: Decision analysis methods in financial portfolio assessment Decision experiments in laboratory settings Technology assessment in medical R&D What is unknown: Decision analysis methods for policy related decisions Decision analysis methods in real decision problems Technology assessment in aerospace Jones, et al (2009) 9

10 National Aeronautics and Space Administration Study Overview Population was aerospace researchers with experience in one or more of the following: Aerospace program/project management Aerospace technology assessment Aerospace technology selection Aerospace R&D portfolio development Methods that were investigated in the study Decision trees Influence diagrams Criteria aggregation methods Explicit tradeoff approaches Participants were questioned about their usage of these methods for aerospace technology assessment Jones, et al (2009) 10

11 National Aeronautics and Space Administration Decision Trees 11 = decision node = chance node = consequence node Jones, et al (2009)

12 National Aeronautics and Space Administration Influence Diagrams 12 decision node chance node consequence node Jones, et al (2009)

13 National Aeronautics and Space Administration Criteria Aggregation Methods Methods in which two sets of aggregated indices are developed and used to evaluate the alternatives in the decision problem Methods in the category include: Analytical Hierarchy Process (AHP) Weighted Sum Model (WSM) 13 Jones, et al (2009) *E. Triantaphyllou, Multi-Criteria Decision Making Methods: A Comparative Study(1995) Example of Simple Weighted Sum Model *

14 National Aeronautics and Space Administration Explicit Tradeoff Approaches Decision analysis methods that are based on value functions Methods in this category include: Multi-Attribute Utility Theory (MAUT) Simplified Multi-Attribute Rating Approach (SMART) 14 Jones, et al (2009)

15 National Aeronautics and Space Administration Excluded Decision Analysis Methods Avoided decision analysis methods that are not popular in U.S. Real world applications are complex with large amounts of uncertainty Specific decision analysis methods that were excluded from study: Outranking methods (e.g., ELECTRE, TOPSIS) Optimization methods Analytic network process (ANP) Jones, et al (2009) 15

16 National Aeronautics and Space Administration Data Collection and Analysis Process Refine List of Candidate Survey Participants Develop Web- Based Survey Instrument Develop List of Pilot Participants Conduct Pilot Survey Review & Analyze Pilot Results Modify Survey Instrument Analyze Results Conduct Survey Legend Survey Development Data Collection Data Analysis Jones, et al (2009) 16

17 National Aeronautics and Space Administration Web-Based Instrument Development Several web-based services examined Questions developed based on several sources: Short surveys at professional meetings Validated research in decision analysis literature Identities of survey participants remained anonymous Jones, et al. (2009) 17

18 National Aeronautics and Space Administration Pilot Survey Conducted with subset of population (10 persons) Think aloud cognitive interviewing techniques used Solicitation of all thoughts and comments Manual recording of information during completion of online survey De-identification of subjects in final documentation 18 Jones, et al. (2009)

19 National Aeronautics and Space Administration Survey Instrument Modification Questions were modified, added or eliminated from the survey based on input from: Pilot survey comments Data analysis of pilot survey data Additional comments from other reviews (e.g., ODU IRB) Number of survey questions reduced from 70 to Jones, et al. (2009)

20 National Aeronautics and Space Administration Data Collection Overview Approval to conduct survey was obtained from NASA Langley and ODU Institutional Review Boards (IRBs) invitation was sent to 260 persons 154 total visits to survey website 16 partial responses 99 completes surveys Out of the 99 completed surveys 76% male, 24% female Highest degree level was 60% Masters, 21% Bachelors, 18% Doctorate, 1% Associates 72% employed as government civil servants Jones, et al. (2009) 20

21 National Aeronautics and Space Administration Job Functions Jones, et al. (2009) 21

22 National Aeronautics and Space Administration Aerospace Experience Jones, et al. (2009) 22

23 National Aeronautics and Space Administration Decision Trees 23 Jones, et al. (2009)

24 National Aeronautics and Space Administration Influence Diagrams 24 Jones, et al. (2009)

25 National Aeronautics and Space Administration Criteria Aggregation Methods 25 Jones, et al. (2009)

26 National Aeronautics and Space Administration Explicit Tradeoff Approaches 26 Jones, et al. (2009)

27 National Aeronautics and Space Administration Jones, et al. (2009) 27 Usage of Decision Analysis Methods Outside of a Classroom Environment

28 National Aeronautics and Space Administration Jones, et al (2009) 28 Categories of Non-ATA Usage of DA Outside of a Classroom Environment

29 National Aeronautics and Space Administration Additional Questions 29 Jones, et al. (2009)

30 National Aeronautics and Space Administration Additional Questions (contd) 30 Jones, et al. (2009)

31 National Aeronautics and Space Administration 31 Jones, et al. (2009) Additional Questions (contd)

32 National Aeronautics and Space Administration Additional Questions (contd) 32 Jones, et al. (2009)

33 National Aeronautics and Space Administration Additional Questions (contd) 33 Jones, et al. (2009)

34 National Aeronautics and Space Administration Likelihood of Future Usage of DA Methods 34 Jones, et al. (2009)

35 National Aeronautics and Space Administration Concluding Remarks This is a subset of the total data There are many additional questions in the study More formal analysis of the data was conducted using structural equation modeling techniques to test a set of hypotheses Survey participants believed that the successful use of decision analysis methods depends on: Selection criteria in the decision model Experience of the person that implements the method Reliability of the input data Training/education does not guarantee future use of a decision analysis method 35 Jones, et al. (2009)

36 National Aeronautics and Space Administration Questions? 36

37 National Aeronautics and Space Administration Your Title Here 37 SignOffPage


Download ppt "Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C."

Similar presentations


Ads by Google