1. The Decision Analysis and Resolution (DAR) Process
Terry Bahill
Systems and Industrial Engineering
University of Arizona
©, , Bahill
This file is located at

2. CMMI
The CMMI model is a collection of best practices from diverse engineering companies.
Improvements to our organization will come from process improvements, not from people improvements or technology improvements.
CMMI provides guidance for improving an organization’s processes.
One of the CMMI process areas is Decision Analysis and Resolution, DAR.
Bahill

3. DAR
Programs and Functions select the decision problems that require DAR and incorporate them in their program plans (e.g. SEMPs).
DAR is a BAE SYSTEMS common process. Common processes are tools that the user gets, customizes and uses.
DAR is invoked throughout the whole program lifecycle whenever a critical decision is to be made.
DAR is invoked by IPT leads on programs, financial analysts, program core teams, etc.
Invoke the DAR Process in Webster work instructions, in gate reviews, in phase reviews or with other triggers, which can be used anytime in the system life cycle.
Bahill

4. Webster
BAE’s common processes are established by SP
Bahill

5. Typical decisions
Decision problems that may require a formal decision process
Trade studies (eng_cat.shtml#GU0238)
Bid/no-bid
Make-reuse-buy (PW A017.html)
Fagan inspection versus checklist inspection (FM xls)
Tool selection
Vendor selection
Cost estimating
Bahill

6. Purpose
“In all decisions you gain something and lose something. Know what they are and do it deliberately.”
Bahill

7. A Simple Model for Human Decision Making, Called Image Theory
Bahill

8. References
The following description of image theory is based on Beach and Connolly (2005) and Bruce Gissing’s Roadmap to Business Excellence.
L. R. Beach and T. Connolly, The Psychology of Decision Making: People in Organizations, Sage Publications, Thousand Oaks, CA, 2005.
B. Gissing, The Roadmap to Business Excellence,
A. T. Bahill and B. Gissing, Re-evaluating systems engineering concepts using systems thinking, IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, SMC-28(4): , 1998.
Bahill

9. Image theory*
Decision Makers (DMs) code their knowledge into three images.
The value image contains principles of behavior.
The trajectory image is the agenda of goals.
The strategic image contains the plans for implementing the goals.
This description of Image Theory models human decision making at a very high level.
Image Theory should be the first part of the DAR process.
Doing tradeoff studies is one particular type of decision making.
Bahill

10. The value image
consists of the DM’s vision, mission, values, morals, ethics, beliefs, evaluation criteria and standards for how things should be and how people ought to behave.
Collectively these are called principles.
They limit
the goals that are worthy of pursuit and
acceptable ways of pursuing these goals.
Potential goals and actions that contradict the principles will be unacceptable.
It is called the value image because it represents the DM’s vision about the state of events that conforms most closely to his or her principles.
Bahill

11. The trajectory image is the agenda of goals the DM wants to achieve.
The goals are dictated by the problem statement, principles, opportunities, desires, competitive issues and gaps encountered in the environment.
The goals are fed back to the value image.
The DM’s goal agenda is called the trajectory image, because it is his or her vision about how the future should unfold.
Bahill

12. The strategic image contains the plans for implementing the goals.
Each plan has two aspects:
tactics are the concrete behavioral aspects that deal with local environment conditions,
forecasts are the anticipation of the future that describe what might result if the tactics are successful.
The plans are also fed back to the value image.
The collection of plans is called the strategic image, because it represents the DM’s vision of what he or she is trying to do to achieve the goals on the trajectory image.
Bahill

13. Framing*
means embedding observed events in a context that gives them meaning.
The DM uses contextual information to probe his or her memory to find image constituents that are relevant to the decision at hand.
This provides information about the goals and plans that were previously pursued in this context.
If a similar goal is being pursued this time, then the plan that was used before may be reused.
We also say that framing can produce one of the cognitive biases.
Framing is the act of ‘placing a picture frame’ within the full reality of the decision situation, for the purpose of reducing sensory input and simplifying the decision. Anchoring is a psychological term that refers to focusing on the reference point when making decisions. For example, a shopper looking for a dog may anchor on the fact that a puppy is a Labrador Retriever, and ignore temperament, health and price. Anchoring and then insufficient adjustment occurs when a decision maker chooses an anchor and adjusts his or her judgments myopically with respect to it. For example, a person may anchor on the first new computer he or she sees, and adjusts insufficiently enough to consider rationally the characteristics of other computers seen later.
Bahill

14. Two types of decisions
Adoption decisions determine whether to add new goals to the trajectory image or new plans to the strategic image.
Progress decisions determine whether a plan is making progress toward achieving a goal.
Bahill

15. Adoption decisions
A new goal or plan can be added if it is compatible with the DM’s relevant principles, does not introduce unacceptable risk and does not interfere with existing goals or ongoing plans.
Adoption decisions are accomplished by
screening potential goals and plans one by one in light of relevant principles, existing goals and ongoing plans. If only one option passes screening, it is adopted.
If two or more options pass the screen, then a tradeoff study determines the best option from among the survivors.
Screening is the more common of these decision mechanism.
This preliminary screening stage and a secondary evaluating stage model probably originated with Prospect Theory.
Bahill

16. Progress decisions
use the plan to forecast the future.
If that future includes achieving a goal, then the plan is retained.
If the forecast does not include achieving the goal, then the plan is rejected and a new plan is adopted in its place.
Bahill

17. Two decision mechanisms
The incompatibility test screens options based on how well they fit the DM’s images.
The profitability test focuses on the quality of the outcomes associated with the options.
Bahill

18. The incompatibility test
screens options (plans and goals) based on their incompatibility with constituents* defined in the three images.
Each option’s incompatibility increases as a function of the weighted sum of the number of violations.**
Violations are defined as negations, contradictions, preventions, retardations or any other form of interference with the realization one of the images’ constituents.
If the weighted sum of the violations exceeds some rejection threshold, then the option is rejected, otherwise it is adopted.
Constituents are the principles (vision, mission, values, morals, ethics, beliefs and standards of behavior), goals, actions, agendas, and plans (tactics and forecasts).
Combining functions other than the weighted sum are seldom used.
Bahill

19. Profitability test
When more than one option survives the incompatibility screen, the DM chooses the best using a profitability test.
The profitability test is not a single decision mechanism.
It is a repertory of strategies such as maximizing subjective expected utility, satisficing and performing tradeoff studies.
The selected strategy depends on
characteristics of the choice,
characteristics of the environment,
characteristics of the DM.
Bahill

20. Image theory for organizations*
Decisions in organizations are made by individual DMs, often forming a consensus.
So for organizational decisions, we can use the individual decision making model that we have just developed.
The only major addition is the need for a case for change.
Stop here and ask the students if that model for human decision making makes sense.
Do they agree that that is how they make decisions?
If they agree, then go on.
Bahill

21. The need for change* People do not make good decisions.
A careful tradeoff study will help you overcome human ineptitude and thereby make better decisions.
Often we need a burning platform to get people to move.
Bahill

22. Rational decisions* One goal Perfect information
The optimal course of action can be described
This course maximizes expected value
This is a prescriptive model. We tell people that, in an ideal world, this is how they should make decisions.
There is one goal and everyone agrees upon it.
DMs have unlimited information and the cognitive ability to use it efficiently. They know all of the opportunities open to them and all of the consequences.
The optimal course of action can be described and it will, in the long run, be more profitable than any other.
A synonym often used for prescriptive model is normative model. In contrast a descriptive model explains what people actually do.
Von Neumann and Morgenstern (1947)
Bahill

23. Satisficing*
When making decisions there is always uncertainty, too little time and insufficient resources to explore the whole problem space.
Therefore, people cannot make rational decisions.
The term satisficing was coined by Noble Laureate Herb Simon in 1955.
Simon proposed that people do not attempt to find an optimal solution. Instead, they search for alternatives that are good enough, alternatives that satisfice.
Systems engineers do not seek optimal designs, we seek satisficing designs.
Systems engineers are not philosophers.
Philosophers spend endless hours trying to phrase a proposition so that it can have only one interpretation.
SEs try to be unambiguous, but not at the cost of never getting anything written.
H. A. Simon, A behavioral model of rational choice, Quarterly Journal of Economics, 59, , 1955.
Bahill

24. Humans are not rational*1
Mark Twain said,
“It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.”
Humans are often very certain of knowledge that is false.
What American city is directly north of Santiago Chile?
If you travel from Los Angeles to Reno Nevada, in what direction would you travel?
Most humans think that there are more words that start with the letter r, than there are with r as the third letter.
Our first example of irrationality is that often we have wrong information in our heads.
What American city is directly north of Santiago Chile?
Most Americans would say that New Orleans or Detroit is north of Santiago, instead of Boston
Or, if you travel from Los Angeles to Reno Nevada, in what direction would you travel?
Most Americans would suggest that Reno is northeast of LA, instead of northwest.
Which end of the Panama canal is farther West the Atlantic side or the Pacific side?
Most Americans would say the Pacific.
These examples were derived from Massimo Piattelli-Palmarini, Inevitable illusions: how mistakes of reason rule our minds, John Wiley & Sons, 1994.
Bahill

25. Illusions* We call these cognitive illusions.
We believe them with as much certainty as we believe optical illusions.
The previous slide gave examples of one type of cognitive illusion.
In the next slides we will give examples of a few more types.
A couple dozen more types are given in
Massimo Piattelli-Palmarini, Inevitable illusions: how mistakes of reason rule our minds, John Wiley & Sons, 1994.
Bahill

26. The Müller-Lyer Illusion*
Probably the most famous and most studied optical illusion was created by German psychiatrist Franz Müller-Lyer in 1889.
Which of the two horizontal line segments is longer?
Although your visual system tells you that the one on the left is longer, a ruler will confirm that they are equal in length.
Do you think that the slide's’ title is centered? It is.
Bahill

27. Bahill Stare at the black cross. When do the green dots come from?
This illusion is from
The illusion only works in PowerPoint presentation mode.
Another good web site for visual illusions is
Bahill

28. Bahill The upper-left quadrant is defined as rational behavior.
EV means expected value. SEV is subjective expected value.
In the next slides we will show how human behavior differs from rational behavior.
Edwards, W., "An Attempt to Predict Gambling Decisions," Mathematical Models of Human Behavior, Dunlap, J.W. (Editor),
Dunlap and Associates, Stamford, CT, 1955, pp
Bahill

29. Humans judge probabilities poorly*
People overestimate events with low probabilities, like being killed by a terrorist or in an airplane crash,
and underestimate high probability events, such as adults dying of cardiovascular disease.
The existence of state lotteries depends upon such overestimation of small probabilities.
At the right side of this figure,
the probability of a brand new car starting every time is very close to 1.0. But a lot of people put jumper cables in the trunk and buy memberships in AAA.
M. G. Preston and P. Baratta, An experimental study of the auction-value of an uncertain outcome, American Journal of Psychology, 61, pp , 1948.
Kahneman, D. and Tversky, A., Prospect Theory: An Analysis of Decision under Risk, Econometrica 46 (2) (1979),
Tversky and Kahneman, (1992)
Drazen Prelec, in D. Kahneman & A. Tversky (Eds.) “Choices, Values and Frames” (2000)
Animals exhibit similar behavior.
People overestimate low probabilities and do not distinguish much between intermediate probabilities. Rats show this pattern too (Kagel 1995).
People are more risk-averse when the set of gamble choices is better.
But humans also violate this pattern, and so do rats (Kagel 1995).
People also exhibit “context-dependence”: Whether A is chosen more often than B can depend on the
presence of an irrelevant third choice C (which is dominated and never chosen).
Context dependence means people compare choices within a set rather than assigning separate numerical utilities.
Honeybees exhibit the same pattern (Shafir, et al. 2002).
Animals are also risk averse, as defined about a dozen slides from here.
John Kagel, Economic Choice Theory: An Experimental Analysis of Animal Behavior, Cambridge University Press, 1995.
S. Shafir, T. M. Waite and B. H. Smith. “Context-dependent violations of rational choice in honeybees (Apis mellifera) and gray jays (Perisoreus
canadensis).” Behavioral Ecology and Sociobiology, 2002, 51,
Bahill

30. Monty Hall Paradox1* Bahill
Humans are not good at computing probabilities, as is illustrated by the Monty Hall Paradox. This paradox was invented by Martin Gardner and published in his Scientific American column in It is called the Monty Hall paradox because of its resemblance to the TV show Let’s Make a Deal. I have taken this version from Massimo Piattelli-Palmarini, Inevitable illusions: how mistakes of reason rule our minds, John Wiley & Sons, 1994.
I am running a game that I can repeat hundreds of times.
On a table in front of me are a stack of ten-dollar bills and three identical boxes, each with a lid.
You are my subject.
Here are the rules for each game.
You leave the room and while you are out, I put a ten-dollar bill in one of the three boxes.
Then I close the lids on the boxes.
I know which box contains the ten-dollar bill, but you don’t.
Now I invite you back into the room and you try to guess which box contains the money.
If you guess correctly, you get to keep the ten-dollar bill.
Bahill

31. Monty Hall Paradox2* Bahill Each game is divided into two phases.
In the first phase, you point to your choice.
(You cannot not open, lift, weigh, shake or manipulate the boxes.)
The boxes remain closed.
Bahill

32. Monty Hall Paradox3* Bahill
After you make your choice, I open one of the two remaining boxes.
I will always open an empty box (remember that I know where the ten-dollar bill is).
Bahill

33. Monty Hall Paradox4* Bahill
Having seen one empty box (the one that I just opened) you now see two closed boxes, one of which contains the ten-dollar bill.
Bahill

34. Monty Hall Paradox5* Now here is your problem.
Are you better off sticking to your original choice or switching?
A lot of people say it makes no difference.
There are two boxes and one contains a ten-dollar bill.
Therefore, your chances of winning are 50/50.
However, the laws of probability say that you should switch.
Leave this slide up for a while and let people discuss what they think.
Bahill

35. Monty Hall Paradox6*
The box you originally chose has, and always will have, a one-third probability of containing the ten-dollar bill.
The other two, combined, have a two-thirds probability of containing the ten-dollar bill.
But at the moment when I open the empty box, then the other one alone will have a two-thirds probability of containing the ten-dollar bill.
Therefore, your best strategy is to always switch!
This explanation is from Massimo Piattelli-Palmarini, Inevitable illusions: how mistakes of reason rule our minds, John Wiley & Sons, 1994.
Bahill

36. Utility
We have just discussed the right column, subjective probability.
Now we will discuss the bottom row, utility
Bahill

37. Utility
Utility is a measure of the happiness, satisfaction or reward a person gains (or loses) from receiving a good or service.
Utilities are numbers that express relative preferences using a particular set of assumptions and methods.
Utilities include both subjectively judged value and the assessor's attitude toward risk.
Bahill

38. Risk
Systems engineers use risk to evaluate and manage bad things that could happen, hazards. Risk is measured with the frequency (or probability) of occurrence times the severity of the consequences.
However, in economics and in the psychology of decision making, risk is defined as the variance of the expected value, uncertainty.* p1 x1 p2 x2
Risk, uncertainty A
1.0
$10 $0
none B
0.5 $5
$15
medium C $1
$19 $9
high
This table explains three bets: A, B and C. The p’s are the probabilities, the x’s are the outcomes, is the mean and is the variance. This table shows, for example, that half the time bet C would pay $1 and the other half of the time it would pay $19. Thus, this bet has an expected value of $10 and a variance of $9. This is a comparatively big variance, so the risk (or uncertainty) is said to be high. Most people prefer the A bet, the certain bet.
To model risk averseness across different situations the coefficient of variability is often better than variance.
Coefficient of variability = (Standard Deviation) / (Expected Value).
In choosing between alternatives that are identical with respect to quantity (expected value) and quality of reinforcement, but that differ with respect to probability of reinforcement humans, rats (Battalio, Kagel and MacDonald, 1985), bumblebees (Real, 1991), honeybees (Shafir, Watts and Smith, 2002) and gray jays (Shafir, Watts and Smith, 2002) prefer the alternative with the lower variance.
To avoid the confusion caused by system engineers and decision theorist using the word risk in two different ways, we can refuse to use the word risk and instead use ambiguity, uncertainty and hazards.
Bahill

39. Ambiguity, uncertainty and hazards*
Hazard: Would you prefer my forest picked mushrooms or portabella mushrooms from the grocery store?
Uncertainty: Would you prefer one of my wines or a Kendall-Jackson merlot?
Ambiguity: Would you prefer my saffron and oyster sauce or marinara sauce?
A little while ago, a wild fire was heading toward our house. We packed our car with our valuables, but we did not have room to save everything, so I put my wines in the swimming pool. We put the dog in the car and drove off. When we came back, the house was burned to the ground, but the swimming pool survived. However, all of the labels had soaked off of the wine bottles. Tonight I am giving a dinner party to celebrate our survival. I am serving mushrooms that I picked in the forest while we were waiting for the fire to pass. There may be some hazard here, because I am not a mushroom expert. We will drink some of my wine: therefore, there is some uncertainty here. You know that none of my wines are bad, but some are much better than others. Finally I tell you that my sauce for the mushrooms contains saffron and oyster sauce. This produces ambiguity, because you probably do not know what these ingredients taste like. How would you respond to each of these choices?
Hazard: Would you prefer my forest picked mushrooms or portabella mushrooms from the grocery store?
Uncertainty: Would you prefer one of my wines or a Kendall-Jackson merlot?
Ambiguity: Would you prefer my saffron and oyster sauce or marinara sauce?
Decisions involving these three concepts are probably made in different parts of the brain. Hsu, Bhatt, Adolphs, Tranel and Camerer [2005] used the Ellsberg paradox to explain the difference between ambiguity and uncertainty. They gave their subjects a deck of cards and told them it contained 10 red cards and 10 blue cards (the uncertain deck). Another deck had 20 red or blue cards but the percentage of each was unknown (the ambiguous deck). The subjects could take their chances drawing a card from the uncertain deck: if the card were the color they predicted they won $10, else they got nothing. Or they could just take $3 and quit. Most people picked a card. Then their subjects were offered the same bets with the ambiguous deck. Most people took the $3 avoiding the ambiguous decision. Hsu et al. recorded functional magnetic resonance images (fMRI) of the brain while their subjects made these decisions. While contemplating decision about the uncertain deck, the dorsal striatum showed the most activity and when contemplating decisions about the ambiguous deck the amygdala and the orbitofrontal cortex showed the most activity.
Ambiguity, uncertainty and hazards are three different things. But people prefer to avoid all three.
Bahill

40. Humans are not rational
Even if they had the knowledge and resources, people would not make rational decisions, because they do not evaluate utility rationally.
Most people would be more concerned with a large potential loss than with a large potential gain. Losses are felt more strongly than equal gains.
Which of these wagers would you prefer to take?*
$2 with probability of 0.5 and $0 with probability 0.5
$1 with probability of 0.99 and $1,000,000 with probability
$3 with probability of and -$1,999,997 with probability
They all have an expected value of $1
The $2 bet means I put down a $2 bill and flip a coin to see if you get it or not.
The $1 bet means I give you one dollar and a state lottery ticket. If the lottery ticket is a winner, you keep the $1 million, else you keep the dollar bill.
The $3 bet has consequences that you might have to give me two million dollars.
The $1 bet has the highest utility for most engineers.
The message of this slide can be dramatically demonstrated with two $2 bills, a coin, two $1 bills, a lottery ticket and the last two slides of this presentation.
Bahill

41. Gains and losses are not valued equally*
This slide also shows saturation.
This slide also shows the importance of the reference point: $10 to a poor man means a lot more than $10 to a rich man.
Kahneman, D. and Tversky, A., Prospect Theory: An Analysis of Decision under Risk, Econometrica 46 (2) (1979),
Massimo would prefer that we label the ordinate and abscissa as subjective worth and numerical value.
Bahill

42. Subjective expected utility
combines two subjective concepts: utility and probability.
Utility is a measure of the happiness or satisfaction a person gains from receiving a good or service.
Subjective probability is the person’s assessment of the frequency or likelihood of the event occurring.
The subjective expected utility is the product of the utility times the probability.
Savage (1954)
Bahill

43. Subjective expected utility theory
models human decision making as maximizing subjective expected utility
maximizing, because people choose the set of alternatives with the highest total utility,
subjective, because the choice depends on the decision maker’s values and preferences, not on reality (e.g. advertising improves subjective perceptions of a product without improving the product), and
expected, because the expected value is used.
This is a first-order model for human decision making.
Sometimes it is called Prospect Theory*.
Kahneman got the Nobel Prize in 2002 for his part in developing Prospect Theory.
Prospect theory is often called a descriptive model for human decision making.
Bahill

44. In the last two dozen slides, we showed how human behavior differed from rational behavior. Next we are going to show that tradeoff studies can help move you toward more rational decisions.
Bahill

45. Why teach tradeoff studies?
Because emotions, cognitive illusions, biases, fallacies, fear of regret and use of heuristics make humans far from ideal decision makers.
Using tradeoff studies judiciously can help you make rational decisions.
We would like to help you move your decisions from the normal human decision-making lower-right quadrant to the ideal decision-making upper-left quadrant.
Bahill

46. The Decision Analysis and Resolution Proces (DAR)
Bahill

47. Specific goals (SG)
A specific goal applies to a process area and addresses the unique characteristics that describe what must be implemented to satisfy the process area. The specific goal for the DAR process area is
SG 1 Evaluate Alternatives.
Bahill

48. Specific practices (SP)
A specific practice is an activity that is considered important in achieving the associated specific goal.
Practices are the major building blocks in establishing the process maturity of an organization.
Bahill

49. Specific Practice Name Example
Number
DAR
Specific Practice Name
Example
1.1
Decide if formal evaluation process is warranted
When to do a trade study
1.2
Establish Evaluation Criteria
What is in a good trade study
1.3
Identify Alternative Solutions
1.4
Select Evaluation Methods
1.5
Evaluate Alternatives
1.6
Select Preferred Solutions
Bahill

50. Bahill

51. When creating a process
the most important facets are
illustrating tasks that can be done in parallel
suggesting feedback loops
including a process to improve the process
configuration management
Bahill

52. A simple tradeoff study
Bahill

53. Decisions Humans make four types of decisions:
Allocating resources among competing projects*
Making plans, which includes scheduling
Negotiating agreements
Choosing amongst alternatives
Alternatives can be examined in series or parallel.
When examined in series it is called sequential search
When examined in parallel it is called a tradeoff or a trade study
“Tradeoff studies address a range of problems from selecting high-level system architecture to selecting a specific piece of commercial off the shelf hardware or software. Tradeoff studies are typical outputs of formal evaluation processes.”*
The task of allocating resources is not a tradeoff study, but it certainly would use the results of a tradeoff study.
The quote is probably from CMMI.
Bahill

54. History
Ben Franklin’s letter* to Joseph Priestly outlined one of the first descriptions of a tradeoff study.
Give the students a copy of the letter, which is available at page 24.
Bahill

55. Tradeoff Study Process*
Decide if Formal
Evaluation is
Needed
Problem
Statement
Select
Evaluation
Methods
Establish
Criteria
Identify
Alternative
Solutions
Proposed
Alternatives
Evaluate
Preferred
Formal
Evaluations
Perform
Expert Review
Present
Results
Put In
PPAL ∑
These tasks are
drawn serially,
but they are not
performed in a serial
manner. Rather, it is
an iterative process
with many feedback
loops, which are not shown.
Perform Decision Analysis and Resolution PS0317
Perform Formal Evaluation PD0240
When designing a process, put as many things in parallel as possible.
Bahill

56. Decide if Formal Evaluation is Needed
Problem
Statement
Select
Evaluation
Methods
Establish
Criteria
Identify
Alternative
Solutions
Proposed
Alternatives
Evaluate
Preferred
Formal
Evaluations
Perform
Expert Review
Present
Results
Put In
PPAL
RF.Decide Formal Evaluation
Bahill

57. Is formal evaluation needed? SP 1.1
Companies should have polices for when to do formal decision analysis. Criteria include
When the decision is related to a moderate or high-risk issue
When the decision affects work products under configuration management
When the result of the decision could cause significant schedule delays
When the result of the decision could cause significant cost overruns
On material procurement of the 20 percent of the parts that constitute 80 percent of the total material costs
RF.Guide Formal Evaluations
Bahill

58. Guidelines for formal evaluation, SP 1.1
When the decision is selecting one or a few alternatives from a list
When a decision is related to major changes in work products that have been baselined
When a decision affects the ability to achieve project objectives
When the cost of the formal evaluation is reasonable when compared to the decision’s impact
On design-implementation decisions when technical performance failure may cause a catastrophic failure
On decisions with the potential to significantly reduce design risk, engineering changes, cycle time or production costs
RF.Guide Formal Evaluations
Bahill

59. Establish Evaluation Criteria
Decide if Formal
Evaluation is
Needed
Problem
Statement
Select
Evaluation
Methods
Establish
Criteria
Identify
Alternative
Solutions
Proposed
Alternatives
Evaluate
Preferred
Formal
Evaluations
Perform
Expert Review
Present
Results
Put In
PPAL
RF.Establish Evaluation Criteria
Bahill

60. Establish evaluation criteria* SP 1.2
Establish and maintain criteria for evaluating alternatives
Each criterion must have a weight of importance
Each criterion should link to a tradeoff requirement, i.e. a requirement whose acceptable value can be more or less depending on quantitative values of other requirements.
Criteria must be arranged hierarchically. The top-level may be performance, cost, schedule and risk.
Program Management should prioritize these four criteria at the beginning of the project and make sure everyone knows the priorities.
All companies should have a repository of generic evaluation criteria.
Some people will do a tradeoff study when buying a house or a car,
but seldom for lesser purchases.
All companies should have a repository of good evaluation criteria that have been used.
Each would contain the following slots
Name of criterion
Description
Weight of importance (priority)
Basic measure
Units
Measurement method
Input (with expected values or the domain)
Output
Scoring function (type and parameters)
Bahill

61. What will you eat for lunch today?
In class exercise.
Write some evaluation criteria that will, help you decide.*
Criteria: Cost, Preparation Time, Tastiness, Novelty, Low Fat, Contains the Five Food Groups, Complements Merlot Wine, Distance to Venue
Bahill

62. Killer trades Evaluating alternatives is expensive.
Therefore, early in tradeoff study, identify very important requirements* that can eliminate many alternatives.
These requirements produce killer criteria.**
Subsequent killer trades can often eliminate 90% of the possible alternatives.
*If these very important requirements are performance related, then they are called key performance parameters.
**Killer criteria for today’s lunch: must be vegetarian, non alcoholic, Kosher, diabetic,
Bahill

63. Identify Alternative Solutions
Decide if Formal
Evaluation is
Needed
Problem
Statement
Select
Evaluation
Methods
Establish
Criteria
Identify
Alternative
Solutions
Proposed
Alternatives
Evaluate
Preferred
Formal
Evaluations
Perform
Expert Review
Present
Results
Put In
PPAL
RF.ID Alternative Solutions
Bahill

64. Identify alternative solutions, SP 1.3
Identify alternative solutions for the problem statement
Consider unusual alternatives in order to test the system requirements*
Do not list alternatives that do not satisfy all mandatory requirements**
Consider use of commercial off the shelf and in-house entities***
*The Creativity Tools Memory Jogger, by D. Ritter & M. Brassard, GOAL/QPC 1998, explains several tools for creative brainstorming.
**If a requirement cannot be traded off then it should not be in the tradeoff study.
***The make-reuse-buy process is a part of the Decision Analysis and Resolution (DAR) process.
Bahill

65. What will you eat for lunch today?
In class exercise.
List some alternatives for today’s lunch.*
Candidate meals: pizza, hamburger, fish & chips, chicken sandwich, beer, tacos, bread and water.
Be sure that you consider left-overs in the refrigerator.
Bahill

66. Select Evaluation Methods
Decide if Formal
Evaluation is
Needed
Problem
Statement
Select
Evaluation
Methods
Establish
Criteria
Identify
Alternative
Solutions
Proposed
Alternatives
Evaluate
Preferred
Formal
Evaluations
Perform
Expert Review
Present
Results
Put In
PPAL
RF.Select Evaluation Methods
Bahill

67. Select evaluation methods, SP 1.4
Select the source of the evaluation data and the method for evaluating the data
Typical sources for evaluation data include approximations, product literature, analysis, models, simulations, experiments and prototypes*
Methods for combining data and evaluating alternatives include Multi-Attribute Utility Technique (MAUT), Ideal Point, Search Beam, Fuzzy Databases, Decision Trees, Expected Utility, Pair-wise Comparisons, Analytic Hierarchy Process (AHP), Financial Analysis, Simulation, Monte Carlo, Linear Programming, Design of Experiments, Group Techniques, Quality Function Deployment (QFD), radar charts, forming a consensus and Tradeoff Studies
Additional sources include customer statements, expert opinion, historical data, surveys, and the real system.
Bahill

68. Collect evaluation data
Using the appropriate source (approximations, product literature, analysis, models, simulations, experiments or prototypes) collect data for evaluating each alternative.
Bahill

69. Evaluate Alternatives
Decide if Formal
Evaluation is
Needed
Problem
Statement
Select
Evaluation
Methods
Establish
Criteria
Identify
Alternative
Solutions
Proposed
Alternatives
Evaluate
Preferred
Formal
Evaluations
Perform
Expert Review
Present
Results
Put In
PPAL
RF.Evaluate Alternatives
Bahill

70. Evaluate alternatives, SP 1.5
Evaluate alternative solutions using the evaluation criteria, weights of importance, evaluation data, scoring functions and combining functions.
Evaluating alternative solutions involves analysis, discussion and review. Iterative cycles of analysis are sometimes necessary. Supporting analyses, experimentation, prototyping, or simulations may be needed to substantiate scoring and conclusions.
Bahill

71. Select Preferred Solutions
Decide if Formal
Evaluation is
Needed
Problem
Statement
Select
Evaluation
Methods
Establish
Criteria
Identify
Alternative
Solutions
Proposed
Alternatives
Evaluate
Preferred
Formal
Evaluations
Perform
Expert Review
Present
Results
Put In
PPAL
RF.Select Preferred Solutions
Bahill

72. Select preferred solutions, SP 1.6
Select preferred solutions from the alternatives based on evaluation criteria.
Selecting preferred alternatives involves weighing and combining the results from the evaluation of alternatives. Many combining methods are available.
The true value of a formal decision process might not be listing the preferred alternatives. More important outputs are stimulating thought processes and documenting their outcomes.
A sensitivity analysis will help validate your recommendations.
Bahill

73. Perform Expert Review ∑ Perform Expert Review Bahill
Decide if Formal
Evaluation is
Needed
Problem
Statement
Select
Evaluation
Methods
Establish
Criteria
Identify
Alternative
Solutions
Proposed
Alternatives
Evaluate
Preferred
Formal
Evaluations
Perform
Expert Review
Present
Results
Put In
PPAL ∑
RF.Expert Review of Trade off Studies
Bahill

74. Perform expert review1
Formal evaluations should be reviewed* at regular gate reviews such as SRR, PDR and CDR or by special expert reviews
Technical reviews started about the same time as Systems Engineering, in The concept was formalized with MIL-STD-1521 in 1972.
Technical reviews are still around, because there is evidence that they help produce better systems at less cost.
The Perform Expert Review process is located at PS0303
Note that this slide says that the formal evaluations should be reviewed.
It does not say that the results of the formal evaluations should be reviewed.
Bahill

75. Perform expert review2
Technical reviews evaluate the product of an IPT*
They are conducted by a knowledgeable board of specialists including supplier and customer representatives
The number of board members should be less than the number of IPT members
But board expertise should be greater than the IPT’s experience base
IPT stands for integrated product team or integrated product development team.
Bahill

76. Who should come to the review?
Program Manager
Chief Systems Engineer
Review Inspector
Lead Systems Engineer
Domain Experts
IPT Lead
Facilitator
Stakeholders for this decision
Builder
Customer
Designer
Tester
PC Server
Depending on the decision, the Lead Hardware Engineer and the Lead Software Engineer
Bahill

77. Present results
Present the results* of the formal evaluation to the original decision maker and other relevant stakeholders.
These results might be the preferred alternatives,
or they could be recommendations to expand the search, re-evaluate the original problem statement, or negotiate goals and capabilities with the stakeholders.
A most important part of these results is the sensitivity analysis.
Bahill

78. Put in the PAL
Formal evaluations reviewed by experts should be put in the organizational Process Asset Library (PAL) or the Project Process Asset Library (PPAL) (e.g. GDE 11 for M601)
Evaluation data for tradeoff studies come from approximations, analysis, models, simulations, experiments and prototypes. Each time better data is obtained the PAL should be updated.
Formal evaluations should be designed with reuse in mind.
Bahill

79. Manage the DAR process
The DAR Process Owner shall manage and improve the DAR process.
The DAR Process Owner will establish a change control board and review the DAR Common Process on a regular basis. This is a high-level review of the DAR Common Process. This review must evaluate the activities, status and results of the DAR process. For instance, it might address use of and training for the many methods of performing DAR.
Bahill

80. Closed Book Quiz, 5 minutes Fill in the empty boxes
Problem
Statement
Proposed
Alternatives
Evaluation
Criteria
Formal
Evaluations
Preferred
Solutions ∑
Bahill

81. Tradeoff Study Example
Bahill

82. Example: What method should we use for evaluating alternatives?*
Is formal evaluation needed? SP 1.1
Check the Guidance for Formal Evaluations
We find that many of its criteria are satisfied including “On decisions with the potential to significantly reduce design risk … cycle time ...”
Establish evaluation criteria, SP 1.2
Ease of Use
Familiarity
Killer criterion
Engineers must think that use of the technique is intuitive.
The title of this slide is the example that we will present in the next 18 slides.
In these next 18 slides, the phrases in pink will be the DAR specific practices (rectangular boxes of the process diagram) we are referring to.
Some people get confused by the recursion in this example.
I do a tradeoff study to select a tradeoff study tool.
Bahill

83. Example (continued)1 Identify alternative solutions, SP 1.3
Linear addition of weight times scores, Multiattribute Utility Theory (MAUT).* This method is often called a “trade study.” It is often implemented with an Excel spreadsheet.
Analytic Hierarchy Process (AHP)**
*MAUT was originally called Multicriterion Decision Analysis. The first complete exposition of MCDA was given in 1976 by Keeney, R. L., & Raiffa, H. Decisions With Multiple Objectives: Preferences and Value Tradeoffs, John Wiley, New York, reprinted, Cambridge University Press, 1993.
**AHP is often implemented with the software tool Expert Choice.
Bahill

84. Example (continued)2 Select evaluation methods, SP 1.4
The evaluation data will come from expert opinion
Common methods for combining data and evaluating alternatives include:
Multi-Attribute Utility Technique (MAUT), Decision Trees, Analytic Hierarchy Process (AHP), Pair-wise Comparisons, Ideal Point, Search Beam, etc.
In the following slides we will use two methods: linear addition of weight times scores (MAUT) and the Analytic Hierarchy Process (AHP)*
Sorry if this is confusing, but this example is recursive.
MAUT and AHP are both the alternatives being evaluated and the methods being used to select the preferred alternatives.
Bahill

85. Example (continued)3
Evaluate alternatives, SP 1.5
Let the weights and evaluation data be integers between 1 and 10, with 10 being the best. The computer can normalize the weights if necessary.
Bahill

86. Multi-Attribute Utility Technique (MAUT)1
In this example we are not using scoring functions, therefore the evaluation data are the Scores.
The evaluation data are derived from approximations, models, simulations or experiments on prototypes.
Typically the evaluation data are normalized on a scale of 0 to 1 before the calculations are done: for simplicity, we have not done that here.
The numbers in this example indicate that MAUT is twice as easy to use as AHP.
Assess evaluation data* row by row
Bahill

87. Multi-Attribute Utility Technique (MAUT)2
Weights are usually based on expert opinion or quantitative decision techniques.
Typically the weights are normalized on a scale of 0 to 1 before the calculations are done: I did not do that here.
How did I we get the weights of importance? I pulled them out of the blue sky. Is there a systematic way to get weights? Yes, there are many. One is the AHP.
Bahill

88. Analytic Hierarchy Process (AHP)
Bahill

89. AHP, make comparisons
Create a matrix with the criteria on the diagonal and make pair-wise comparisons*
If you had ten criteria, then this matrix would be ten by ten.
Bahill

90. AHP, compute weights Create a matrix Square the matrix Add the rows
Normalize*
Remember the numbers in the right column. They will go into the matrix seven slides from here.
Expert Choice has two methods for normalization, and they often give slightly different numbers.
It might be difficult to square large matrices, so Saaty (1980) gave 4 approximation methods.
AHP, exact solution
Raise the preference matrix (with forced reciprocals) to arbitrarily large powers, and divide the sum of each row by the sum of the elements of the matrix to get a weights column. (Dr. Bahill’s example, with a power of 2)
To compute the Consistency Index:
Multiply preference matrix by weights column
Divide the elements of this new column by the elements in the weights column
Sum the components and divide by the number of components. This gives λmax (called the maximum or principal eigenvalue).
The closer λmax is to n, the elements in the preference matrix, the more consistent the result.
Deviation from consistency may be represented the Consistency Index (C.I.) = (λmax – n)/(n-1)
Calculating the average C.I. from a many randomly generated preference matrices gives the Random Index (R.I.),
which depends on the number of preference matrix columns (or rows):
1,0.00; 2,0.00; 3,0.58; 4,0.90; 5,1.12; 6,1.24; 7,1.32; 8,1.41; 9,1.45; 10,1.49; 11,1.51; 12,1.48; 13,1.56; 14,1.57; 15,1.59.
The ratio of the C.I. to the average R.I. for the same order matrix is called the Consistency Ratio (C.R.). A Consistency Ratio of 0.10 or less is considered acceptable.
Saaty, T. L. The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation. New York, McGraw-Hill, 1980.
Saaty gives 4 approximation methods:
The crudest: Sum the elements in each row and normalize by dividing each sum by the total of all the sums, thus the results now add up to unity. The first entry of the resulting vector is the priority of the first activity (or criterion); the second of the second activity and so on.
Better: Take the sum of the elements in each column and form the reciprocals of these sums. To normalize so that these numbers add to unity, divide each reciprocal by the sum of the reciprocals.
Good: Divide the elements of each column by the sum of that column (i.e., normalize the column) and then add the elements in each resulting row and divide this sum by the number of elements in the row. This is a process of averaging over the normalized columns. (Dr. Goldberg’s example)
Good: Multiply the n elements in each row and take the nth root. Normalize the resulting numbers.
Bahill

91. In-class exercise
Use these criteria to help select your lunch today.
Closeness, distance to the venue. Is it in the same building, the next building or do you have to get in a car and drive?
Tastiness, including gustatory delightfulness, healthiness, novelty and savoriness.
Price, total purchase price including tax and tip.
Bahill

92. To help select lunch today1
closeness is ??? more important than tastiness,
closeness is ??? more important than price,
tastiness is ??? more important than price.
Closeness
Tastiness
Price
Filling in this table is an in-class exercise
Bahill

93. To help select lunch today2
closeness is strongly more important (5) than tastiness,
closeness is very strongly more important (7) than price,
tastiness is moderately more important (3) than price.
Closeness
Tastiness
Price 1 5 7 3
All of the students should get this far.
Bahill

94. To help select lunch today3
Closeness
Tastiness
Price
Weight of Importance 1 5 7
0.73
1/5 3
0.19
1/7
1/3
0.08
Some of the students might do this.
Bahill

95. AHP, get scores Compare each alternative on the first criterion Bahill
Remember the numbers in the right column. They will go into the matrix two slides from here.
Bahill

96. AHP, get scores2 Compare each alternative on the second criterion
Remember the numbers in the right column. They will go into the matrix on the next slide.
Bahill

97. AHP, form comparison matrix**
Combine with linear addition*
*The AHP software (Expert Choice) can also use the product combining function.
Of course there is AHP software (e. g. Expert Choice) that will do all of the math for you.
**The original data had only one significant figure, so these numbers should be rounded to one digit after the decimal point.
Bahill

98. Example (continued)4 Select Preferred Solutions, SP 1.6
Linear addition of weight times scores (MAUT) was the preferred alternative
Now consider new criteria, such as Repeatability of Result, Consistency*, Time to Compute
Do a sensitivity analysis
The AHP software computes an inconsistency index. If A is preferred to B, and B is preferred to C, then A should be preferred to C. AHP detects intransitivities and presents it as an inconsistency index.
Bahill

99. Sensitivity analysis, simple
In terms of Familiarity, MAUT was strongly preferred (5) over the AHP. Now change this 5 to a 3 and to a 7.
The result is robust.
Changing the scores for Familiarity does not change the recommended alternative.
This is good.
It means the Tradeoff study is robust with respect to these scores.
Bahill

100. Sensitivity analysis, analytic
Compute the six semirelative-sensitivity functions, which are defined as
which reads, the semirelative-sensitivity function of the performance index F with respect to the parameter  is the partial derivative of F with respect to  times  with everything evaluated at the normal operating point (NOP).
Bahill

101. Sensitivity analysis2
For the performance index use the alternative rating for MAUT minus the alternative rating for AHP*
F = F1 - F2 = Wt1×S11 + Wt2×S21 – Wt1×S12 –Wt2×S22
For a tradeoff study with many alternatives, where the rankings change often, a better performance index is just the alternative rating of the winning alternative, F1.
This function gives more weight to the weights of importance.
Bahill

102. Sensitivity analysis3 S11 is the most important
The semirelative-sensitivity functions*
S11 is the most
important
parameter. So go back and
reevaluate it.
We only care about absolute values.
If the sensitivity is positive it means when the parameter gets bigger, the function gets bigger.
If the sensitivity is negative it means when the parameter gets bigger, the function gets smaller.
Bahill

103. Sensitivity analysis4
The most important parameter is the score for MAUT on the criterion Ease of Use
We should go back and re-evaluate the derivation of that score
Bahill

104. Bahill

105. Example (continued)5 Perform expert review of the tradeoff study.
Present results to original decision maker.
Put tradeoff study in PAL.
Improve the DAR process.
Add some other techniques, such as AHP, to the DAR web course
Fix the utility curves document
Add image theory to the DAR process
Change linkages in the documentation system
Create a course, Decision Making and Tradeoff Studies
Improve the DAR process.
Add some other techniques, such as AHP, to the DAR web course, not done yet
Fix the utility curves document, done by Harley Henning Spring 2005
Add image theory to the DAR process, proposed for summer 2007
Change linkages in the documentation system, done Fall 2004
Create a course, Decision Making and Tradeoff Studies, done Fall 2004
Bahill

106. Quintessential example
A Tradeoff Study of Tradeoff Study Tools
is available at
Bahill

107. Generic goals (GG)
Achievement of a generic goal in a process area signifies improved control in planning and implementing the processes associated with that process area.
Generic goals are called “generic” because the same goal statement appears in (almost) all process areas.
Each process area has only one generic goal for each maturity level.
And the generic goal is different for each maturity level.
Bahill

108. Maturity level 2 generic goal
GG 2: The DAR process is institutionalized as a managed process.
A managed process is a performed process that is planned and executed in accordance with policy; employs skilled people having adequate resources to produce controlled outputs; involves relevant stakeholders; is monitored, controlled, and reviewed; and is evaluated for adherence to its process description.
Bahill

109. Maturity level 3 generic goal
GG 3 The DAR process is institutionalized as a defined process.
A defined process is establish by tailoring the selected process according to the organization’s tailoring guidelines to meet the needs of a project or organizational function. With a defined process, variability in how the process is performed across the organization is reduced and process assets, data, and learning can be effectively shared.
Bahill

110. Generic practices (GP)
Generic practices contribute to the achievement of the generic goal when applied to a particular process area.
Generic practices are activities that ensure that the processes associated with the process area will be effective, repeatable, and lasting.
Bahill

111. Generic practices1 GP 2.1: Establish an Organizational Policy,
Establish and maintain an organizational policy for planning and performing the DAR process.
The BAE solution
SP Organizational Business Practices
OM A001 Perform Decision Analysis and Resolution
RW A004 Perform Formal Evaluation
RF 1 Quantitative Methods for Tradeoff Analyses.doc …
RF 12 Manage and Improve the DAR Process.doc
These documents are located at Users at Bluelnk\Bludfs001\Shared\Users\Bahill_AT\Draft DAR Process Docs
And O:\ENGR_LIB\SysPCRDocs\Reference Docs
Bahill

112. Generic practices2
GP 3.1 Establish and maintain the description of a defined decision analysis and resolution process.
BAE company compliance documents
SP Organizational Business Practices
OM A001 Perform Decision Analysis and Resolution
RW A004 Perform Formal Evaluation
BAE program implementation evidence
Tailoring reports, program plans and trade studies with evidence of use of SP 1.2 to 1.6.
Bahill

113. Generic practices3 GP 2.2: Plan the Process,
Establish and maintain the plan for performing the DAR process.
Bahill

114. Generic practices4 GP 2.3: Provide Resources,
Provide adequate resources for performing the DAR process, developing the work products, and providing the services of the process.
GP 2.4: Assign Responsibility,
Assign responsibility and authority for performing the process, developing the work products, and providing the services of the DAR process.
GP 2.5: Train People,
Train the people performing or supporting the DAR process as needed.
Bahill

115. Generic practices5 GP 2.6: Manage Configurations,
Place designated work products of the DAR process under appropriate levels of configuration management.
GP 2.7: Identify and Involve Relevant Stakeholders,
Identify and involve the relevant stakeholders of the DAR process as planned.
GP 2.8: Monitor and Control the Process,
Monitor and control the DAR process against the plan for performing the process and take appropriate corrective action.
Bahill

116. Generic practices6
GP 3.2 Collect Improvement Information such as work products, measures, measurement results, and improvement information derived from planning and performing the decision analysis and resolution process to support the future use and improvement of the organization’s processes and process assets.
Bahill

117. Generic practices7 GP 2.9: Objectively Evaluate Adherence,
Objectively evaluate adherence of the DAR process against its process description, standards, and procedures, and address noncompliance.
GP 2.10: Review Status with Higher Level Management,
Review the activities, status, and results of the DAR process with higher level management and resolve issues.
Bahill

118. Example Examples of trade studies are given in
O:\ENGR_LIB\DAR\DAR Training\Web-based DAR Course\dar_index.html
Bahill

119. Webster Tradeoff Study References
Utility Curves (Trade-off Study) FM
Evaluate Design Solutions RW A010
Trade-off Study Matrix (template) FM
Bahill

120. Webster DAR References
Organizational Business Practices SP
Perform Decision Analysis and Resolution OM A001
Perform Formal Evaluation RW A004
RF.QM Tradeoff Analyses
RF.Decide Formal Evaluation
RF.Guide Formal Evaluations
RF.Other DAR Methods
RF.Establish Evaluation Criteria
RF.ID Alternative Solutions
RF.Select Evaluation Methods
RF.Evaluate Alternatives
RF.Select Preferred Solutions
RF.Expert Review of Trade off Studies
RF.Retention Formal Decisions
RF.Manage Improve DAR
Bahill

121. Bahill

122. How to print To print this file, do this one time. View
Color/grayscale
Grayscale
Settings
Light grayscale
Close grayscale view
Bahill