Applied Information Economics: Kickoff for Risk Return Analysis

Empirical Decision Theory What is AIE? Applied Information Economics is the practical application of scientific and mathematical methods to quantify the value of IT-enabled business investments Economics Decision/Game Theory Empirical Decision Theory Operations Research Applied Information Economics Modern Portfolio Theory Statistics Information Engineering Information Theory Software Metrics

Real Solutions to… …the economics of information …the economics of IT infrastructure …the economics of risk …the economics of labor reduction when headcount is not reduced Bottom Line: AIE assesses and prioritizes IT investments based on quantitative and economically rational methods

What Do the Critics Say? “Quantifying the risk and comparing its risk/return with other investments sets AIE apart from other methodologies. It can substantially assist in financially justifying a project -- especially projects that promise significant intangible benefits.” The Gartner Group “AIE represents a rigorous, quantitative approach to improving IT investment decision making…..this investment will return multiples by enabling much better decision making. Giga recommends that IT executives learn more about AIE and begin to adopt its tools and methodologies, especially for large IT projects.” Giga Information Group “AIE-like methods must become the standard way to make (IT) investment decisions.” Forrester Research, Inc.

Basic Risk/Return Analysis Organization Procedure 3. Make Measure- ments 4. Conduct VIA 5. Conduct Risk/Return Analysis 6. Make Recommenda- tions 1. Describe & Classify 2. Clarify Decision Model Tools The above overview of the RAVI method illustrates the different components: In practice, the RAVI method describes a procedure implemented by an organization and based on spreadsheet tools. A RAVI evaluation produces a document used to decide whether or not to start up an investment. Deploying the RAVI method in a company consists in training the different people involved in the decision-making process and in customizing the method to take account of the particular requirements of the company (structure of the project portfolio, observed project cancellation rate, reliability of development cost evaluations, corporate investment policy). Each stage in the process generates a part of the final document which will be given to the decision-maker. Certain steps consist in gradually completing a spreadsheet which is an essential part of the evaluation deliverables. The calculations performed using the spreadsheet provide the risk and return. The evaluation process is optimized inasmuch as it is used to take a decision as soon as sufficient information is gathered. As far as organizational matters are concerned, the RAVI method advocates compliance with certain principles in order to increase objectivity in the evaluation process. The method sets out to improve the objectivity of the decision-making processes by implementing different roles which mitigate the consequences of any conflicts of interest. AIE Deliverables AIE

Describe, Classify & Plan 3. Conduct Measurements 4. Conduct VIA 5. Conduct Risk & Return Analysis 6. Make Recommendations 1. Classify & Plan 2. Clarify Intangibles & CBA Purpose: Agree on the specific investment to be analyzed, Agree on the specific question to be answered, Plan the rest of the analysis

Clarify The Decision Model 3. Make Measure- ments 4. Conduct VIA 5. Conduct Risk/Return Analysis 6. Make Recommenda- tions 1. Describe & Classify 2. Clarify Decision Model During Clarification we translate the “Intangibles” into measurable units These are ultimately modeled in a spreadsheet

Understanding Measurement (The Measurement.com approach) Gilb’s Law “Anything can be measured in a way which is superior to not measuring it at all” The perceived impossibility of measurement is an illusion caused by not understanding: the Concept of measurement the Object of measurement the Methods of measurement See my “Everything is Measurable” article in CIO Magazine (got to “articles” link on www.hubbardresearch.com

Conduct Measurements 3. Make Measure- ments 4. Conduct VIA 5. Conduct Risk/Return Analysis 6. Make Recommenda- tions 1. Describe & Classify 2. Clarify Decision Model We use the variety of measurement methods previously discussed We usually start with what we know now (i.e. calibrated estimates) More elaborate measurements (large controlled experiments or surveys) are only taken if we can show they are economically justified

Calibrated Estimates Measuring your own uncertainty about a quantity is a general skill that can be taught with a measurable improvement Studies show that most managers are statistically “overconfident” when assessing their own uncertainty Training can “calibrate” people so that when they provide a 90% confidence interval, it still has a 90% chance of being right (even though it is subjective) When asked to provide a subjective 90% confidence interval, most managers provide a range that only has about a 40%-50% chance of being right Perceived 90% Confidence Interval Actual 90% Confidence Interval

Calculate the Value of Information 3. Make Measure- ments 4. Conduct VIA 5. Conduct Risk/Return Analysis 6. Make Recommenda- tions 1. Describe & Classify 2. Clarify Decision Model The value of additional information can be calculated for each uncertain variable in the analysis Measurement efforts will be more productive by focusing on variables that matter the most (results are often surprising) This method is based on the probability of a change in a decision with additional information and the difference in the value of the decision The value of information is derived by the following reasoning: information reduces uncertainty, reducing uncertainty improves the quality of the decisions, better decisions improve performance better performance makes for better profits. A bad decision consists in starting up an investment for which the real outcome is unfavorable (loss), but also in not starting up an investment for which the outcome is favorable (loss of opportunity). The entire investment project can be characterized in terms of the mean of the possible potential losses as a result of bad decisions. This mean is calculated using a spreadsheet, and simulating a large number of scenarios. For each scenario, the possible values of the different parameters are taken at random according to their probability profiles. This criterion, the mean of the potential losses or Expected Opportunity Loss (EOL), is calculated again for a variant of the project where perfect information is supposed to be available for a particular parameter. The difference in the two values, that for the entire project, and that for this particular variant of it, is the value of having perfect information on that parameter, in other words its contribution to overall performance. This process is repeated for each parameter to indicate the most critical parameters. One practical rule consists in assigning between 2 and 20% of the corresponding value to reduce uncertainty on the most critical parameters. A more advanced method can be used to determine the optimum value of this effort. $$$ $

The Economic Value of Information The Decision Theory Formula: What it means: Information reduces uncertainty Reduced uncertainty improves decisions Improved decisions satisfy business objectives (by definition)

Conduct Risk/Return Analysis 3. Make Measure- ments 4. Conduct VIA 5. Conduct Risk/Return Analysis 6. Make Recommenda- tions 1. Describe & Classify 2. Clarify Decision Model Administrative Cost Reduction 5% 10% 15% % Improvement in Customer Retention 10% 20% 30% Total Project Cost $2 million $4 million $6 million ROI 0% 50% 100%

Make Recommendations The recommendations include: Deliverables include Measure- ments 4. Conduct VIA 5. Conduct Risk/Return Analysis 6. Make Recommenda- tions 1. Describe & Classify 2. Clarify Decision Model The recommendations include: To accept or reject the investment Possible modifications to the proposed investment Various risk management tactics Deliverables include The written report The spreadsheet The presentation

Overview of RRA Analysis Classification 5% 10% 15% 20% 30% $2 mill $4 mill $6 mill Measurements Value of Info. $ Intangibles “Customer Satisfaction” “Strategic Alignment” “Technology Risk” “Information Quality” etc. $$$ $$ Measurables Errors in Decision X Change to Strategic Measure M Productivity in Activity Y Chance of cancellation, etc. Each company will decide on its own what “compliance”, “strategic” and “economic” mean precisely. Also, they will determine their own boundaries on the classification chart and the specific analysis requirements in each “zone” of the chart. Risk Calculate Risk/Return Position "expected" ROI 50% 100% 150% 200% 250% -50% 0% Probability of a negative ROI Probability of a positive ROI Organization's investment limit Acceptable region of investment Return

Workshops Much of the initial data gathering is from a series of workshops We need to schedule 5-6 workshops for the following activities: Define & Classify the investment Clarify Decision Model Measurement (initial) Calibration Estimation

Defining the Investment What is the objective of this investment? (A one-sentence description of why) What costs are unique to this investment? What benefits are unique to this investment? What are the risks of the investment? What “decision dimensions” are important besides just an accept/reject recommendation? Is all of the investment optional? The decision is analyzed on behalf of which investor?

The Concept Of Measurement Sometimes one believes that a thing is “immeasurable” only because they do not actually understand the concept of measurement The “Measurement Theory” definition of measurement: “A measurement is an observation that results in information (reduction of uncertainty) about a quantity.” Any “reduction of uncertainty” about a quantity can be of value ?

Real-world Measurements vs. Ideal Values Ideal Values: Point Real-world Meas. Normal Distribution Uniform Distribution Lognormal Distribution Hybrid Threshold confidence 15% 85%

The Object of Measurement If a thing seems like and immeasurable “intangible” it may just be ill-defined Often, if we can define what we mean by a certain “intangible” we find ways to measure it ?

The Clarification Chain AIE assumes that if a benefit or cost is defined unambiguously, then it is measurable. If it is “Better” it is different in some relevant way... If it is relevantly different then it is observable... If it is observable then it is observable in some amount... If we can observe it in some amount then it is measurable.

The “Thought Experiment” Imagine that you are a scientist capable of making clones of entire companies and that you have a cloned pair of your company Change one of the companies so that one has the stated “intangible” and the other does not Ask what would you actually observe that would be different between the two companies

Examples of Clarification The “Intangible” Possible Meanings After Clarification “Employee Empowerment” Less management overhead Certain decisions are more accurate and faster “Information Availability” Time and cost of searching is reduced Certain costly errors are less frequent “Customer Relationship” Increased repeat business Tools like “The Clarification Chain” are used to identify unit-of-measure variables hidden beneath the “intangible” label I offer a challenge that given any intangible, I can clarify it and identify a method of measurement within 15 minutes (I’ve never lost)