Presentation is loading. Please wait.

Presentation is loading. Please wait.

How To Measure Anything:

Similar presentations

Presentation on theme: "How To Measure Anything:"— Presentation transcript:

1 How To Measure Anything:
Finding the Value of ‘Intangibles’ in Business Copyright HDR 2007

2 How to Measure Anything
It started 12 years ago… I conducted over 60 major risk/return analysis projects so far that included a variety of “impossible” measurements I found such a high need for measuring difficult things that I decided I had to write a book The book was released in July 2007 with the publisher John Wiley & Sons This is a “sneak preview” of many of the methods in the book Copyright HDR 2007

3 How To Measure Anything
“I love this book. Douglas Hubbard helps us create a path to know the answer to almost any question, in business, in science or in life.” Peter Tippett, Ph.D., M.D. Chief Technology Officer at CyberTrust and inventor of the first antivirus software “Doug Hubbard has provided an easy-to-read, demystifying explanation of how managers can inform themselves to make less risky, more profitable business decisions.” Peter Schay, EVP and COO of The Advisory Council “As a reader you soon realize that actually everything can be measured while learning how to measure only what matters. This book cuts through conventional clichés and business rhetoric and it offers practical steps to using measurements as a tool for better decision making.” Ray Gilbert, EVP Lucent “This book is remarkable in it's range of measurement applications and it's clarity of style. A must read for every professional who has ever exclaimed ‘Sure, that concept is important but can we measure it?’” Dr. Jack Stenner, CEO and co-founder of MetaMetrics, Inc. Copyright HDR 2007

4 CFO Measurement Problem
The Risk Paradox: The largest and riskiest decisions often get the least quantitative risk analysis The Measurement Inversion: According to an economic valuation of the benefits of a measurement, most measurement priorities are the opposite of the optimal solution. Better alternatives than: Traditional business cases that don’t quantify uncertainty, risks, and intangibles “Scores” that quantify nothing Copyright HDR 2007

5 Copyright HDR 2007
Style vs. Substance If you are adding and multiplying subjective “scores” on a scale of 1-5 for things like risk, alignment, etc. chances are your method doesn’t improve on your intuition Also don’t be fooled by the terms “structured” or “formal” (Astrology is both structured and formal, it just doesn’t work) The Following Charts Mean Nothing: Innovation Strategy 8 6 Alignment 4 Efficiency 2 Relationship Process Customer Value Effectiveness Copyright HDR 2007

6 Assessing Assessment Methods
“Proven” should mean more than some previous users feel good about it (the “testimonial proof”) Only empirical evidence that forecasts and decisions are actually improved can separate real benefits from a “placebo effect” Effective methods for evaluating IT investments should have a lot in common with well-known methods in other fields (actuarial science, portfolio optimization, etc.) Copyright HDR 2007

7 My Three Measurement “Heroes”
Eratosthenes – measured the Earth’s circumference to within 1% accuracy Enrico Fermi – the physicist who used “Fermi Questions” to break down any uncertain quantity (and was the first to estimate the yield of the first atom bomb) Emily Rosa – the 11 yr old who was published in JAMA (youngest author ever) for her experiment that debunked “therapeutic touch” Copyright HDR 2007

8 Copyright HDR 2007
Three Illusions of Intangibles (The “” approach) The perceived impossibility of measurement is an illusion caused by not understanding: the Concept of measurement the Object of measurement the Methods of measurement See my “Everything is Measurable” article in CIO Magazine (go to “articles” link on Copyright HDR 2007

9 Copyright HDR 2007
An Approach Model what you know now Compute the value of additional information Where economically justified, conduct observations that reduce uncertainty Update the model and optimize the decision Copyright HDR 2007

10 Uncertainty, Risk & Measurement
Measuring Uncertainty, Risk and the Value of Information are closely related concepts, important measurements themselves, and precursors to most other measurements The “Measurement Theory” definition of measurement: “A measurement is an observation that results in information (reduction of uncertainty) about a quantity.” We model uncertainty statistically – with Monte Carlo simulations Copyright HDR 2007

11 Copyright HDR 2007
A Few of My Examples Risk of IT The value of better information The value of better security Forecasting the demand for space tourism Forecasting fuel for Marines in the battlefield Measuring the effectiveness of combat training to reduce roadside bomb/IED casualties The Risk of obsolescence The value of a human life The value of saving an endangered species The value of public health The value of IQ points lost by children exposed to Methyl-Mercury Copyright HDR 2007

12 Copyright HDR 2007
Calibrated Estimates Decades of studies show that most managers are statistically “overconfident” when assessing their own uncertainty Studies also show that measuring your own uncertainty about a quantity is a general skill that can be taught with a measurable improvement Training can “calibrate” people so that of all the times they say they are 90% confident, they will be right 90% of the time Copyright HDR 2007

13 1997 Calibration Experiment
16 IT Industry Analysts and 16 CIO’s , the analysts were calibrated In January 1997, they were asked To Predict 20 IT Industry events Example: Steve Jobs will be CEO of Apple again, by Aug 8, True or False? 100% 17 68 152 65 45 21 90% “Ideal” Confidence 80% Statistical Error 70% Giga Clients Percent Correct 25 75 71 65 58 21 60% Giga Analysts 50% 99 # of Responses 40% 30% 50% 60% 70% 80% 90% 100% Assessed Chance Of Being Correct Source: Hubbard Decision Research Copyright HDR 2007

14 The Value of Information
I use macro in Excel for this formula. In the book, I made a simple table that can estimate it with some simple multiplication What it means: Information reduces uncertainty Reduced uncertainty improves decisions Improved decisions have observable consequences with measurable value 0.5 100 100 80 80 0.4 60 60 0.3 40 40 20 0.2 20 10 8 6 10 4 0.1 10 8 1 10 0.8 2 0.6 0.4 0.2 1 6 0.1 0.05 4 -0.1 2 1 -0.2 0.8 1 0.6 0.4 -0.3 0.2 0.1 -0.4 0.08 0.06 .01 0.04 -0.5 Copyright HDR 2007

15 Next Step: Observations
Now that we know what to measure, we can think of observations that would reduce uncertainty The value of the information limits what methods we should use, but we have a variety of methods available Take the “Nike Method”: Just Do It – don’t let imagined difficulties get in the way of starting observations Copyright HDR 2007

16 Some Useful Suggestions
It has been done before You have more data than you think You need less data than you think It is more economical than you think Copyright HDR 2007

17 The “Math-less” Statistics Table
Measurement is based on observation and most observations are just samples Reducing your uncertainty with random samples is not made intuitive in most statistics texts This table makes computing a 90% confidence interval easy Copyright HDR 2007

18 Measuring to the Threshold
Measurements have value usually because there is some point where the quantity makes a difference Its often much harder to ask “How much is X” than “Is X enough” Number Sampled 2 4 6 8 10 12 14 16 18 20 50% 40% 30% 20% Chance the Median is Below the Threshold 10% 5% 2% 1% 0.5% 0.2% 0.1% 1 2 3 4 5 6 7 8 9 10 Samples Below Threshold Copyright HDR 2007

19 Copyright HDR 2007
Statistics Goes to War Several clever sampling methods exist that can measure more with less data than you might think Examples: estimating the population of fish in the ocean, estimating the number of tanks created by the Germans in WWII, extremely small samples, etc. Copyright HDR 2007

20 Reducing Inconsistency
The “Lens Model” is another method used to improve on expert intuition The chart shows the reduction in error from this method on intuitive estimates In every case, this method equaled or bettered the judgment of experts My Studies IT Portfolio Priorities Battlefield Fuel Forecasts Student ratings of teaching effectiveness Cancer patient life-expectancy Psychology course grades Graduate students grades Changes in stock prices IQ scores using Rorschach tests Mental illness using personality tests Business failures using financial ratios Life-insurance salesrep performance 0% 10% 20% 30% 40% Source: Hubbard Decision Research Reduction in Errors Copyright HDR 2007

21 Copyright HDR 2007
The Simplest Method Bayesian methods in statistics use new information to update prior knowledge Bayesian methods can be even more elaborate that other statistical methods BUT… It turns out that calibrated people are already mostly “instinctively Bayesian” Copyright HDR 2007

22 Comparison of Methods Copyright HDR 2007
Calibrated Estimator Bayesian Typical Un-calibrated Non- Statistics Ignores Prior Knowledge; Emphasizes new data Ignores New data; Emphasizes Prior Knowledge Stubborn Gullible Under-confident (Stated uncertainty is higher than rational) Overconfident (Stated uncertainty is lower than rational) Cautious Vacillating, Indecisive Skeptic Copyright HDR 2007

23 Risk/ROI w/ “Monte Carlo”
A Monte Carlo simulation generates thousands of random scenarios using the defined probabilities and ranges The result is a range ROI not a point ROI Administrative Cost Reduction 5% 10% 15% Customer Retention Increase 10% 20% 30% Total Project Cost $2 million $4 million $6 million ROI -50% 0% 50% 100% Copyright HDR 2007

24 Quantifying Risk Aversion
The simplest element of the Nobel Prize-winning method “Modern Portfolio Theory” is documenting how much risk an investor accepts for a given return Acceptable Risk/Return Boundary Probability of a negative ROI Average ROI 0% 10% 20% 30% 40% 50% 100% 150% 200% Investment Region Investment Copyright HDR 2007

25 Approach Summary 1 2 Define Decision Model Calibrate Estimators 3 Populate Model with Calibrated Estimates & Measurements 5 Measure according to VIA results and update model 4 Conduct Value of Information Analysis (VIA) Average ROI Risk 0% 10% 20% 30% 40% 50% 100% 150% 200% 7 6 Analyze Remaining Risk Optimize Decision Copyright HDR 2007

26 Copyright HDR 2007
Final Tips Learn how to think about uncertainty, risk and information value in a quantitative way Assume its been measured before You have more data than you think and you need less data than you think Methods that reduce your uncertainty are more economical than many managers assume Don’t let “exception anxiety” cause you to avoid any observations at all Just do it Copyright HDR 2007

27 Copyright HDR 2007
Questions? Doug Hubbard Hubbard Decision Research Copyright HDR 2007

Download ppt "How To Measure Anything:"

Similar presentations

Ads by Google