Presentation on theme: "Uncertainty and Sensitivity Analysis H. Scott Matthews 12-706 / 19-702."— Presentation transcript:
Uncertainty and Sensitivity Analysis H. Scott Matthews 12-706 / 19-702
Admin zHW 2 Due Wednesday yPlease don’t send emails/come by an hour before class and expect answers - from me or TA’s yPlease also think about sensitivity analysis for pipeline case zFriday recitation: microeconomics review zHow to distribute @RISK CDs? yAll CEE teaching cluster machines being pre-installed.
Problem of Unknown Numbers If we need a piece of data, we can: Look it up in a reference source Collect number through survey/investigation Guess it ourselves Get experts to help you guess it Often only ‘ballpark’, ‘back of the envelope’ or ‘order of magnitude needed Situations when actual number is unavailable or where rough estimates are good enough E.g. 100s, 1000s, … (10 2, 10 3, etc.) Source: Mosteller handout
Definitions Uncertainty: The lack of certainty, A state of having limited knowledge where it is impossible to exactly describe existing state or future outcome(s) (Hubbard 07) zMeasuring uncertainty: yStated by giving a range yRepeated experiments (u = stdev, stderr)
The steps or path zSensitivity analysis - computing the effect of changes in inputs on model predictions zUncertainty propagation - calculating the uncertainty in the model outputs induced by input uncertainty zUncertainty analysis - comparing importance of input uncertainties in terms of their relative contributions to uncertainty in the outputs z(Henrion and Morgan 1990)
Uncertainty zInvestment planning and benefit/cost analysis is fraught with uncertainties yforecasts of future are highly uncertain yapplications often made to preliminary designs ydata is often unavailable zStatistics has confidence intervals – economists need them, too.
Definition: “Base Case” zGenerally uses single values and our ‘best guesses’ zSensitivity Analysis acknowledges uncertainty exists zIncorporate variables instead of constant assumptions zIf our ‘Net Benefits’ remain positive over a wide range of reasonable assumptions, then robust results
How many variables? zChoosing ‘variables’ instead of ‘constants’ for all parameters is likely to make model unsolvable zPartial sens. Analysis - change only 1 Equivalent of y/ x yDo for the most ‘critical’ assumptions yCan use this to find ‘break-evens’
Best and Worst-Case Analysis zAnalogous to “upper and lower bounds” used in estimation problems zDoes any combination of inputs reverse the sign of our answer? yIf so, are those inputs reasonable? yE.g. using very conservative ests. yMight want NB > 0, but know when NB < 0 ySimilar to ‘breakeven analysis’
Question 2.4 from Boardman z3 projects being considered R, F, W yRecreational, forest preserve, wilderness yWhich should be selected?
Question 2.4 Project “R with Road” has highest NB
Question 2.4 w/ uncertainty zWhat if we are told that Benefits/Costs of each project are uncertain by, for example plus or minus 10%? ye.g. instead of Project R having benefits of $10 million, could be as low as $9 million or as high as $11 million yRepeat for all project combinations zNow which project is ‘best’?
Question 2.4 w/ uncertainty Best Case: R w/Road (same) Worst Case: Road Only General Case: Could be several But difficult to determine that from this chart - can we do better?
Using error/uncertainty bars zShow ‘original’ point as well as range of uncertainty associated with point yRange could be fixed number, percentage, standard deviation, other zExcel tutorial available at: yhttp://phoenix.phys.clemson.edu/tutorials/excel/advgraph.html ySee today’s spreadsheet on home page yGraphs original points, and min/max deviations from that as error bars…
Error bar result Easier to see results - imagine moving ruler up and down the axis There is a range from about $3 to $4 million with 3+ options possible Net Benefits ($M)
An Easy Visual Clue - Slider Bars zUse built-in excel functions to visually see effects of incremental changes in variables y“Scroll bars” (form toolbar) yTutorial at: xhttp://aitt.acadiau.ca/tutorials/Excel97/Sliderbars.htm yLet’s look at our old TV estimation problem xTool gives easy, visual aid for how sensitive parameters are (but not very quantitative)
Case: Photo-sensors for lighting zSave electricity by installing sensors in areas where natural light exists ySensors ‘see’ light, only turn light fixtures on when needed zMAIN Posner 2nd floor hallway uses 106 15- watt fluorescent bulbs for 53 fixtures zHow could we make a model to determine whether this makes sense? yAssume only one year time frame
Photo-sensors for lighting zAssume we only care about ‘one year project’ zCosts = Labor cost, installation cost, electricity costs, etc. yAssume each bulb costs $6 zBenefits = ? How should we set up model? yAssume equal, set up as ‘show minimum cost’ option zCase 1 ‘Status quo’: assume lights used as is yOn all the time, bulbs last 10,000 hours ~ burn out once per year) zCase 2 ‘PS’: pay to install sensors now, bulbs off between 1/3 and 1/2 of time
Lighting Case Study - Status Quo zCosts(SQ) - lights on all the time yLabor cost: cost of replacing used bulbs x“How many CMU facilities employees does it take to change a light bulb?” - and how long does it take? xAssume labor cost = $35/hr, 15 mins/bulb x26.5 hours to change all bulbs each year, for a total labor cost of $927.50! Also, bulb cost $636/yr yElectricity: 106*15W ~ 14,000 kWh/yr (on 24-7) xCost varies from 2.5 - 7.5 cents/kWh ~ $350-$1045 xCost Replacing bulbs is same ‘order of magnitude’ as the electricity! (Total range [$1,911 - $2,608])
Lighting Case Study - PS sensors zCosts(PS) - probably ‘off’ 1/3 - 1/2 of time yLabor cost: cost of installing sensors = ‘unknown’ yLabor cost: cost of installing new bulbs xCould assume 1/2 - 2/3 of bulbs changed per year instead of ‘all of them’ [Total $464 - $700] xBulbs cost [$318-$424] yElectricity: 106*15W ~ 7,000 kWh/yr @ 1/2 x9,333 kWh if off 1/3 of the time xCost varies [ 2.5 - 7.5 cents/kWh] ~ $175-$700 yTotal cost (w/o sensors) ~ [$955 - $1,740] yHow much should we be WTP for sensors if time horizon is only one year?
PS sensors analysis zWTP [$170 - $1,700] per year (NB>0) yWe basically ‘solved for’ benefit yBut our main sensitive value was elec. Cost, so range is probably [$919 - $1,129] per yr. yNo overlap in ranges - PS always better zShould consider effects over several years yCould do a better bulb replacement model zUse more ranges - Bulb cost, labor, time yCheck sensitivity of model answer to changes yFind partial sensitivity results for each yLook at spreadsheet model yThis is fairly complicated - easier way?
Sens. Analysis for Photo Sensors zSeveral built-in options from the plugins @RISK-TopRank [Win only - mac with VPC, Parallels, etc] to check sensitivity. yOne-way (one variable at a time) yTornado and spider diagrams (all at a time) yTwo-way (two at a time)
One-Way Sensitivity Analysis zUse plugin. zMakes graph that varies a single variable from a low- to high-end range and shows output (eg NPV) as a function of variable zThe resulting graph shows how “sensitive” your answer is to changes in the one variable yShows simple trend related to one variable
Tornado Diagrams zShows results of many one-way plots on single chart (one changes while all others held constant) zLength of bars tells you how sensitive output is to each variable yBigger bar = more sensitive ySoftware typically “sorts” most to least yLooks like a tornado zSpider diagrams are similar
Plug-in notes zSee Clemen pp.193- for TopRank tutorial
Two-way SA zShows a 2-dimensional plot of what happens when we change 2 variables at once yGraph generated is a “frontier” of feasible options between two variables
EXCEL’s TABLE function zOne- and two-input data tables zSort of a built-in tool for sensitivity analysis (without fancy graphs/etc). ySee PDF posted for examples and instructions yTopRank type analysis typically easier, however EXCEL TABLE requires no plug in (or expensive software)