Uncertainty surrounding the Cone of Uncertainty Todd Little “It’s tough to make predictions, especially about the future.” – Yogi Berra

IEEE Software, May/June 2006

Managing the Coming Storm Inside the Tornado When will we get the requirements? All in good time, my little pretty, all in good time But I guess it doesn't matter anyway Doesn't anybody believe me? You're a very bad man! Just give me your estimates by this afternoon No, we need something today! I already promised the customer it will be out in 6 months No, we need it sooner. Not so fast! Not so fast!... I'll have to give the matter a little thought. Go away and come back tomorrow Ok then, it will take 2 years. Team Unity Project Kickoff

We’re not in Kansas Anymore My! People come and go so quickly here! I may not come out alive, but I'm goin' in there! The Great and Powerful Oz has got matters well in hand. "Hee hee hee ha ha! Going so soon? I wouldn't hear of it! Why, my little party's just beginning! Developer Hero Reorg Testing

Hurricane Rita

About Landmark Commercial Supplier of Oil and Gas Exploration and Production Software Users are Geophysicists, Geologists, Engineers Subsidiary of Halliburton Energy Services Integrated suite of ~60 Products ~50 Million lines of code Some products 20 years old

Landmark Product Suite Common Model Representation Well data Production data Seismic data Velocity data Reservoir / Fluid data Structural / Stratigraphic data Common Model Representation

Data in the Portfolio 3 years of data ( ) 570 projects –106 valid (Shipped commercial product) –Remainder: Currently active, placeholder projects, internal projects, non-commercial releases, deferred projects, etc. Relatively Unbiased. –Each week the Program Manager recorded the state of the project and the current release estimate. –No “improvement goal” bias

Data from LGC Developing Products in Twice the Time

Data from Tom DeMarco It’s déjà vu all over again

Cumulative Distribution Curve for Actual/Estimate (DeMarco)

CDF Distribution Curve (LGC)

Probability Distribution Curve

It’s tough to make predictions, especially about the future.” – Yogi Berra That idea is so damned nonsensical and impossible that I'm willing to stand on the bridge of a battleship while that nitwit tries to hit it from the air. Newton Baker, U.S. secretary of war in 1921, reacting to the claim of Billy Mitchell (later Brigadier General Mitchell) that airplanes could sink battleships by dropping bombs on them. Heavier-than-air flying machines are impossible. Lord Kelvin, British mathematician, physicist, and president of the British Royal Society, spoken in 1895.

How does Estimation Accuracy Improve Over Time? At the “end” of each phase, compare the most current estimate with the resulting end date. –Envisioning –Planning –Developing

Estimation Accuracy (Boehm) 0.5 2

So what does LGC data look like?

Landmark Cone of Uncertainty

Cumulative Distribution (CDF) Curve

But is Uncertainty Really Reduced? “Take away an ordinary person’s illusions and you take away happiness at the same time.” Henrik Ibsen--Villanden

Remaining Uncertainty

The Pipe of Uncertainty 0.5 2

Does Landmark Suck at Estimation? A severe depression like that of is outside the range of probability. Harvard Economic Society, Weekly Letter, November 16, I think there is a world market for about five computers. Thomas J. Watson, chairman of IBM, They couldn't hit an elephant at this dist… General John B. Sedgwick, Union Army Civil War officer's last words, uttered during the Battle of Spotsylvania, 1864

Estimation Quality Factor (EQF) Elapsed Time Value to be Estimated Actual Value Initial Estimate Actual End Date Link to article by Tim Lister Blue Area Red Area EQF =

EQF from Lister/DeMarco An EQF of 5 is pretty good (i.e. averaging about 1/5 or 20 percent off.) The median for schedule estimating is about a 4, with the highest sustained scores at 8 to 9. Lister and DeMarco have never known anybody to sustain a 10 (just 10 percent off). Typical disaster project is 1.8

EQF Distribution Curve (LGC) EQF for duration has a theoretical minimum of 2.0

We slip one day at a time, EQF=2 Elapsed Time Value to be Estimated Actual Value Initial Estimate Actual End Date Blue Area Red Area EQF =

(EQF-2) Distribution Curve (LGC data)

LGC Estimation Quality LGC’s EQF measurement is pretty good. Our p(50) is 4.8, versus an industry average around 4 and a best sustained in the ~8-10. Our p(10) is 2.8, which is not bad.

Don’t know that we don’t know Knowable Unknowable Uncertainty Know that we know Know that we don’t know Don’t know that we know

Don’t know that we don’t know Knowable Unknowable Uncertainty Know that we know Know that we don’t know Don’t know that we know Uncertainty Management PlanningRisk Management p10p50 p90

The Cone of Uncertainty

Successful Projects?