1. 1 Predicted-versus-Actual Studies: Why/how to do them and Lessons Learned Ken Cervenka Federal Transit Administration TRB Transportation Planning Applications Conference in Houston, Texas May 20, 2009

2. TRB Transportation Planning Applications Conference in Houston2May 2009 Topics Why do them? How to do them? Lessons learned (so far)

3. TRB Transportation Planning Applications Conference in Houston3May 2009 Why do them? Forecasts (should) matter If they don’t matter, what are we doing here? Supports informed decision-making Poor track record for accurate predictions Even aggregate numbers are often way off Capital costs Weekday traffic volumes Weekday transit ridership (boardings)

4. TRB Transportation Planning Applications Conference in Houston4May 2009 Predicted-versus-Actual Traffic 104 Completed Toll Road Projects Data Source: Standard & Poor’s, 2005 (Robert Bain)

5. TRB Transportation Planning Applications Conference in Houston5May 2009 Predicted-versus-Actual Ridership 18 Transit Projects Completed 2003-2007 Average = 74.5% 50th Percentile = 63.8%

6. TRB Transportation Planning Applications Conference in Houston6May 2009 So why do them? Learn from past failures - and successes What went wrong and what went right More than just aggregate checks Avoid hand-waving speculation Identify major drivers for errors (in either direction) Insights for improved prediction tools Better understanding of uncertainties More informed decision-making Better use of limited funds

7. TRB Transportation Planning Applications Conference in Houston7May 2009 Predicted-versus-Actual Some Examples New transportation project Roadway and/or transit Policy change New developments Change over time (“validation”) 2009 forecast of 2005 base year 2000 forecast of 2005 base year (backcast)

8. TRB Transportation Planning Applications Conference in Houston8May 2009 So why else do them? Well… Required for FTA discretionary funding of major transit projects (New Starts program) Annual report to Congress Before-and-after comparisons Predicted-versus-actual (after) comparisons See Session 11 from FTA’s March 2009 Travel Forecasting for New Starts workshop http://www.fta.dot.gov/planning/newstarts/planning_ environment_9547.html http://www.fta.dot.gov/planning/newstarts/planning_ environment_9547.html

9. TRB Transportation Planning Applications Conference in Houston9May 2009 How to do them Start with the big picture comparisons What we thought would happen What actually happened Gain insights by digging into the details Not just traffic volumes and transit passenger boardings Forensic analysis Reasons for big picture misses Confirmation of big picture successes Prepare new post-implementation forecasts New model runs with corrected inputs Special runs to track down sources for other errors

10. TRB Transportation Planning Applications Conference in Houston10May 2009 How to do them Examples Assess the accuracy of forecast inputs District-level demographics Roadway system Transit service levels and fares Auto-related costs, etc. Check major transit rider travel patterns District-to-district flows Trip purpose and socio-economic class Access and egress modes

11. TRB Transportation Planning Applications Conference in Houston11May 2009 How to do them FTA’s Approach Required: preservation of forecasts Project scope, capital cost, service levels, O&M cost, and ridership (plus others as needed) For different project planning milestones Analysis/explanation of changes in forecasts Ability to replicate the forecasts, e.g. for ridership: Input demographics/networks and outputs Scripts and application documentation  DVDs to FTA do not include proprietary software

12. TRB Transportation Planning Applications Conference in Houston12May 2009 How to do them FTA’s Approach Approval of Before-and-After Study Plan Required for grant approval Data collection plan E.g., before and after transit rider surveys Analytical approaches Approval of Before-and-After Study Report Required to close out a grant Analysis/explanation of project impacts Analysis/explanation of prediction errors

13. TRB Transportation Planning Applications Conference in Houston13May 2009 Lessons Learned Roadway and Transit Forecasts Compounded optimism (optimism bias) The “high side” of feasible assumptions Vested interests, ethics, and objectivity Travel model inputs matter Networks and demographics Parking costs, fares, etc. Value of time assumptions Person trip tables (!)

14. TRB Transportation Planning Applications Conference in Houston14May 2009 Lessons Learned Roadway and Transit Forecasts Quality control checks matter Model testing Data adequate for the task Caution on over-specification Plausibility checks of behaviors Big picture insights for key markets

15. TRB Transportation Planning Applications Conference in Houston15May 2009 Lessons Learned Roadway and Transit Forecasts Recognition of uncertainties Local predictions of new behaviors Local calibration not possible Toll roads in areas without toll roads Choice riders in areas with few/no choice riders Park-and-ride service Premium service (e.g., light rail)

16. TRB Transportation Planning Applications Conference in Houston16May 2009 Lessons Learned FTA Perspective Useful information for the region and the profession Biggest missed opportunity of New Starts program A compelling case for clear explanations Early and often coordination Clearly defined responsibilities and budgets Preservation and analysis performed while memories are still fresh Greater attention to opening-year forecasts