1. Building the Knowledge Base How to Successfully Evaluate A TCSP Project presented by William M. Lyons U.S. Department of Transportation/Volpe Center and John H. Suhrbier Christopher D. Porter Cambridge Systematics, Inc. September 14/15, 2000

2. Agenda n Role of evaluation in the TCSP program William Lyons William Lyons n Responsibilities for evaluation William Lyons William Lyons n Guidelines for project evaluations John Suhrbier and Chris Porter John Suhrbier and Chris Porter n Examples of approaches to project evaluation Washington, DC Washington, DC Hartford, CT Hartford, CT n Questions and discussions

3. Why Evaluation? n Legislative intent of Section 1604 of TEA-21 (TCSP) Emphasis on learning Emphasis on learning n Improve knowledge of the costs and benefits of innovative approaches to integrating: Transportation investments and strategies Transportation investments and strategies Community preservation, Community preservation, Land development patterns, and Land development patterns, and Environmental quality Environmental quality n Investigate relationships between transportation and community and system preservation

4. Project and Program Evaluation Crucial and Complementary n Project evaluation Results of individual projects Results of individual projects Responsibility of grantee, with technical assistance Responsibility of grantee, with technical assistance Direct source of learning Direct source of learning Challenge  objectivity, quality, consistency, credibility Challenge  objectivity, quality, consistency, credibility -Results useful locally as well as nationally n Program evaluation Project evaluations, taken together Project evaluations, taken together Focus on program goals, analysis, synthesis Focus on program goals, analysis, synthesis Assessment of success of overall program Assessment of success of overall program

5. Desired Results What We Are Trying to Do n Build national knowledge base, leverage results n Provide insights into what is and what is not working n Evaluate innovations and results through systematic analysis Document learning Document learning -Problems as well as successes -Falling short of expected goals is acceptable n Foster innovation, encourage experimentation n Demonstrate success of program

6. Context Major Considerations Evaluation plan  how comprehensive? Evaluation plan  how comprehensive? Timing  early planning and action are critical Timing  early planning and action are critical n Focus on key innovations n Process versus outcome results n Description versus documentation and credible analysis n Importance of flexibility n Effort should be interesting and worthwhile for all involved Session focus  practical strategies to improve evaluation

7. Roles and Responsibilities How to Improve Effectiveness? Grantees  evaluation of individual TCSP projects Grantees  evaluation of individual TCSP projects FHWA HQ  evaluation of overall TCSP pilot program FHWA HQ  evaluation of overall TCSP pilot program Availability of technical assistance Availability of technical assistance Resource centers Resource centers U.S. DOT/Volpe Center and Cambridge Systematics, Inc. U.S. DOT/Volpe Center and Cambridge Systematics, Inc. n Division Offices Encourage and support evaluation Encourage and support evaluation Point of contact for technical assistance Point of contact for technical assistance Need for flexibility Need for flexibility What’s realistic? What’s realistic?

8. TCSP Evaluation Most Simply Is n Identifying the set of messages to be used in n Telling the TCSP story

9. What is Evaluation? n TCSP grants are resulting in things (planning and projects) being done differently n Most basically, evaluation addresses What are the benefits of these changes? What are the benefits of these changes? What actions are important in achieving these benefits? What actions are important in achieving these benefits? What do these changes cost in terms of time and dollars? What do these changes cost in terms of time and dollars? Are longer term cost savings being achieved? Are longer term cost savings being achieved? Are the benefits worth the extra effort? Are the benefits worth the extra effort? How can these successes be repeated, or “institutionalized”, within other projects? How can these successes be repeated, or “institutionalized”, within other projects?

10. Target Audiences For Project Evaluation Results n TCSP project staff Help refine project work plan Help refine project work plan -What is working? -What is not working? n Local elected and appointed officials What is the project accomplishing that is different? What is the project accomplishing that is different? n Other TCSP grantees Learning from others Learning from others n Input to overall TCSP program evaluation

11. Evaluation Assessment of Experience to Date n Most people (grantees, FHWA, and other program participants) Recognize that TCSP is resulting in lots of interesting initiatives Recognize that TCSP is resulting in lots of interesting initiatives Acknowledge importance of assessing benefits of these initiatives Acknowledge importance of assessing benefits of these initiatives Want to do a good job Want to do a good job They have only a limited understanding of evaluation approaches and methodologies They have only a limited understanding of evaluation approaches and methodologies Not really sure how to get started Not really sure how to get started n Most also admit Anecdotal evidence is helpful but not sufficient Anecdotal evidence is helpful but not sufficient Evaluation is more than public relations or marketing Evaluation is more than public relations or marketing

12. Available Evaluation Resources n Guidance for the preparation of TCSP evaluation plans n Selected references evaluating the relationships between travel and land use, including contact information Literative reviews (9) Literative reviews (9) Individual studies (16) Individual studies (16) n Evaluation examples Upgrade of a rural highway corridor Upgrade of a rural highway corridor Transit supportive land use development Transit supportive land use development Implementation of urban design, land use, and TDM, strategies for an urban office complex Implementation of urban design, land use, and TDM, strategies for an urban office complex n TCSP web site n Technical assistance

13. Components of an Evaluation Process DefinitionProjectInitiation On-Going Completion Assess Implementation Experience Compare “After” to “Before” Conditions DiscretionaryGrantsPreliminary Evaluation Plan Reporting Refined EarmarkedGrants Reporting Assess Implementation Experience Compare “After” to “Before” Conditions

14. Steps in Developing an Evaluation Plan Identify Data Sources and Evaluation Methods Identify Performance Measures Define Project Goals and Objectives

15. How Much Effort is Required For Evaluation? n For a planning grant Not much Not much More a willingness to step back and ask the right questions More a willingness to step back and ask the right questions n For an implementation grant More effort required More effort required Longer term monitoring possible with Longer term monitoring possible with -Before/after data -Statistical control

16. TCSP Web Site Home Page

17. Web Page Project Descriptions