Smart Commute Evaluation Tools, Techniques and Lessons Learned in Monitoring and Evaluating Workplace-based TDM Programs

Outline 1. Background 2. Approach 3. Monitoring Tools 4. Evaluation 5. Results 6. Challenges 7. Lessons 8. Next Steps Ryan Lanyon Team Lead, Smart Commute Metrolinx (GTTA) October 22, 2008

At-a-Glance  Build evaluation into your initial planning  Monitor activities, benefits and customer service  Scrutinize and understand indicators  Learn from results and mistakes  Resources available at  Just do it! Smart Commute works with various stakeholders to relieve traffic congestion, improve air quality and health, and reduce emissions that cause climate change.

Background  Smart Commute  Focus on commuters and workplaces  Partnership with municipalities  Staged development –2001: Pilot TMA - BCRTMA –2004: Pilot regional implementation - Municipalities –2008: On-going implementation – Metrolinx + Municipalities

Background  Metrolinx (Greater Toronto Transportation Authority)  Formed in 2007  Coordinates transportation across GTHA  Regional Transportation Plan  6.13 million residents, 3 million commuters

Background  Two-tiered program delivery  Metrolinx –Central coordinating body –Centralized service operation e.g. Carpool Zone –TMA Toolkit –Funding agency – 50%  Local Smart Commute / TMA –Delivery agent –Municipality, NGO, Chamber of Commerce/Board of Trade –Find other funding – municipalities, grants, service fees

Background  2001: BCRTMA / NTV  2004:  2005:  Mississauga  NE Toronto  2006:  Brampton-Caledon  Halton  2007:  Central York  Durham  2008: Toronto-Central  2009: Airport?

Background  Employer program  Improve and encourage commuting options  Assessment of current conditions - Baseline  Development of a plan  Implementation – ‘Commuter benefits program’ –Carpool Zone and ERH –Walking, cycling, transit, carpool, telework programs –Campaigns, contests and special events  Follow-up assessment – Measured against baseline

Approach - Ideal Goals ObjectivesRegional Transportation Plan Municipal PlansImplementation Strategic Plan Monitoring Evaluation

Approach - Actual GoalsObjectives RTP Municipal Plans Implementation Strategic Plan Monitoring Evaluation Federal Funding Requirements Implementation

Approach  Contribution Agreement  Negotiated between funder and municipalities  Monitoring Framework  Established by Smart Commute stakeholders –TMAs and Smart Commute Association (implementers) –Municipalities and Metrolinx (funders)  Multipurpose –Measure activity levels for funders –Track impacts to justify funding –Collect statistics for marketing purposes

Approach  Inputs / Activities  Number of businesses –Active –Engaged –Supporting  Number of pamphlets distributed  Number of cycling programs  Number of media releases  Number of commuters reached

Approach  Outputs / Benefits / Results  Brand awareness –% of employees aware; change from baseline  Commuter satisfaction –Ratings of Smart Commute program  Reductions –GHG emissions –CAC emissions –Vkt and trips  Commuter cost savings

Monitoring  Levels of monitoring  Regional  TMA  Employer  Commuter / customer  Types of monitoring  Behaviour change  Attitudinal change  Customer service

Monitoring  Regional  Existing sources –Statistics Canada –Transportation Tomorrow Survey (TTS) –Conducted by University of Toronto –Detailed behaviour survey –Conducted every five years –Cordon Count Program –Conducted by municipalities, compiled by University of Toronto –Observed data –Conducted every two to three years

Monitoring  Regional  New sources –Commuter Attitudes Survey –Conducted by Metrolinx –Detailed attitudinal questions –Conducted every two years –Carpool Zone –Conducted by Metrolinx – Monthly activity survey –TMAs and Employers –Compiled data from localized sources

Monitoring  TMA  Quarterly reports – tied to Metrolinx funding  Summary of activities per employers  Communications and outreach activity  Qualitative results  Comparison of activity levels between TMAs

Monitoring  Employer – Baseline Module  Baseline employee survey –Standardized across all TMAs and employers –Census survey  Site assessment –Current conditions –Inventory of advantages and deficiencies  Vehicle and occupancy count –Discreet sites –1,000+ employees

Monitoring  Customer – Commuter  Service-related –Monthly Carpool Zone microsurvey –Emergency Ride Home post-ride survey  Employee surveys  Campaign-related data collection e.g. Car Free Day

Monitoring Mode Statistics Canada (2006) TTS (2006) CAS (2008) Surveys ( ) Drive Alone 65.90%64.7%48%75.91% Transit19.93%19.7%28%10% Cycling0.97%0.7%1%0.89%

Evaluation  Benchmarks and Trends  75 employee baseline surveys  Statistics Canada  Transportation Tomorrow Survey  Cordon Counts  Commuter Attitudes Survey

Evaluation  External  Employer surveys –Biannual survey – program satisfaction rating –Satisfaction with Smart Commute services –Conducted by third party through Metrolinx  Commuter surveys –Employee follow-up surveys – program satisfaction rating –Annual survey – Carpool Zone satisfaction  Internal  Smart Commute Technical Committee

Evaluation  External Review  Consultant evaluation of operational models  Research Partnerships  University of Toronto (Mississauga) –Factors for successful carpool formation –Further research  Ryerson University –TMA Toolkit assessment

Results  Almost 100 members and partners  March 2007 – 50 employers  March 2008 – 75 employers  June 2008 – 89 employers

Results  Transportation indicators  76.4 million vkt  1.3 million trips  Equivalent of 10,000 cars  Increase in carpooling –7% to 12% to 13% Cars Off the Road

Results  Primary Mode Shift Drive alone Public Transit Car/vanpool Walk/Jog Bike

Results  Environmental indicators  17,500 tonnes of greenhouse gas emissions –Enough to fill Rogers Centre almost six times  100,000 kg of criteria air contaminants –Equivalent amount of NOx to smoking 3 billion cigarettes

Results  Communications and Outreach  400+ stories  75+ million media impressions  Brand recognition –Smart Commute ↑ –Carpool Zone ↓ Smart Commute Carpool Week Clean Air Commute Carpool Zone Website Commuter Challenge Awareness (2008)

Challenges  Pressures to implement right away  Show action  Pent-up demand  More interesting than planning or evaluating  All the pieces not always in place  Monitoring not appreciated until results urgently needed  Objectivity  Of course my ‘baby’ was successful!

Challenges  Measurement and Evaluation  Employer buy-in is difficult –Some see value in surveys, monitoring –Others feel employees are oversurveyed –Often considered an expensive waste of employee time  Standardization required, but unenforceable –Moving to centralized service provision –Standardization vs. continuous improvement  Data comparison can be unequal –Apples to apples and apples to oranges

Challenges  Defining success  Setting targets without enough information –Are examples applicable?  Are targets achievable with the resources alloted?  Can these targets be effectively monitored?  What happens if we don’t reach these targets?

Lessons  Measurement and Evaluation  Some items simple, but time-consuming –Do you really need the data?  Project benefits less easy to isolate –Double-counting also possible through employee surveys and Carpool Zone surveys –External factors – environmental concerns, gas prices  One clear conclusion: an impact was made.  Time and consistency open up opportunities –Benchmarking

Lessons  Measure and Evaluation  Access resources at your disposal for help  Universities  Municipalities  Provincial and Federal Governments  Evaluate what you do and how you do it  Give an adjustment period to monitoring  Test! Test! Test!  Allow partners time to understand and see value

Lessons  Provide incentives for monitoring  Tie completion to access  Award and reward success  Measurement and Evaluation  Learn from mistakes – you will make them!  Learn from poor results –Was the program effective? –Was the monitoring effective?

Next Steps  Continue and improve data collection  Address data management and manipulation  Benchmarks and comparisons  Strategic planning  Service evaluation  Targets  Baseline module refinement

Questions? Ryan Lanyon Team Lead, Smart Commute Metrolinx