1. NCHRP 20-65 (47) - MTAP Survey Tool Used to Assess FTA Contractor Performance of State DOT Triennial and Other FTA Reviews - An Update Biennial FTA State Programs Meeting / State Public Transit Partnerships Conference August 7, 2103 KFH Group, Inc. Beth Hamby Sue Knapp

2. Background To assist FTA with improving State Review processes, MTAP collects input from State DOTs on their experience following each FTA review via an online survey tool. Survey was created in Vovici (used by AASHTO for other surveys). Value of survey has been somewhat limited by: ▫open-ended answer formats ▫challenges in using Vovici

3. Research Objective: Update MTAP’s online survey to: ▫improve efficiency / ease of completion ▫increase the value added nature of the process (improve results for ease of use, benefits) ▫incorporate technology improvements ▫provide for more detailed responses to 2-3 specific questions of interest to MTAP, SCOPT, FTA - not just a "comments" field

4. Project Tasks Task 1 – Assess Survey Tool Improvement Needs Task 2 – Revise Survey Content & Structure Task 3 – Assess Available Survey Technology Options Task 5 – Set Up & Test Revised Survey Using Selected Technology Task 6 – Prepare Final Report

5. Project Status Task 1 – Assess Survey Tool Improvement Needs – completed ▫Reviewed current survey tool - http://vovici.com/wsb.dll/s/54c9g50982 http://vovici.com/wsb.dll/s/54c9g50982 ▫Reviewed reports generated from current tool ▫Gathered feedback/suggestions from  States/MTAP Committee  FTA Office of Oversight and Program Guidance  MTAP Coordinator

6. Project Status Task 2 – Revise Survey Content & Structure – nearing completion ▫Modified existing questions – standardized responses where possible to make analysis easier ▫Added new questions ▫Prepared revised survey for review by MTAP/ Steering Committee; sent late July

7. Draft Revised Survey Numerous new questions seeking specific feedback ▫Basic experience of reviewer ▫Consultant performance on specific compliance topics ▫For each scaled rating (Excellent/ Good/ Fair/ Poor), asks why (open-ended) ▫Extent of Regional participation ▫Effectiveness of entrance/exit meetings ▫SMR workshop effectiveness ▫Interest in follow-up discussion with MTAP or FTA Standardized responses; limited open-ended questions to explanation and “other” Reworded some questions to align with responses

8. Draft Revised Survey 1.Contact and Background Info ▫Added: Basic experience of respondent 2.Consultant Performance ▫Added: Fields for explanation 3.General Experience with Review ▫Added: Regional office participation ▫Added: Entrance/Exit meetings ▫Added: Overall organization and clarity 4.SMR-Specific Feedback ▫Added: Respondent’s participation ▫Workshop/workbook effectiveness ▫Additional training/tech assistance needs 5.Additional Feedback

9. Project Status Task 3 – Assess Available Survey Technology Options – under way ▫Criteria include:  User-friendliness to create & administer survey  User-friendliness to complete & submit surveys  Functionality/customization options  Reporting capabilities  Cost

10. Survey Approaches General options for getting feedback from states: ▫Online survey tools – basic to advanced  Focus of our assessment ▫Pen & paper  Too labor intensive ▫Telephone interviews  May be appropriate for follow-up

11. Basic Online Survey Tools Examples include: ▫SurveyMonkey Basic / Select / Gold ▫SurveyMoz Free / Premium ▫SurveyGizmo Basic ▫QuestionPro Professional

12. Basic Online Survey Tools Advantages: ▫Easy to learn and use ▫Provides some skip logic functionality ▫Provides some basic cross-tab analysis ▫Can export results to Excel for additional analysis ▫Inexpensive ($0 - $25/month)

13. Basic Online Survey Tools Disadvantages: ▫Limited collaboration capabilities ▫Limited survey customization options ▫Limited reporting / analysis functions – need to export to third party application Basic tools are becoming increasingly sophisticated (e.g., SurveyMoz) Some offer higher levels of customization for a higher cost (e.g., SurveyMonkey Platinum - $65/month)

14. Advanced Online Survey Tools What MTAP is currently using (Vovici) Known as Enterprise Feedback Management (EFM) Systems Examples include: ▫Key Survey ▫Qualtrics ▫QuestionPro Corporate ▫SurveyGizmo Enterprise ▫Vovici

15. Advanced Online Survey Tools Advantages: ▫Allows for greater collaboration in survey administration (behind the scenes)  Multiple user accounts possible  Can share surveys with other survey administrators  Private results through secure dashboard accounts ▫Greater flexibility in branding and customizing ▫Can build sophisticated logic into survey ▫Allows for complex analysis ▫Custom reports can be developed within the tool

16. Advanced Online Survey Tools Disadvantages: ▫Complex interface ▫Requires training / steeper learning curve ▫May require frequent use to maintain skills ▫May require experience in Cascading Style Sheets (CSS) to customize look and feel of survey ▫Expensive ($100 or more/month – few disclose pricing online)  However, no out-of-pocket cost to MTAP for Vovici; AASHTO already pays for it for other uses

17. Next Steps Fine-tune survey questions with input from MTAP/ Committee Finish technology assessment Set up & test revised survey using selected technology Prepare final report

18. For more information or feedback Beth Hamby KFH Group bhamby@kfhgroup.com 206-448-6749