Managing Evaluations for Consistently High Quality American Evaluation Association Annual Conference Molly Hageboeck

USAID M&E Projects Overseas Managed by MSI Columbia Uganda Kenya South Sudan Ethiopia Pakistan Afghanistan

USAID M&E Projects Overseas Managed by MSI Keys to Success Evaluations are projects – they can be managed Identify key intervention points – quality checkpoints Create tools for exerting quality control at the checkpoints Share the tools with clients and evaluation teams -- Field handbook -- New website MSI build for USAID E3 to improve M&E includes evaluation management tools.

MSI Evaluation Management Checkpoints for USAID’s Process The Evaluation Management Process Stage 1Stage 2Stage 3Stage 4Stage 5 Decision to Evaluate to Issuance of SOW Proposal Review to Approval for Data Collection to Begin Support During Data Collection and Analysis Initial Evaluation Results Briefing to Final Report Dissemination of Final Report to Assessment of Evaluation Influence  Decision to evaluate  Evaluation Manager assigned  Evaluation parameters defined (type, timing)  Development partner input (as appropriate)  Evaluation design/plan developed (USAID/ initial version)  Evaluation dissemination/ utilization plan developed by USAID (initial version – include list of what evaluation team needs to provide to USAID)  Design/plan reviewed/approved (USAID/initial version)  SOW drafted  SOW reviewed and approved (Quality Control checkpoint)  Solicitation issued (if external evaluators are to be involved)  Proposals reviewed/ team selected  Team inception report on performance monitoring findings (if required by SOW) (Quality Control checkpoint)  Team planning meeting (TPM)  Initial meetings with development partners  Detailed evaluation design/plan developed/refined by team  Evaluation design/plan (or modifications) approved (Quality Control checkpoint)  Register the evaluation with USAID/ Washington  Weekly status review with team against field work plan and schedule  Troubleshooting as needed to assist evaluation team in the field  Initial briefing (on completeness) of evaluation findings, conclusions and recommendations (Quality Control Checkpoint)  Approval to proceed to drafting report (if approval is required by SOW)  Submission of Draft Report  Oral briefing on draft report (if required by SOW)  Review of draft report – feedback to team (Quality Control Checkpoint)  Evaluation dissemination/ utilization plan updated/expanded by USAID (final version) Submission of Final Report  Review/acceptance of final report and other deliverables  Dissemination of evaluation report and executive summary (per dissemination/ utilization plan)  Formal evaluation review meeting  Evaluation review minutes disseminated  Follow-up on implementation status of accepted recommendations (per dissemination/ utilization plan)  Follow-up on impact of evaluation (per utilization plan)

Quality Checkpoint 1 Evaluation Statement of Work (SOW) Common Problems Management purpose is not clear/transparent Evaluation Questions – to many, not matched to purpose, not feasible There isn’t always an opportunity to comment on or negotiate the SOW Solution: Help your Clients Improve the SOWS they Prepare MSI Checklist for Developing/Reviewing Evaluation SOWS Built it in about 2000 Gave it to USAID in 2010

Quality Checkpoint 2 Written Review of Existing Information Before Final Design Common Problems Late receipt of project reports/performance data Team reviews often cursory – important data not extracted & shared Solutions: Ask for reports when the SOW is issued. Develop/require a structured desk review product within a short time frame MSI Desk Review Template – First Deliverable from Teams – Before Final Design

Quality Checkpoint 3 Final Evaluation Design/Plan Prior to Field Work Common Problems The field team did not prepare the proposal stage design – and may not follow it Teams too often start the field work without a final design, data collection and analysis (and sampling plan and all necessary instruments Solution: Detailed evaluation design and formal review/approval on a question by question basis from the actual team including all instruments before they get the keys to the jeep. Provide teams with a structured format to get started

MSI “Getting to Answers” Matrix Built it in about 2005 Gave it to USAID in 2010

Quality Checkpoint 4 Post-Field Work and Analysis Pre-Draft Briefing Common Problems Teams start writing before they work out a clear flow of findings, conclusions and recommendations grounded in their evaluation evidence. Many reports not well supported by evidence Many mix up findings, conclusions and recommendations – and confuse readers. Solution: Required oral briefing in bullets to ensure all questions have been addressed and F-C-R flow Is logical Block remaining LOE until this step is passed – as the team may need to get more data before it writes.

Quality Checkpoint 5 Structured Quality Focused Review of Draft Report Common Problems Clients tend to review draft evaluation reports on substantive reports often skipping over structural and professional quality aspects. Quality fine points may not get attention until the final stage – when all LOE has been spent Or they remain missed until a meta-evaluation finds the flaws Solution: Evaluation quality review checklist – shared with teams the day they start and all members of draft report review teams. Checklist based feedback to team – and repeat use of checklist with final report to verify that improvements have been made MSI Checklist for Reviewing Evaluation Reports Built it in about 2000 Gave it to USAID in 2010

Current “News” on MSI’s Evaluation Management System Update of MSI Handbook for Field Teams is underway Recent meta-evaluation for USAID of evaluations found problems that greater internal use of an evaluation management system and associated tools would have caught -- and a recommendation on strengthen internal evaluation management practices in USAID has been provided.