1. Longer-Term Evaluation of APEC Projects:
EWG Evaluation Results & Next Steps
EWG53, Singapore, April 2017
Key messages:
We have done this evaluation
EWG did comparably to APEC overall
How can we improve the response rate of POs and Participants?
Prepared by APEC Project Management Unit
Copyright © 2015 APEC Secretariat

2. Longer-term Evaluation:
Purpose
A knowledge objective: To assess how APEC projects collectively contribute to the achievement of APEC capacity building objectives; and,
A management objective: To assist in improving the results-orientation of APEC projects and decision making processes with the aim of improving the overall quality of APEC’s interventions. Complements our standard monitoring andevaluation processes (MRs and CRs)

3. Longer-term Evaluation:
Methodology
Methodology:
Evaluate projects a year or longer after completion – to see if any changes can be detected as a result of the project
Online survey of project overseers and participants:
Project Overseer Survey (15 standard questions)
Project Participant Survey (13 standard questions)
Compare survey results with previous evaluations

4. 2015: PMU conducted an evaluation on projects from the ANSSR sub-fund
Longer-term Evaluation:
Background
2013: BMC approved framework for longer-term evaluation of APEC projects, including a pilot evaluation of selected PREE and JODI projects from EWG
2014: PMU integrated the framework into APEC’s project processes, forms and templates
2015: PMU conducted an evaluation on projects from the ANSSR sub-fund
BMC 2, 2013: framework for longer-term evaluation of APEC projects approved based on a pilot evaluation of selected projects undertaken by consultants. – 4 projects under PREE or Peer Review of Energy Efficiency; 4 projects under JODI or Joint Energy Data Inventory.
2014: PMU integrated the framework into APEC’s project processes, forms and templates so that evaluation information could be collected (e.g. contact information for project participants)
BMC 1, 2015: implementation of the framework complete so PMU proposed to undertake the first longer-term evaluation by late 2015
Late 2015: PMU conducted a longer-term evaluation of projects from the APEC New Strategy on Structural Reform (ANSSR) sub-fund completed 2013 and This was the first longer-term evaluation to be conducted ‘in-house’
2016: PMU conducted a baseline evaluation of all projects completed in 2014, including 24 EWG projects

5. Longer-term Evaluation:
Key Findings: 2016 Baseline Evaluation
62.5% of EWG projects had POs respond, which is slightly higher than the APEC-wide average of 57%
Raw numbers: 15/24 or versus 54/94.

6. Longer-term Evaluation:
EWG Participant results: 2013 vs 2016
2013
2016
17 participant respondents for 8 EWG projects
77 total respondents from 24 EWG projects
88% felt the project focussed on an important or top priority (top two possible choices in the survey scale).
81% indicated that the project focussed on an important or top priority. This is higher than the APEC-wide average of 76%.
94% indicated that they applied what they had learned in the project almost every week or infrequently but as a regular part of their jobs.
98% of EWG participants applied what they had learned in the project almost every week or infrequently but as a regular part of their jobs. This is slightly higher than the 96% for overall APEC.
Approximately 53% of participants felt that sustainable changes in policy or practice had occurred due to this workshop (PREE was higher than JODI).
53% of the participants indicated that the project contributed to changes in policy or practice within their division or department. This is the same as the APEC average of 53%.
All of the percentages – across the years and from the two different sample sets (EWG only and APEC-wide) are very similar.

7. Longer-term Evaluation:
Key Findings – Examples from EWG POs
EWG POs indicated that the primary achievements of their projects were:
Recommendations in the efforts to deploy more ultra-supercritical power plants which improves plant efficiency and reduces coal consumption.
Development of education programs and training courses to on next generation of coal-based energy facilities such as IGCC and CCS technologies.
Establishment of a Lighting Research and Innovation Centre at KMUTT in Thailand and agreement to develop the Singapore Lighting Technology Centre.
Sponsorship of experts in unconventional natural gas exploration and development. to key counterparts that have significant unconventional gas resource potential but require technical assistance to initiate or accelerate its development.
Clear definitions of nearly/net zero energy building in China, USA and Japan.

8. Longer-term Evaluation:
Key Findings – Examples from EWG Participants
Participants found practical use in the following aspects of EWG projects:
Using the calculations on carbon emission
Contributing to the preparation of the National Energy Security, Presidential Decree “Energy Emergency Countermeasure”
Sharing the energy storage business model with the team that is formulating Energy Storage policy
Applying state of the art carbon capture techniques to developing economy contexts

9. Longer-term Evaluation:
Challenges
Low response rate:
This has been consistently an issue; compares with 13% for APEC overall

10. Longer-term Evaluation:
Next Steps: 2017 Evaluation
Will commence in October 2017
Efforts to improve response rates:
Shorter lag between project completion and evaluation: 2017 evaluation will cover all projects completed in 2016
Changes to the Travel Undertaking to alert participants of their responsibility to respond
Continued effort to improve indicators in project proposals
Continued effort to secure full participant lists