1. 1 Utilization-focused Evaluation of a Portfolio of Research, Development, & Demonstration Programs Helen Kim, Larry Pakenas - NYSERDA Rick Ridge – Heschong Mahone Group, Inc. Scott Albert – GDS Associates, Inc. Gretchen Jordan – Sandia National Laboratory American Evaluation Association Conference Portland, Oregon November 2, 2006

2. 2 New York State Energy Research and Development Authority (NYSERDA) Mission Use innovation and technology to solve New Yorks energy challenges in ways that benefit the States economy and environment Vision Serve as a catalyst for change – enabling New Yorkers to realize affordable energy, a growing and vibrant economy, greater energy independence, and a cleaner environment

3. 3 NYSERDA and RD&D Funding Statutory Funding for RD&D –Levy on interstate sales of gas and electricity –Annual funding ~$13 million for RD&D Public Benefits Funds –Annual budget ~175 million total ~$45 million for RD&D NYSERDA Staff –Total 220 –RD&D 40 (29 before Public Benefits Funding)

4. 4 Public Benefits Funded RD&D Program Portfolio

5. 5 Distribution of Public Benefits RD&D Funds Awarded: 1999 to 2005 Total $172 million

6. 6 Project Selection Competitive Solicitations Proposals reviewed by a Technical Evaluation Panel Mix of external and internal reviewers Score and rank projects Criteria for selection –Technical merit (Does it make sense?) –Resources (Does the team have the resources to succeed?) –Benefits (Will the project result in energy and economic benefits for New York State?)

7. 7 Portfolio Level Logic Model Activities Policy research/ Technology Analysis Product Development (Proof of Concept, Test Products, Develop Products) Demonstration Outputs-White papers -Workshops -Papers -Articles -Data from showcases Short-Term Outcomes - Informed/new policies & standards - R&D opportunities identified -Data from early applied research -Recognized additions to knowledge base -Lab prototypes -Future R&D & product options -Commercial scale product developed -Potential demonstrated -Investment/interest growing -Product proven introduced in market -Producers & consumers see value Long-term outcomes Knowledge benefits, economic, energy, and environmental benefits

8. 8

9. 9 Peer-Review Methodology Year 1 –GDS and Staff developed Accomplishment Packets for five projects and sent to reviewers –Assessments were returned to GDS –Obtained feedback from reviewers about process Year 2 –GDS and Staff developed Accomplishments Packets for two Programs and sent to reviewers –Staff made presentations to the peer reviewers –Peer reviewers provided opportunity to discuss the program amongst themselves –Peer reviewers scored assessments and returned to GDS

10. 10 Peer-Review Methodology Recruitment of Peer Reviewers Staff were asked to provide names and contact information of appropriate individuals An email invitation was sent to recruit Peer Reviewer candidates Peer Reviewer Agreement form used to solicited information on credentials and possible conflicts of interest

11. 11 Peer-Review Methodology Accomplishments Packets Consisted of Background information including project goals and justification Summary of resources including funding, co-funding and staff time Summary the accomplishments categorized into the five criteria: knowledge created, disseminated, commercialization progress, energy, economic, and environmental benefits A mini case study

12. 12 First-Year Peer Review (Five Projects) 1.21 st Century HVAC Research Consortium –$200,000; $1.7 million Co-Funding –Goal: Drive basic research on heating, ventilation, air conditioning, and refrigeration technologies –Activity Type: Information for Policy and Researchers 2.Aggregating Distributed Generators –$450,000; $590,000 Co-Funding –Goal: Demonstrate that existing backup generators can be economically aggregated and dispatched –Activity Type: Demonstration 3.Continuous Ambient Particulate Monitor –$500,000; $1.2 million Co-funding –Goal: Improve measurement of ambient PM2.5 mass –Activity Type: Product Development

13. 13 First Year Peer Review (Continued) 3.Truck Stop Electrification –$1.5; $3.3 million Co-funding –Goal: Evaluate potential, construct demonstration sites, establish new businesses, develop off-peak market for electricity –Activity Type: Market Analysis, Product Development, Demonstration 5.Turnkey Pump and Compressed Air –$370,000; $353,000 Co-funding –Goal: Increase use of compressed air efficiency projects among manufacturers –Activity Type: Demonstration

14. 14 Assessment Criteria in Year One Knowledge Creation –Quantity –Significance Knowledge Dissemination –Availability of Knowledge Products –Impact on target audience Commercialization Progress –Capital Attraction –Technical Achievement –Market Advancement Realized and Potential Energy Benefits Realized and Potential Economic Benefits Realized and Potential Environmental and Health Benefits Value versus Costs

15. 15 Year-One Results Not rated

16. 16 Year-One Results

17. 17 Year-One Results

18. 18 Year-One Results

19. 19 Year-Two Activity Conducted a peer-review assessment of two programs Combined Heat and Power Demonstration Program: ~$50 million awarded –3-hour teleconference with presentation by program manager Environmental Monitoring Program: ~ $20 million awarded –5-hour on-site meeting with presentations by staff on critical research findings Reviewers held 30-minute meeting after presentations

20. 20 Assessment Criteria in Year 2 Knowledge Creation –Quantity –Significance Knowledge Dissemination –Availability of Knowledge Products: –Impact on target audience. Commercialization Progress –Capital Attraction: –Technical Achievement: –Market Advancement: Likelihood of Realizing Significant Energy Benefits Likelihood of Realizing Significant Economic Benefits Likelihood of Realizing Significant Environmental and Health Benefits Value versus Costs

21. 21 Year-Two Results Not Rated

22. 22 Year-Two Results

23. 23 Year-Two Results

24. 24 Reviewer Comments Knowledge (Quantity) The program has produced a large number of published papers in quality journals. Staff have gone the extra mile to summarize these for those interested in an overview of findings. Improvements could be made to expand the amount of knowledge (as opposed to the amount of data) created..

25. 25 Knowledge (Quality) The information on sulfur, nitrogen, and mercury deposition and its effects on surface waters and fauna as well as studies of the responses of surface waters to changes in emissions are of particular value and probably the most important contribution of this program. Availability of Knowledge Product The program has done an exemplary job of making results from their projects available to multiple audiences ranging from the scientific community to policy-makers. To the extent possible, links to the actual papers would be very helpful as well. EMEP should consider more detailed evaluation of the internet contact information. That information would improve the programs knowledge of which documents are found to be useful by web users. Reviewer Comments

26. 26 Reviewer Comments Target Audience Given the mission of NYSERDA over all, and the focus of the program on impacts of power industry operations, there should be greater efforts to engage the industry. EMEP might consider some sort of survey of the relevant target audiences to estimate the extent to which the appropriate target markets are reached. Environmental Determining whether the likelihood of increase is significant or if those increases are likely to be substantial is extremely difficult to gauge. A large number of new fish consumption advisories were found to be needed as a consequence of the monitoring. Some health benefits are likely, but their significance is probably difficult to quantify. Overall Value versus Cost Estimating the value of research on a dollar basis is next to impossible, particularly given that much of the benefit from program-supported research is not likely to be measurable for some time after the research is completed. I was amazed that the oversight of the program is conducted with only 2.5 full-time-equivalent employees.

27. 27 Year-Two Results

28. 28 Year-Two Results

29. 29 Conclusions Logic model was useful in identifying project types and assessment criteria Peer reviewers were able to provide useful feedback The assessment criteria appear to be clear and relevant Assessment activity validated the project types Framework can be applied to a group of related projects Although too early to tell, R&D programs appear to be on track to achieving long-term energy, economic, and environmental benefits

30. 30 Next Steps Develop methodologies for measuring economic impacts –Economic case studies –Macroeconomic impact analysis based on new investments and sales of new products Develop ratios to forecast future benefits of product development activities