1. A Theory Driven Multi-year Strategic Evaluation Plan for a Multi-program Government Agency A Presentation for American Evaluation Association Annual Conference November 2010 John H. Reed Moria Morrissey 305 Summer Garden Way Rockville, MD 20850 301 340-8701 jreed@innovologie.com

2. Overview of the presentation Introduction FEMP Strategic Evaluation Plans Elements of the Plan and Examples

3. What Does the Federal Energy Management Program Do? The stated mission of FEMP is facilitating “the Federal Government's implementation of sound, cost-effective energy management and investment practices to enhance the nation's energy security and environmental stewardship. This is delivered through project transaction services, applied technology services, and decision support services.”

4. What FEMP really does… It’s important to distinguish that FEMP facilitates energy efficiency and renewable energy rather than directly delivering energy and renewable energy services The difference between facilitation and direct implementation is an important distinction with respect to FEMP’s evaluation activities What is needed is outcome evaluations that focus on assessing the extent to which this agencies products and services influence its customers are a higher priority than assessing the direct reductions in energy use and intensity

5. Utility Energy Savings Contracts: An Example of Facilitation Develop a model contract Convince utilities to offer UESC and get agreements in place Train agency management and staff so that they understand and will approve UESC “off- book” financing May agency staff aware of UESCs and how they work Market UESCs to agencies

6. Key Questions Is the agency effective in its role as a facilitator? What facilitation services are needed?

7. What the Evaluation Needs to Do The priorities for the evaluation activities are to: –Understand the current activities and their effects and the kinds of facilitation that is needed –Demonstrate that these activities are focused on those needs –Identify opportunities for implementation activities to further strengthen the facilitation role Evaluation activities need to demonstrate that FEMP’s actions assist customer agencies in: –Acting to reduce energy and water consumption or implement more renewable energy –Accelerating actions to reduce energy –Achieving greater savings or implementing greater amounts of renewable energy and at lower cost –Demonstrating that the agency savings are tangible

8. Why a Strategic Evaluation Plan Define evaluations that provide broad based and systematic feedback to management in support of planning, budgeting, benefit assessments, and program Implementation Enable the program to demonstrate that it is meeting its goals (or not) Provide affordability and flexibility Have evaluations that build upon one another in terms of content, tools, and clearance processes Integrate programmatic and evaluation data needs to minimize the cost of evaluation and to provide program support for both the program and the evaluation Move in the direction of real-time evaluation and a program support system that provides up-to-the-minute information allowing the agency to quickly and effectively respond to upper management and stakeholders

9. Key elements of a Strategic Evaluation Plan Scope of the plan Program description and goals Evaluation Issues Metrics Prioritization of issues Evaluation strategy Description of the studies Timeline Budget

10. Scope of the Strategic Evaluation Plan Market assessment activities are important to know and understand the current and future needs of the agencies so that activities can be focused on those needs Process evaluations provide insight into the functioning of program delivery mechanisms Market effects outcome evaluations assess the behavioral impacts of activities and product usage Ultimate outcome evaluations may be ongoing or conducted after sufficient time has passed so that target audiences have had a chance to become aware of the program, decide to use it, and take action. Benefit-cost evaluation allows the program manager to understand the effectiveness of such things as alternative delivery strategies or assists a program manager in selecting components of the program that produce comparative advantage.

11. Categories of Goals that Guide FEMP Activities Facility energy intensity reduction Renewable energy use increases Water conservation requirements Advanced metering and measurement New facility construction and major renovations Energy efficiency product procurement Petroleum use reduction /alternative fuel use increase

12. Legislation Energy Independence and Security Act of 2007 (EISA 2007): EISA mandates annual energy intensity reductions for Federal facilities (2003 baseline) ranging from two percent in 2006 to 30 percent by 2015. Energy Policy Act of 2005 (EPAct 2005) requires the Secretary of Energy to ensure to the extent feasible and practical that increasing percentages, to at least 7.5 percent by 2013 and thereafter, of electricity consumed by the Federal Government be from renewable sources. It also requires the installation of 20,000 solar energy systems in Federal buildings by 2010. National Energy Conservation Policy Act (NECPA): purpose of this part to promote the conservation and the efficient use of energy and water, and the use of renewable energy sources, by the Federal Government.

13. Executive Orders Executive Order 13423 –Reduction of energy intensity by 3% annually through FY 2015 (baseline 2003). –Ensures that half of the required renewable energy consumed by the agency in a fiscal year comes from new renewable sources –Decrease petroleum consumption by 2% per year through 2015 (baseline 2005) and increase alternative fuel use by 10% year over year –Mandates increasing annual water reductions culminating in a cumulative 16 percent decrease in consumption in FY2015 (2007 baseline) –With new construction and major renovation each agency comply with the Guiding Principles for Federal Leadership in High Performance and Sustainable Buildings

14. The FEMP Story Logic Model Part 1

15. The FEMP Story Logic Model Part 2

16. Federal Energy Management Program Logic Model This model is too detailed to include in this paper. It can be viewed at www.evalframework.org/logicmodels. One challenge with the FEMP logic model was and continues to be the very large number of activities. The figure to the right shows the outcomes for two FEMP Technology Service activities Outcomes range along a continuum from short-term through intermediat The traditional diffusion process stages are represented at the right The boxes at the bottom represent intermediate outcomes or the potential results of the short-term outcomes. Awareness Persuasion Decision Implementation Intermediate outcomes

17. Researchable Issues Effectiveness of Customer Service Management Awareness, need for, use of, improvements to, coordination with other services and effectiveness of FEMP support and technology services The Facilities Initiative (formerly TEAM) and its applicability to other agencies The use of existing financing mechanisms (ESPCs, UESCs, Public Benefits Funds, Power Purchase Agreements, others) within DOE and other Agencies Progress toward efficient and sustainable buildings The efficiency of data centers Water Fleets Business support operations within FEMP The roles and value of interagency working groups and the use of their products

18. Program Metrics From the logic model key performance indicators can be identified that can be used to track performance. Metrics are directional –In early stages of an effort the value of metrics can be expected to increase –In mature programs the value of some metrics may be expected to decrease as the effects of activities take hold, activities are transformed, and there is less to be done. It is intended that these metrics be tracked regularly. They are intended to be used as indicators of where evaluation efforts should be focused and from time to time they should cause the evaluation activity to be re-focused.

19. Sample Performance Metrics Efficient and Sustainable Buildings –Percentage of new construction utilizing FEMP provided information and services –Energy intensity of new buildings –Percent change in energy intensity traceable to FEMP assistance and support Data Centers –Energy intensity of data centers –Number of data center projects utilizing FEMP support and guidance –Percent change in energy intensity traceable to factors other than FEMP –Percent change in energy intensity traceable to FEMP assistance and support

20. Evaluation Strategy Build on existing data Develop annual study plans Focus on behavior (not attitudes or satisfaction) Include estimates of facilitated savings

21. Studies Follow a General Pattern Examine existing records and data Conduct a few key interviews Develop a more detailed work plan for the specific evaluation Analyze existing data in-depth Conduct primary data collection involving either in- depth interviews, a survey, or or other methods as needed Analyze the data Provide interim reports to provide program feedback Create an annual reporting mechanism summarizing overall findings and linking the various pieces of the evaluation.

22. Evaluation Study Example One Customer Service Management Evaluation: Should be one of the first studies completed. Primarily a market assessment and process evaluation. Purpose: assess the initial steps toward developing a strong customer service support program and provide input into the future direction for a customer service management system. The major steps in this study are to: –Review effective customer service management systems –Examine availability of automated system support –Conduct interviews with FEMP staff –Conduct interviews with agency counterparts –Assess current customer service management

23. Technical Assistance and Project Support Evaluations: –General study to identifying general technical and project support needs and requirements. –Follow by studies specifically to assess the effectiveness of current activities and to recommend improvements General assessment activities: –Review current activities –Develop in-depth interview guides –Identify relevant actors –Conduct one-to-one interviews Specific study activities: –Review the current approach to the specific activity. –Identify agencies and projects that have received a specific assistance or support in the last three years –Develop interview guides –Conduct interviews with a census or sample Evaluation Study Example Two

24. Data Centers: There is widespread concern about the growth of energy-use associated with data centers. At the same time, there is a tremendous amount of activity in both the private and public sector Purpose: is to examine the data center initiative to assess its relevance and effectiveness. The major activities in this evaluation are anticipated to be: –Review the current data center efficiency support activities –Review data center efficiency activities more broadly –Review current assessment of data center activity and energy consumption in the Federal government –Develop an interview guide for use with Federal IT directors or their staff –Conduct interviews with Federal IT staff. Evaluation Study Example Three

25. Other studies Metering, Operations, and Maintenance Evaluation Evaluation of Procurement Activity Effectiveness of Current Outreach Activities Evaluated Evaluation of Financing Programs Efficient and Sustainable Buildings Data Centers Interagency Working Groups Water Fleets

26. Timeline

27. Estimated Evaluation Budget