1. Building America: Retrospective Evaluation of a Unique DOE R&D and Market Diffusion Program
As Jack and Daniel noted, the Guide for Evaluating Realized Impacts of DOE’s EERE R&D Program provides a standard method for conducting evaluations of R&D investments which it followed. However, this study is the first in a series of 7 EERE retrospective impact evaluations for which market-oriented components—rather than R&D--comprise the majority of the investment. Thus, while the evaluation team followed the Guide where applicable, it was required to decide if additional approaches were needed to address the market-oriented investment components. This required identifying where the differences lay and the nature of the differences. I will provide background to Daniel’s presentation by focusing on how evaluation of R&D and systems investments may differ from evaluation of market-oriented investment.
Evaluating r&D & Market Diffusion benefits: Methodology Issues
Rosalie Ruegg
TIA Consulting
*Based on work funded by the U.S. Department of Energy Office of Energy Efficiency and Renewable Energy.

2. Evaluating Impacts of Different Types of Investment—e. g
Evaluating Impacts of Different Types of Investment—e.g., R&D, Systems Integration, & Market Stimulation: Methodology Issues
How do different types of investments generate impacts?
What difference does it make to evaluation?
TIA Consulting, Inc.

3. DOE/EERE Investments Generally Share Long-term Goals
More efficient use of energy and other resources.
Reduction of air emissions from fossil fuel use.
Energy security.
Improved health, environment, and economic wellbeing of citizens.
Generation of overall positive net benefits for the nation.
Thus, when the time horizon is long, EERE investments of different types can often be evaluated using a single set of metrics.
TIA Consulting, Inc.

4. Most DOE/EERE Investments Generate Costs in Similar Ways
Budget items
Measured in dollars (current converted to constant dollars)
Weighed against benefits to assess if the investment has been worthwhile.
So far, so good, but , now let’s consider differences that give rise to methodology issues.
TIA Consulting, Inc.

5. Market-Oriented Investments mainly affect demand for product/process.
Many Similarities, BUT Different Types of Investments May Differ in How They Generate Impacts
R&D Investments (and certain types of Systems Investments) mainly affect supply of a resulting product/process
- Improved performance
- Reduced costs
Market-Oriented Investments mainly affect demand for product/process.
- Changed behaviors of builders, financiers, & homeowners by r raising their awareness/knowledge/attitudes
TIA Consulting, Inc.

6. R&D & Systems Integration Investments Produce Changes in Supply Market-oriented Investments Produce Changes in Demand
Demand/Supply/Price
Supply 1
Supply 2 P3 P1 P2
Demand 2
Demand 1 Q1 Q2 Q3
TIA Consulting, Inc.

7. These Investments differ in timing
Add Market Investments
Add DOE R&D & Systems Investments
Without DOE Investments
$ Net Cash Flow
Time during which R&D & Market investments and impacts--& evaluation occur
Market-oriented investments tend to lag R&D investments—and they don’t have the same period over which their benefits are realized. This difference in timing may affect the metrics and methods used to evaluated each type of investment –next pp.
R&D & Systems Investments
Market-Oriented Investments
*NOTE: Framework Guide cover retrospective impact evaluation of R&D
TIA Consulting, Inc.

8. How long are evaluation time horizons?
Can Evaluation be Accomplished with a Single Set of Impact Metrics or Will Multiples be Needed?
How long are evaluation time horizons?
How far along are R&D and systems integration investments?
How far along are market-oriented investments?
How fragmented are the activities and targeted constituencies of the market-oriented investments?
(e.g., some actions may affect financiers, others may affect some part of the building community, while others may affect some but not all homeowners.)
TIA Consulting, Inc.

9. Impact Metrics are Sensitive to Type of Investment and Stage of Evaluation
Evaluation Metrics for R&D & Systems Investments Affecting Mainly Supply
Evaluation Metrics for Market-Oriented Investments Affecting Mainly Demand
Early Stage
Supply Chain metrics
Net Chain metrics
Social Network metrics
Knowledge base metrics
Behavioral Change metrics by group
Replication after demonstrations A, B, C
Take-up of recommended practices X, Y, Z
Trend in sales before & after intervention
Growth in different market sectors
Later Stage
Energy and other resource effects
Environmental effects
Energy security effects
Knowledge diffusion effects
Economic impacts
Economic Impacts
TIA Consulting, Inc.

10. When Faced with Multiple Investment Types…
Develop a logic model of the program/portfolio, activities, outcomes
Assess the timing of different types of investment and the timing of impacts
For retrospective R&D investment, follow EERE guide for assessing impacts in terms of
-Energy and other Resource Impacts
-Environmental Healthcare Impacts
-Environmental Climate Impacts
-Energy Security Impacts
-Knowledge Impacts
If non-R&D investment, analyze the type, timing and appropriate treatment
Systems & process investments can usually be treated same as R&D
For market-oriented investments, develop a strategy that best fits the characteristics & timing of each component
Report on investment impacts, combining impacts where feasible
TIA Consulting, Inc.

11. Rosalie Ruegg, Managing Director TIA Consulting, Inc. ruegg@ec.rr.com
Questions & Comments
Rosalie Ruegg, Managing Director
TIA Consulting, Inc.