1. B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT: Methods for Evaluating Residential Behavior-based Programs RTF Presentation February 2, 2010 Lauren Gage (lsmgage@bpa.gov) Bobbi Wilhelm (bobette.wilhelm@pse.com)

2. Slide 2 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Goals for RTF Meeting  Introduce methods developed  Gain feedback from RTF members  Bring final methods to RTF in March for action/ approval

3. Slide 3 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Background Residential behavioral-based programs are gaining popularity and promise significant savings potential Multiple programmatic approaches are being tested throughout the country In-home feedback devices Energy benchmarking information Others (e.g., community-based programs). Current research on quantity and persistence of savings is sparse

4. Slide 4 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Process  Group of NW utilities and energy organizations held meetings to discuss regional evaluation strategies for behavior-based energy programs. – Puget Sound Energy, BPA, Seattle City Light, Snohomish County PUD, Energy Trust of Oregon, and Eugene Water & Electric Board.  The group agreed on the need for standard methods for evaluation of behavior-based conservation programs within the Pacific Northwest.  Group drafted previous version, received input from multiple national experts and revised to current version

5. Slide 5 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Outcomes and Caveats  These are recommended guidelines and/or methods for evaluating behavioral programs for Pacific Northwest utilities.  They consider the research completed to date and are designed to evolve as more studies are completed.  They should not be viewed as the only acceptable approaches to estimating savings.  Once reviewed and approved by the RTF, they would be an agreed-upon approach that provides guidance to future evaluations

6. Slide 6 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Residential Behavior-Based Programs (RBBP) Definitions  Programs designed to save energy through: – Changes in behavior - e.g., turning off lights, setting thermostats – Increasing investments in energy-efficiency measures  Program Examples: – Energy benchmarking compares participant consumption to historical consumption or to peers. – Feedback devices use monitoring or metering devices to provide information on instantaneous demand or consumption over time at the whole-house or end-use level – Others include: Information and training programs, Schools-based programs and marketing/community-based programs

7. Slide 7 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N The Power Act and RBBP  Historically, the Council has excluded measures and practices that reduced the level of service or utility (in economic terms) provided to consumers by the current measure or practice. – 839a(3). "Conservation" means any reduction in electric power consumption as a result of increases in the efficiency of energy use, production, or distribution. [Northwest Power Act, §3(3), 94 Stat. 2698.] – Savings from lowering thermostats for space heating have not been considered conservation under the Act – Savings from lowering the thermostat on water heaters from 140 F or 130 F to 120 F have been.  Rationale for behavioral change programs as “utility neutral”, or non-sacrificial? – Persistent savings are unlikely to be sacrifices – Programs are asking customers to reduce behavior when it does not change utility (e.g., turning off lights or thermostat down when not in the room) – Recent interest in sustainability has created utility for reducing energy consumption, particularly enabled through technology or information

8. Slide 8 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Contents of the Methods  Development of an evaluation plan  Participant and comparison group selection  Collect sufficient and relevant data  Conduct data cleaning consistent with best practices  Estimate program savings  Estimate savings from other non-RBBP programs  Estimate the persistence of savings

9. Slide 9 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N Development of Evaluation Plan  Each program may call for different evaluation approaches. Therefore, first step is to develop an evaluation plan.  Evaluation plan should include: – Program design and logic model, if applicable – General research questions to be addressed in the evaluation – Program participants and comparison selection – Methods for data collection, cleaning – Analytical approaches – Multi-year approach DRAFT for RTF DISCUSSION

10. Slide 10 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Participant and Comparison Selection: Utility-selected Participants  If utility selects the program participants – Ideally, use experimental research design – If not possible, then should select comparison group at the time of program deployment.  In selecting participants, the evaluation plan should document – (1) sample selection criteria, – (2) estimated sample size and – (3) the methodology used to determine the appropriate sample size, including the assumed precision and confidence level.

11. Slide 11 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N Participant and Comparison Selection: Comparison group (non-experimental design)  If participants self select or a comparison group is not chosen through experimental research design, then the comparison group should be chosen to be similar to the participant group and at least as large in size.  Minimally, this criteria would include home type, location, and total baseline consumption. – Additional characteristics may be necessary or may increase the relevance of the comparison group - home characteristics, heating fuels, household size, community characteristics, occupant demographics or motivations.  In selecting a comparison group, the evaluation plan should document (1) the timing of comparison group selection, (2) selection criteria, (3) estimated sample size and (4) the methodology used to determine the appropriate sample size, including the assumed precision and confidence level. DRAFT for RTF DISCUSSION

12. Slide 12 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Collect Sufficient and Relevant Data  Billing data comparison is primary method discussed in methods.  Evaluation plan should outline process to collect sufficient and relevant data for the study.  Minimum data requirements are: – Billing data: Daily (or monthly or bi-monthly) kWh and/or therm data for comparison and participant groups - at least one year prior to the study period, during and after participation – Weather data: HDD and CDD for all months/days used in the billing analysis – Participation group information: Participant or comparison group assignments, frequency of participation, participation start date – Energy-efficiency measure installation: Utility-sponsored installations, Optional survey data on other measure installation for a sub-sample  Other data may improve the analysis or be necessary depending on program design and evaluation, including household and building characteristics data and attitudinal or behavioral information.

13. Slide 13 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N Conduct Data Cleaning Consistent with Best Practices  Data should be managed, cleaned and reviewed using industry- best practices.  All steps taken in data cleaning should be documented, attrition rates should be tracked and removed data should be available for review.  Some data cleaning methods to be documented in the evaluation report include households with zero energy reads, households with estimated reads, households lacking sufficient pre- or post- participation data, households with change in occupancy, and households with exceptionally high or low consumption. DRAFT for RTF DISCUSSION

14. Slide 14 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Estimating Savings  In general, these methods recommend a regression- based model that considers other factors that affect consumption. Program savings should, at a minimum, be measured annually.  At a minimum, the model(s) should estimate the difference in the change in consumption between participant and comparison groups, accounting for weather and other key factors.

15. Slide 15 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N Estimating Savings: Models  PRISM-type approach - estimates Normalized Annual Consumption for pre- and post- periods for participant and comparison groups; calculate program savings as the difference- in-differences  Monthly billing data model: describes monthly energy usage as a function of weather, a participation indicator, a period indicator and relevant interaction variables.  Either model option could be expanded to include additional variables to improve the model’s explanatory power. DRAFT for RTF DISCUSSION

16. Slide 16 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N Estimate Savings Achieved Through other Non-RBBP Programs  Estimating observed savings from EE measures installed with a utility incentive - avoids double counting of savings and facilitates understanding of investment behavior in participants.  Possible methods: – Calculate savings based on non-RBBP estimates. Match participant and comparison HHs with utility program participation information, using existing estimates – Utilize regression-based models. Variables for non-RBBP program participation can be included or could exclude any participants in non- RBBPs. DRAFT for RTF DISCUSSION

17. Slide 17 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N Persistence  At this point it is unclear how long expected behavior modifications, and the savings associated with those behavior modifications will last.  Evaluation plan, program design must account for persistence measurement.  Generally, methods recommend: estimate the same regression model for each evaluation year. Differences in the year-to-year savings of the participant group will indicate how the savings change over time.  For additional understanding of persistence and optimal program design, the evaluation could adjust the sample over time and/or add survey data. DRAFT for RTF DISCUSSION

18. Slide 18 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Q&A