Achieving High Savings from Weatherization Programs Jackie Berger NASCSP Conference – September 2015

Session Outline Introduction Maximizing Energy Savings Evaluation findings Using information to improve outcomes Introduction Target high energy users Install major measures where cost-effective Provide effective service delivery Maximizing Energy Savings System to measure and improve outcomes Performance Measurement Key data to assess the program Building on program success Summary & Recommendations 2

INTRODUCTION 3

Weatherization Program Evaluation and Assessment Wx programs have many different goals Evaluations need to assess accomplishments with respect to those goals Program managers need to refine program based on goal prioritization This session focuses on one goal – energy savings Savings and cost-effectiveness is a key performance measure for energy programs 4

Weatherization Program Goals Energy Savings Cost-Effectiveness Bill Reduction Health and Safety Improvement Environmental Impact Economic Impact Delivery to Vulnerable Households 5

Session Objectives 6 Weatherization Programs have Great Potential for Saving Energy Improving Lives of Low-Income Participants Research Shows Differences in Energy Savings BETWEEN and WITHIN Programs Discuss Factors Associated with Higher Savings Recommendations Ongoing Performance Measurement 6

Information Sources National WAP Evaluation 2008 Program 2010 Program 10 State-Level WAP Evaluations Over last 5 years By APPRISE and others 10 State and Utility Ratepayer Low-Income Evaluations By APPRISE 7

CONCEPTS 8

Definition of high usage varies by geography Natural Gas Usage Definition of high usage varies by geography Mountain North Census Division 950 Therms Midwest Census Region 950 Therms Northeast Census Region 900 Therms Pacific Census Division 450 Therms South Census Region 600 Therms Mountain South Census Division 450 Therms 9

Energy Program Savings Estimates NOT projections of energy savings Comparison group Weather normalization Use of utility data Energy Savings based on billing analysis 10

Energy Savings A B A-B C-D C D A-B C-D Net Savings 12 Months Pre-Wx Energy Usage Weather Normalize 12 Months Post-Wx Energy Usage Weather Normalize Wx Date A-B Gross Energy Savings 13-24 Months Pre-Wx Energy Usage Weather Normalize 1-12 Months Pre-Wx Energy Usage Weather Normalize C-D Comp. Group Wx Date Comp. Group Savings C D Net Savings Gross Energy Savings Comparison Group Savings A-B C-D 11

Foundation/Rim/Floor Insulation Major Measures Measures for single family homes In order of average impacts in comprehensive program 70-150 Therms Furnace Replacement 60-120 Therms Wall Insulation 50-100 Therms Attic Insulation Major Air Sealing 10-30 Therms Duct Sealing Thermostats Foundation/Rim/Floor Insulation 12

Major Measures Furnace Replacement Wall Insulation Attic Insulation Major Air Sealing 1000 cfm50+ Major Measures for Single Family Homes Refers to These Measures 13

Average Measure Spending WAP Programs MAJOR MEASURES 2008 2010 Note: A home with 0 major measures may have air sealing, floor/rim/ foundation insulation, setback thermostat, duct sealing/insulation, other important measures. $1,500 $2,400 $2,400 $4,000 1 $3,400 $5,300 2 $5,000 $6,700 3 $7,500 $8,500 4 14

CONSISTENT FININGS FROM PROGRAM EVALUATIONS 15

Increase Savings by … Identify, prioritize, and install appropriate measures Ensure that weatherization staff do high quality work Target high usage homes that need major measures 16

Intersection of… Practices Policies Target homes with highest potential Prioritize measures with greatest impact Furnish providers with right incentives Practices Ensure staff have needed skills and tools Use best practices for measure selection Complete high quality installation Identify problems, give feedback, resolve issues SUCCESS! 17

FACTORS ASSOCIATED WITH HIGH SAVINGS: Targeting High Usage 18

Treating High Usage Across Program Analysis Seven low-income electric efficiency program evaluations over past eight years. Seven low-income gas efficiency program evaluations over past four years. 19

Usage Impact Analysis National Weatherization Program Single Family Homes – Gas Savings Pre-Treatment Usage PY 2008 PY 2010 Obs Therms % Save Pre Use Net Save <750 therm/yr. 858 571 6 11.8% 2,181 557 64 11.5% 750-1000 963 875 133 15.2% 1,723 873 126 14.4% 1000-1250 726 1,120 206 18.4% 1,227 1,111 199 17.9% 1250-1500 472 1,367 271 19.8% 683 1,362 237 17.4% >=1500 therm/yr. 479 1,879 414 22.1% 778 1,921 345 20

State #1 WAP Energy Impacts for Single Family Site-Built Homes Net Gas Savings for Natural Gas Main Heat by Pre-Weatherization Gas Usage (therms/year) Pre-WAP Gas Use (therms/yr) # of Major Measures # Homes Gas Use Pre-WAP Net Savings % of Pre All Clients 1.4 937 983 130 (±10) 13.2% (±1.1%) <750 th/yr. 1.1 245 640 64 (±12) 10.0% (±1.9%) 750-1000 1.3 296 880 105 (±14) 12.0% (±1.6%) 1000-1250 1.6 226 1,097 142 (±22) 12.9% (±2.0%) 1250-1500 101 1,355 219 (±42) 16.2% (±3.1%) >=1500 th/yr. 2.0 69 1,731 269 (±65) 15.6% (±3.7%)

  State #2 PY 2010 WAP Energy Impacts for Single Family Site-Built Homes Net Gas Savings for Natural Gas Main Heat by Pre-Weatherization Gas Usage (therms/year) Pre-WAP Gas Use (therms/yr) # of Major Measures # Homes Gas Use Pre-WAP Net Savings % of Pre All Clients 1.9 4,065 1,043 163 (±8) 15.7% (±.0.7%) <750 th/yr. 1.7 790 642 71 (±10) 11.0% (±1.5%) 750-<1000 1.8 1,371 878 123 (±10) 14.0% (±1.1%) 1000-<1250 2.0 999 1,112 182 (±15) 16.4% (±1.4%) 1250-<1500 509 1,359 217 (±25) 15.9% (±1.9%) >=1500 th/yr. 2.2 396 1,829 365 (±43) 20.0% (±2.4%)

FACTORS ASSOCIATED WITH HIGH SAVINGS: Install Major Measures 23

Usage Impact Analysis National Weatherization Program Single Family Homes – Gas Savings Major Measures PY 2008 PY 2010 Obs Therms % Save Pre Use Net Save None 342 866 59 6.8% 733 823 37 4.5% One 983 989 118 12.0% 1,811 928 103 11.1% Two 973 1,035 181 17.5% 1,916 1,005 168 16.7% Three 619 1,146 286 25.0% 1,031 1,070 256 24.0% Four 192 1,220 414 33.9% 304 1,124 369 32.8% 24

State #1 WAP Energy Impacts for Single Family Site-Built Homes Gas Savings for Homes with Natural Gas Main Heat By Measure Combination (therms/year) Group/Breakout # Homes Gas Use Pre-WAP Net Savings % of Pre No Major Measures 202 877 37 (±15) 4.2% (±1.7%) One Major Measure 298 957 121 (±17) 12.7% (±1.8%) Two Major Measures 211 1,003 162 (±20) 16.1% (±2.0%) Three Major Measures 115 1,111 236 (±29) 21.2% (±2.6%) All Four Major Measures 33 1,179 382 (±71) 32.4% (±6.1%)

State #2 PY 2010 WAP Energy Impacts for Single Family Site-Built Homes Gas Savings for Homes with Natural Gas Main Heat By Measure Combination (therms/year) Group/Breakout # Homes Gas Use Pre-WAP Net Savings % of Pre No Major Measures 223 942 -7 (±14) 0.8% (±1.5%) Any One Major Measure 1,188 996 91 (±9) 9.1% (±0.9%) Any Two Major Measures 1,631 1,026 164 (±9) 16.0% (±0.9%) Any Three Major Measures 862 1,116 253 (±13) 22.6% (±1.2%) All Four Major Measures 156 1,305 416 (±39) 31.9% (±3.0%)

FACTORS ASSOCIATED WITH HIGH SAVINGS: Agency Performance 27

State #1 Gas Impact Results by Agency for Gas Heated Single Family Site-Built Homes Agency ID Gas Use Pre- WAP Net Savings % of Pre # of Measures A 1,077 187 (± 21) 17.3% (±1.9) 2.2 B 992 122 (± 32) 12.3 % (±3.2) 0.9 C 1,028 119 (± 40) 11.6% (±3.9) 1.2 D 948 118 (± 19) 12.4% (±2.0) 1.3 E 1,012 113 (± 64) 11.1% (±6.3) 0.7 F 937 109 (± 44) 11.6% (±4.7) G 945 107 (± 50) 11.3% (±5.3) H 875 94 (± 26) 10.7% (±3.0) I 929 94 (± 41) 10.1% (±4.4) 1.1 J 889 58 (± 27) 6.5% (±3.0) 0.5 Total 983 130 (±11) 13.2% (±0.7) 1.4

PY 2010 Gas Impact Results by Agency for Gas Heated State #2 PY 2010 Gas Impact Results by Agency for Gas Heated Single Family Site-Built Homes (therms/year) Agency ID Gas Use Pre-WAP Net Savings % of Pre # of Measures A 1,268 281 (±65) 22.2% (±5.1%) 2.2 B 1,025 250 (±43) 24.4% (±4.2%) 2.3 C 1,037 240 (±53) 23.1% (±5.1%) D 1,130 216 (±55) 19.1% (±4.9%) 2.4 E 911 211 (±41) 23.2% (±4.5%) 2.0 F 997 204 (±58) 20.5% (±5.9%) 1.4 G 1,190 195 (±17) 16.3% (±1.4%) 1.9 H 993 180 (±16) 18.1% (±1.6%) I 938 160 (±18) 17.1% (±1.9%) J 1,035 158 (±12) 15.3% (±1.2%) K 1,012 150 (±23) 14.8% (±2.2%) L 1,252 150 (±41) 12.0% (±3.2%) M 1,023 141 (±33) 13.8% (±3.3%) 1.7 N 1,039 137 (±12) 13.2% (±1.2%) O 921 130 (±32) 14.2% (±3.4%) 1.8 P 893 129 (±29) 14.5% (±3.2%) Q 988 111 (±16) 11.3% (±1.6%) 1.3 R 962 109 (±29) 11.3% (±3.1%) S 1,104 95 (±76) 8.6% (±6.9%) Total 1,043 163 (±8) 15.7% (±.0.7%) *Agencies with less than 30 homes with energy savings results are not shown. but are included in the total savings figures.

Factors by Savings Group Agencies with High Savings Agencies with Moderate Savings Agencies with Low Savings PreWX Usage 1,061 therms 1,063 therms 984 therms # of Major Measures 2.1 1.9 1.7 Average Cost of Measures $4,181 $4,625 $3,900 Mean Savings 234 therms 162 therms 118 therms Saving Percent 22.1% 15.2% 12.0% 31

State #3 249 therms 197 therms 118 therms 1,040 therms 1,098 therms Factor High Savings Moderate Savings Low Savings Savings 249 therms 197 therms 118 therms PreWX Usage 1,040 therms 1,098 therms 1,190 therms % Attic Insulation 61% 47% 54% % Wall Insulation 24% 13% 7% % Furnace Replacement 55% 46% 42% Average Spending $4,681 $4,304 $4,495 Windows/Doors Spending $1,191 $1,271 $1,369 Health and Safety $308 $438 $384 32

State by State Comparison Year 1 Year 2 PreWX Usage 1,043 1,380 984 949 785 Mean Savings 163 171 256 183 145 Saving Percent 16% 12% 24% 18% 17% Average Cost $5,200 $4,500 $9,000 $7,500 $$ per Therm Savings $32 $26 $35 $41 $52 33

Assessing Work quality WAP FIELD STUDY 34

WAP Field Study Focus Audit approach and implementation Energy education delivery Measure installation/operation of crews Post-weatherization quality assurance inspection approach and implementation Client interaction Interactions among agency staff Implications for management, procedures, and training 35

WAP Field Study Methodology Develop check lists and rating scales Train experts to implement consistently Quantify findings across all observations and inspections Enrich data with descriptive information Make recommendations based on prevalent issues 36

WAP Field Study Methodology One week observation at agency Weatherization manager and staff interviews Observations – client intake, audit, measure installation, final inspection Written agency summary report Social Scientists 3-Week Observations at Agency Observations – audit, measure installation, final inspection Post Observation Data Entry Weatherization Experts Training Monthly update calls Final debriefing Communication 37

WAP Field Study Methodology Weatherization best practices as agreed on by a national team of weatherization experts Assessment of where there are opportunities – standards, equipment, training Visited 19 agencies in 19 states Agencies selected to represent: Urban/Rural Education } Based on agency survey PY 2008 planned jobs Training Measure selection Quality Control 38

WAP Field Study Methodology Number of Observations Social Scientist Weatherization Expert Total Audit 43 112 155 Measure Installation 45 114 159 Final Inspection 37 91 128 125 317 442 39

WAP Field Study Data Collection Check List Example Audit Air Leakage and Insulation Diagnostics Applicable Observations Action Taken # % Measured surfaces 100 94 94% Inspected windows 96 96% Inspected all accessible attics 78 69 88% Measured insulation in all accessible attics 79 70 89% Created access to inaccessible attics 33 3 10% Measured insulation in exterior walls 95 47 49% Measured insulation in basement/crawlspace 74 55 74% Inspected for all typical bypasses 62 62% Visual inspection for air sealing opportunities 83 83% Used blower door while inspecting for leaks 64 67% 40

WAP Field Study Data Collection Rating Design 1 2 3 4* 5* % of Points needed 0%-49% 50%-74% 75%-89% 90%-99% 100% Total points Number of Points needed for each rating 4 0-1 -- 5 0-2 6 3-4 7 0-3 8 4-5 9 0-4 5-6 10 5-7 11 0-5 6-7 8-9 12 6-8 9-10 41

WAP Field Study Data Collection Quality of Attic Insulation 1 All air sealing work completed first 2 Exhaust fans vented to exterior as needed 3 Heat producing devices or systems protected from insulation contact 4 Attic checked for knob and tube wiring 5 Workers wore respirators, safety glasses, gloves, and hard hats while insulating attic 6 Insulation installed in sufficient quantity (bags per ft2) to meet R-value requirement 7 Proper insulation material chosen for attic conditions 8 Open blow insulation is level and of consistent depth 9 Attic ventilation maintained 10 Confined areas blown to dense pack 11 Proper containment used to protect client and belongings Needs Improvement Excellent Mean Rating # Rated Rating 1 2 3 4 5 Total Points needed 0-5 6-7 8-9 10 11 Bold Points needed 6 Attic Insulation Quality 4% 18% 42% 19% 3.3 57 42

WAP Audit Strengths Inspection – 99% inspected every accessible room Heating system – 94% inspected Filters – 72% inspected Ventilation 72% inspected the kitchen 78% inspected the bathroom ventilation. Water heater – 93% inspected Air conditioning – 82% inspected outside unit Insulation – 89% measured all accessible attics Testing – 97% conducted blower door (84% correctly) 43

WAP Audit Opportunities Bathroom ventilation flow – 24% assessed Hot water – 39% checked temperature at the faucet Shower flow – none of the auditors measured flow Insulation – 49% measured in exterior walls Blower door – 67% used while inspecting for leaks Zonal pressure diagnostics – 42% done (87% correctly) IR camera – 49% used (70% correctly) 44

WAP Communication of Audit Findings Audit write-ups do not convey educational opportunities Work order not always clear or specific enough for installer to understand the desired outcome Pre-work auditor walk through with installer Attaching photos of critical areas to the work order In process check-ups with the installers Tools that improved information transfer Air sealing is a particular area for improved communication Need for flexibility to change work order when problems are found during installations 45

WAP Audit Findings Opportunities Strengths Increase client engagement Respect and concern for clients Explanation of WAP program and process Use of data collection forms Auditors meet with contractors to explain audit findings and work scope Utilize energy bills Increase client engagement Identify client-specific opportunities for reducing energy use Auditor understanding of pressure boundaries Auditor understanding of testing purpose and procedures Conduct worst case draft testing 46

WAP Measure Installation Observation Number Percent Crews 35 31% Contractors 74 65% Both 5 4% TOTAL 114 100% 47

WAP Measure Installation Strengths Air sealing prioritized the right areas 77 percent prioritized sealing at the top and bottom of the envelope Attic insulation followed best practices 77 percent completed attic floor sealing prior to insulation 88 percent had no gaps or voids 95 percent used appropriate materials 48

WAP Measure Installation Mid-High Ratings Area Ratings Window work 3.5 – 4.0 Door work 2.8 – 3.5 Attic insulation 4.0 Wall insulation 2.9 Basement insulation 3.6 Crawl space work 3.1 – 4.0 Ventilation work 4.1 – 4.8 Thermostat installation 4.1 Air conditioning work 3.0 – 4.0 Water heater work 49

WAP Measure Installation Opportunities Air Sealing Diagnostics 22 percent used the blower door to guide air sealing 11 percent used the zonal pressure test to affirm appropriate pressure boundaries Air Sealing 57 percent sealed all major opportunities Low Ratings Area Ratings Air sealing 2.4 Heating system work 2.0 – 2.5 Smoke detector installation 2.6 CO detector installation 2.0 50

WAP Measure Installation Findings Opportunities Strengths Increased use of blower door when air sealing Respect for clients’ homes (booties, covering furniture) Crew member safety Increased assessment of HVAC contractors Explain CFLs when installing Client education Experienced HVAC contractors Some very high quality work Crew problem solving Crew flexibility and dedication (above and beyond work scope) 51

WAP Final Inspection Strengths 75 percent discussed the work that was performed with the client The inspectors discussed the work that was performed 90 percent examined all accessible rooms Inspections were complete 85 percent conducted this test (85 percent of those performed correctly) Blower door testing was usually conducted 52

WAP Final Inspection Opportunities Zonal Pressure Test 33 percent performed the test (87 percent correctly) IR Camera 44 percent used the camera (95 percent correctly) Combustion Safety 31 percent performed this test on the gas stove (100 percent correctly) Worst Case Draft Test 60 percent performed this test on the heating system (92 percent correctly) 53

WAP Final Inspection Findings Opportunities Strengths Increased client education Explain measures installed Reinforce client action plan Improved testing quality Increased assessment of installation quality Reduced use of auditor to conduct final inspection – additional perspective Good job when following work order closely Time saved when contractor attends Information to clients – who to call if they have problems Referrals to additional assistance programs 54

WAP Training Needs Energy bill utilization – to assess needs and educate clients Building science fundamentals Critical thinking – assessing unique situations Combustion safety testing Zonal pressure testing IR camera use Ventilation assessment Safe work practices – worker safety and lead safe work Interviewing skills – how to understand the client’s needs Client education 55

WAP Management Opportunities Standards and Procedures Policy Manuals Forms and Checklists 56

WAP Equipment Needs GPS IR Camera Boroscope and Fiber Optic Scope and Video Personal Safety Equipment 57

Performance measurement 58

Performance Measurement Steps Program Statistics Refine Pilot Changes Quality Control Hold Contractors Accountable Assess Inputs and Outputs Assess Inspection Results Assess Energy Savings 59

Performance Measurement 1- Develop Baseline Stats Pre-treatment usage Major measure installation rates Energy savings measured through billing analysis 60

Performance Measurement 2- Refine Procedures Review and refine procedures Ensure documentation reflects expectations and best practices Train contractors on weak areas Review contractor understanding of program requirements 61

Performance Measurement 3- Pilot Program Changes Incremental changes Pilot test innovative strategies Examples of innovations Contractor compensation plan Procedures for treating different types of homes Low usage High baseload usage Health and safety problems Home previously treated 62

Performance Measurement 4- Conduct Quality Control Observe work in the field Frequently inspect completed jobs Verify procedures are correctly implemented Verify comprehensive work Review all aspects of the work Audits, work scope, installation, third party inspection Require contractors to fix unacceptable work Ensure all parties agree to specifications and procedures 63

Performance Measurement 5- Require Contractor Accountability Require contractors to return to home to fix problems Provide training in problem areas Set goals for contractor performance Conduct additional observations and inspections Remove contractors who do not improve 64

Performance Measurement 6- Assess Inputs and Outputs Pre-treatment usage Measure spending distribution Outputs Measure installation rates Are they improving enough to lead to better results? 65

Performance Measurement 7- Assess Inspection Results Review rates of: Comprehensive installation High quality installations Missed opportunities Poor work quality Early indication of energy saving expectations. 66

Performance Measurement 8- Assess Energy Savings Conduct billing analysis on an annual basis Needed to ensure expected results Develop procedures to more easily extract data Reduced evaluation cost 67

Performance Measurement Repeat Compare Results Over Time Assess What is Working Refine Program 68

summary 69

Achieving High Savings Lessons Learned It is challenging to meet savings expectations Target high usage customers Ensure major measures are installed where opportunities exist Maximize use of proven home performance techniques Conduct performance measurement 70

Contact Information Jackie Berger President jackie-berger@ appriseinc.org 609-252-8009 David Carroll Managing Director david-carroll@ 609-252-8010 APPRISE 32 Nassau Street Suite 200 Princeton, NJ 08542 71