1. Achieving High Savings from Weatherization Programs
Jackie Berger
NASCSP Conference – September 2015

2. Session Outline Introduction Maximizing Energy Savings
Evaluation findings
Using information to improve outcomes
Introduction
Target high energy users
Install major measures where cost-effective
Provide effective service delivery
Maximizing Energy Savings
System to measure and improve outcomes
Performance Measurement
Key data to assess the program
Building on program success
Summary & Recommendations 2

3. INTRODUCTION 3

4. Weatherization Program Evaluation and Assessment
Wx programs have many different goals
Evaluations need to assess accomplishments with respect to those goals
Program managers need to refine program based on goal prioritization
This session focuses on one goal – energy savings
Savings and cost-effectiveness is a key performance measure for energy programs 4

5. Weatherization Program Goals
Energy Savings
Cost-Effectiveness
Bill Reduction
Health and Safety Improvement
Environmental Impact
Economic Impact
Delivery to Vulnerable Households 5

6. Session Objectives 6 Weatherization Programs have Great Potential for
Saving Energy
Improving Lives of Low-Income Participants
Research Shows
Differences in Energy Savings
BETWEEN and WITHIN Programs
Discuss Factors Associated with Higher Savings
Recommendations
Ongoing Performance Measurement 6

7. Information Sources National WAP Evaluation 2008 Program 2010 Program
10 State-Level WAP Evaluations
Over last 5 years
By APPRISE and others
10 State and Utility Ratepayer Low-Income Evaluations
By APPRISE 7

8. CONCEPTS 8

9. Definition of high usage varies by geography
Natural Gas Usage
Definition of high usage varies by geography
Mountain North
Census Division
950 Therms
Midwest
Census Region
950 Therms
Northeast
Census
Region
900 Therms
Pacific
Census
Division
450 Therms
South
Census
Region
600 Therms
Mountain South
Census Division
450 Therms 9

10. Energy Program Savings Estimates
NOT projections of energy savings
Comparison group
Weather normalization
Use of utility data
Energy Savings based on billing analysis 10

11. Energy Savings A B A-B C-D C D A-B C-D Net Savings 12 Months Pre-Wx
Energy Usage
Weather Normalize
12 Months Post-Wx
Energy Usage
Weather Normalize Wx
Date
A-B
Gross Energy Savings
13-24 Months Pre-Wx
Energy Usage
Weather Normalize
1-12 Months Pre-Wx
Energy Usage
Weather Normalize
C-D
Comp.
Group Wx
Date
Comp. Group Savings C D
Net Savings
Gross Energy Savings
Comparison Group Savings
A-B
C-D 11

12. Foundation/Rim/Floor Insulation
Major Measures
Measures for single family homes
In order of average impacts in comprehensive program
Therms
Furnace Replacement
Therms
Wall Insulation
Therms
Attic Insulation
Major Air Sealing
10-30 Therms
Duct Sealing
Thermostats
Foundation/Rim/Floor Insulation 12

13. Major Measures
Furnace Replacement
Wall Insulation
Attic Insulation
Major Air Sealing
1000 cfm50+
Major Measures for Single Family Homes Refers to These Measures 13

14. Average Measure Spending WAP Programs
MAJOR
MEASURES
2008
2010
Note: A home with 0 major measures may have air sealing, floor/rim/ foundation insulation, setback thermostat, duct sealing/insulation, other important measures.
$1, $2,400
$2, $4,000 1
$3, $5,300 2
$5, $6,700 3
$7, $8,500 4 14

15. CONSISTENT FININGS FROM PROGRAM EVALUATIONS15

16. Increase Savings by …
Identify, prioritize, and install appropriate measures
Ensure that weatherization staff do high quality work
Target high usage homes that need major measures 16

17. Intersection of… Practices Policies
Target homes with highest potential
Prioritize measures with greatest impact
Furnish providers with right incentives
Practices
Ensure staff have needed skills and tools
Use best practices for measure selection
Complete high quality installation
Identify problems, give feedback, resolve issues
SUCCESS! 17

18. FACTORS ASSOCIATED WITH HIGH SAVINGS: Targeting High Usage18

19. Treating High Usage Across Program Analysis
Seven low-income electric efficiency program evaluations over past eight years.
Seven low-income gas efficiency program evaluations over past four years. 19

20. Usage Impact Analysis National Weatherization Program
Single Family Homes – Gas Savings
Pre-Treatment Usage
PY 2008
PY 2010
Obs
Therms
% Save
Pre Use
Net Save
<750 therm/yr.
858
571 6
11.8%
2,181
557 64
11.5%
963
875
133
15.2%
1,723
873
126
14.4%
726
1,120
206
18.4%
1,227
1,111
199
17.9%
472
1,367
271
19.8%
683
1,362
237
17.4%
>=1500 therm/yr.
479
1,879
414
22.1%
778
1,921
345 20

21. State #1 WAP Energy Impacts for Single Family Site-Built Homes Net Gas Savings for Natural Gas Main Heat by Pre-Weatherization Gas Usage (therms/year)
Pre-WAP Gas Use (therms/yr)
# of Major Measures
# Homes
Gas Use Pre-WAP
Net Savings
% of Pre
All Clients
1.4
937
983
130 (±10)
13.2% (±1.1%)
<750 th/yr.
1.1
245
640
64 (±12)
10.0% (±1.9%)
1.3
296
880
105 (±14)
12.0% (±1.6%)
1.6
226
1,097
142 (±22)
12.9% (±2.0%)
101
1,355
219 (±42)
16.2% (±3.1%)
>=1500 th/yr.
2.0 69
1,731
269 (±65)
15.6% (±3.7%)

22.   State #2 PY 2010 WAP Energy Impacts for Single Family Site-Built Homes Net Gas Savings for Natural Gas Main Heat by Pre-Weatherization Gas Usage (therms/year)
Pre-WAP Gas Use (therms/yr)
# of Major Measures
# Homes
Gas Use Pre-WAP
Net Savings
% of Pre
All Clients
1.9
4,065
1,043
163 (±8)
15.7% (±.0.7%)
<750 th/yr.
1.7
790
642
71 (±10)
11.0% (±1.5%)
750-<1000
1.8
1,371
878
123 (±10)
14.0% (±1.1%)
1000-<1250
2.0
999
1,112
182 (±15)
16.4% (±1.4%)
1250-<1500
509
1,359
217 (±25)
15.9% (±1.9%)
>=1500 th/yr.
2.2
396
1,829
365 (±43)
20.0% (±2.4%)

23. FACTORS ASSOCIATED WITH HIGH SAVINGS: Install Major Measures23

24. Usage Impact Analysis National Weatherization Program
Single Family Homes – Gas Savings
Major Measures
PY 2008
PY 2010
Obs
Therms
% Save
Pre Use
Net Save
None
342
866 59
6.8%
733
823 37
4.5%
One
983
989
118
12.0%
1,811
928
103
11.1%
Two
973
1,035
181
17.5%
1,916
1,005
168
16.7%
Three
619
1,146
286
25.0%
1,031
1,070
256
24.0%
Four
192
1,220
414
33.9%
304
1,124
369
32.8% 24

25. State #1 WAP Energy Impacts for Single Family Site-Built Homes Gas Savings for Homes with Natural Gas Main Heat By Measure Combination (therms/year)
Group/Breakout
# Homes
Gas Use
Pre-WAP
Net Savings
% of Pre
No Major Measures
202
877
37 (±15)
4.2% (±1.7%)
One Major Measure
298
957
121 (±17)
12.7% (±1.8%)
Two Major Measures
211
1,003
162 (±20)
16.1% (±2.0%)
Three Major Measures
115
1,111
236 (±29)
21.2% (±2.6%)
All Four Major Measures 33
1,179
382 (±71)
32.4% (±6.1%)

26. State #2 PY 2010 WAP Energy Impacts for Single Family Site-Built Homes Gas Savings for Homes with Natural Gas Main Heat By Measure Combination (therms/year)
Group/Breakout
# Homes
Gas Use
Pre-WAP
Net Savings
% of Pre
No Major Measures
223
942
-7 (±14)
0.8% (±1.5%)
Any One Major Measure
1,188
996
91 (±9)
9.1% (±0.9%)
Any Two Major Measures
1,631
1,026
164 (±9)
16.0% (±0.9%)
Any Three Major Measures
862
1,116
253 (±13)
22.6% (±1.2%)
All Four Major Measures
156
1,305
416 (±39)
31.9% (±3.0%)

27. FACTORS ASSOCIATED WITH HIGH SAVINGS: Agency Performance27

28. State #1 Gas Impact Results by Agency for Gas Heated Single Family Site-Built Homes
Agency ID
Gas Use Pre- WAP
Net Savings
% of Pre
# of Measures A
1,077
187 (± 21)
17.3% (±1.9)
2.2 B
992
122 (± 32)
12.3 % (±3.2)
0.9 C
1,028
119 (± 40)
11.6% (±3.9)
1.2 D
948
118 (± 19)
12.4% (±2.0)
1.3 E
1,012
113 (± 64)
11.1% (±6.3)
0.7 F
937
109 (± 44)
11.6% (±4.7) G
945
107 (± 50)
11.3% (±5.3) H
875
94 (± 26)
10.7% (±3.0) I
929
94 (± 41)
10.1% (±4.4)
1.1 J
889
58 (± 27)
6.5% (±3.0)
0.5
Total
983
130 (±11)
13.2% (±0.7)
1.4

29. PY 2010 Gas Impact Results by Agency for Gas Heated
State #2
PY 2010 Gas Impact Results by Agency for Gas Heated
Single Family Site-Built Homes (therms/year)
Agency ID
Gas Use
Pre-WAP
Net Savings
% of Pre
# of Measures A
1,268
281 (±65)
22.2% (±5.1%)
2.2 B
1,025
250 (±43)
24.4% (±4.2%)
2.3 C
1,037
240 (±53)
23.1% (±5.1%) D
1,130
216 (±55)
19.1% (±4.9%)
2.4 E
911
211 (±41)
23.2% (±4.5%)
2.0 F
997
204 (±58)
20.5% (±5.9%)
1.4 G
1,190
195 (±17)
16.3% (±1.4%)
1.9 H
993
180 (±16)
18.1% (±1.6%) I
938
160 (±18)
17.1% (±1.9%) J
1,035
158 (±12)
15.3% (±1.2%) K
1,012
150 (±23)
14.8% (±2.2%) L
1,252
150 (±41)
12.0% (±3.2%) M
1,023
141 (±33)
13.8% (±3.3%)
1.7 N
1,039
137 (±12)
13.2% (±1.2%) O
921
130 (±32)
14.2% (±3.4%)
1.8 P
893
129 (±29)
14.5% (±3.2%) Q
988
111 (±16)
11.3% (±1.6%)
1.3 R
962
109 (±29)
11.3% (±3.1%) S
1,104
95 (±76)
8.6% (±6.9%)
Total
1,043
163 (±8)
15.7% (±.0.7%)
*Agencies with less than 30 homes with energy savings results are not shown. but are included in the total savings figures.

31. Factors by Savings Group
Agencies with High Savings
Agencies with Moderate Savings
Agencies with Low Savings
PreWX Usage
1,061 therms
1,063 therms
984 therms
# of Major Measures
2.1
1.9
1.7
Average Cost of Measures
$4,181
$4,625
$3,900
Mean Savings
234 therms
162 therms
118 therms
Saving Percent
22.1%
15.2%
12.0% 31

32. State #3 249 therms 197 therms 118 therms 1,040 therms 1,098 therms
Factor
High Savings
Moderate Savings
Low Savings
Savings
249 therms
197 therms
118 therms
PreWX Usage
1,040 therms
1,098 therms
1,190 therms
% Attic Insulation
61%
47%
54%
% Wall Insulation
24%
13% 7%
% Furnace Replacement
55%
46%
42%
Average Spending
$4,681
$4,304
$4,495
Windows/Doors Spending
$1,191
$1,271
$1,369
Health and Safety
$308
$438
$384 32

33. State by State Comparison
Year 1
Year 2
PreWX Usage
1,043
1,380
984
949
785
Mean Savings
163
171
256
183
145
Saving Percent
16%
12%
24%
18%
17%
Average Cost
$5,200
$4,500
$9,000
$7,500
$$ per Therm Savings
$32
$26
$35
$41
$52 33

34. Assessing Work quality WAP FIELD STUDY34

35. WAP Field Study Focus Audit approach and implementation
Energy education delivery
Measure installation/operation of crews
Post-weatherization quality assurance inspection approach and implementation
Client interaction
Interactions among agency staff
Implications for management, procedures, and training 35

36. WAP Field Study Methodology
Develop check lists and rating scales
Train experts to implement consistently
Quantify findings across all observations and inspections
Enrich data with descriptive information
Make recommendations based on prevalent issues 36

37. WAP Field Study Methodology
One week observation at agency
Weatherization manager and staff interviews
Observations – client intake, audit, measure installation, final inspection
Written agency summary report
Social Scientists
3-Week Observations at Agency
Observations – audit, measure installation, final inspection
Post Observation Data Entry
Weatherization Experts
Training
Monthly update calls
Final debriefing
Communication 37

38. WAP Field Study Methodology
Weatherization best practices as agreed on by a national team of weatherization experts
Assessment of where there are opportunities – standards, equipment, training
Visited 19 agencies in 19 states
Agencies selected to represent:
Urban/Rural
Education }
Based on agency survey
PY 2008 planned jobs
Training
Measure selection
Quality Control 38

39. WAP Field Study Methodology
Number of Observations
Social Scientist
Weatherization Expert
Total
Audit 43
112
155
Measure Installation 45
114
159
Final Inspection 37 91
128
125
317
442 39

40. WAP Field Study Data Collection
Check List Example
Audit Air Leakage and Insulation Diagnostics
Applicable Observations
Action Taken # %
Measured surfaces
100 94
94%
Inspected windows 96
96%
Inspected all accessible attics 78 69
88%
Measured insulation in all accessible attics 79 70
89%
Created access to inaccessible attics 33 3
10%
Measured insulation in exterior walls 95 47
49%
Measured insulation in basement/crawlspace 74 55
74%
Inspected for all typical bypasses 62
62%
Visual inspection for air sealing opportunities 83
83%
Used blower door while inspecting for leaks 64
67% 40

41. WAP Field Study Data Collection Rating Design1 2 3 4* 5*
% of Points needed
0%-49%
50%-74%
75%-89%
90%-99%
100%
Total points
Number of Points needed for each rating 4
0-1 -- 5
0-2 6
3-4 7
0-3 8
4-5 9
0-4
5-6 10
5-7 11
0-5
6-7
8-9 12
6-8
9-10 41

42. WAP Field Study Data Collection
Quality of Attic Insulation 1
All air sealing work completed first 2
Exhaust fans vented to exterior as needed 3
Heat producing devices or systems protected from insulation contact 4
Attic checked for knob and tube wiring 5
Workers wore respirators, safety glasses, gloves, and hard hats while insulating attic 6
Insulation installed in sufficient quantity (bags per ft2) to meet R-value requirement 7
Proper insulation material chosen for attic conditions 8
Open blow insulation is level and of consistent depth 9
Attic ventilation maintained 10
Confined areas blown to dense pack 11
Proper containment used to protect client and belongings
Needs Improvement
Excellent
Mean Rating
# Rated
Rating 1 2 3 4 5
Total Points needed
0-5
6-7
8-9 10 11
Bold Points needed 6
Attic Insulation Quality 4%
18%
42%
19%
3.3 57 42

43. WAP Audit Strengths Inspection – 99% inspected every accessible room
Heating system – 94% inspected
Filters – 72% inspected
Ventilation
72% inspected the kitchen
78% inspected the bathroom ventilation.
Water heater – 93% inspected
Air conditioning – 82% inspected outside unit
Insulation – 89% measured all accessible attics
Testing – 97% conducted blower door (84% correctly) 43

44. WAP Audit Opportunities
Bathroom ventilation flow – 24% assessed
Hot water – 39% checked temperature at the faucet
Shower flow – none of the auditors measured flow
Insulation – 49% measured in exterior walls
Blower door – 67% used while inspecting for leaks
Zonal pressure diagnostics – 42% done (87% correctly)
IR camera – 49% used (70% correctly) 44

45. WAP Communication of Audit Findings
Audit write-ups do not convey educational opportunities
Work order not always clear or specific enough for installer to understand the desired outcome
Pre-work auditor walk through with installer
Attaching photos of critical areas to the work order
In process check-ups with the installers
Tools that improved information transfer
Air sealing is a particular area for improved communication
Need for flexibility to change work order when problems are found during installations 45

46. WAP Audit Findings Opportunities Strengths Increase client engagement
Respect and concern for clients
Explanation of WAP program and process
Use of data collection forms
Auditors meet with contractors to explain audit findings and work scope
Utilize energy bills
Increase client engagement
Identify client-specific opportunities for reducing energy use
Auditor understanding of pressure boundaries
Auditor understanding of testing purpose and procedures
Conduct worst case draft testing 46

47. WAP Measure Installation Observation
Number
Percent
Crews 35
31%
Contractors 74
65%
Both 5 4%
TOTAL
114
100% 47

48. WAP Measure Installation Strengths
Air sealing prioritized the right areas
77 percent prioritized sealing at the top and bottom of the envelope
Attic insulation followed best practices
77 percent completed attic floor sealing prior to insulation
88 percent had no gaps or voids
95 percent used appropriate materials 48

49. WAP Measure Installation Mid-High Ratings
Area
Ratings
Window work
3.5 – 4.0
Door work
2.8 – 3.5
Attic insulation
4.0
Wall insulation
2.9
Basement insulation
3.6
Crawl space work
3.1 – 4.0
Ventilation work
4.1 – 4.8
Thermostat installation
4.1
Air conditioning work
3.0 – 4.0
Water heater work 49

50. WAP Measure Installation Opportunities
Air Sealing Diagnostics
22 percent used the blower door to guide air sealing
11 percent used the zonal pressure test to affirm appropriate pressure boundaries
Air Sealing
57 percent sealed all major opportunities
Low Ratings
Area
Ratings
Air sealing
2.4
Heating system work
2.0 – 2.5
Smoke detector installation
2.6
CO detector installation
2.0 50

51. WAP Measure Installation Findings
Opportunities
Strengths
Increased use of blower door when air sealing
Respect for clients’ homes (booties, covering furniture)
Crew member safety
Increased assessment of HVAC contractors
Explain CFLs when installing
Client education
Experienced HVAC contractors
Some very high quality work
Crew problem solving
Crew flexibility and dedication (above and beyond work scope) 51

52. WAP Final Inspection Strengths
75 percent discussed the work that was performed with the client
The inspectors discussed the work that was performed
90 percent examined all accessible rooms
Inspections were complete
85 percent conducted this test (85 percent of those performed correctly)
Blower door testing was usually conducted 52

53. WAP Final Inspection Opportunities
Zonal Pressure Test
33 percent performed the test (87 percent correctly)
IR Camera
44 percent used the camera (95 percent correctly)
Combustion Safety
31 percent performed this test on the gas stove (100 percent correctly)
Worst Case Draft Test
60 percent performed this test on the heating system (92 percent correctly) 53

54. WAP Final Inspection Findings
Opportunities
Strengths
Increased client education
Explain measures installed
Reinforce client action plan
Improved testing quality
Increased assessment of installation quality
Reduced use of auditor to conduct final inspection – additional perspective
Good job when following work order closely
Time saved when contractor attends
Information to clients – who to call if they have problems
Referrals to additional assistance programs 54

55. WAP Training Needs
Energy bill utilization – to assess needs and educate clients
Building science fundamentals
Critical thinking – assessing unique situations
Combustion safety testing
Zonal pressure testing
IR camera use
Ventilation assessment
Safe work practices – worker safety and lead safe work
Interviewing skills – how to understand the client’s needs
Client education 55

56. WAP Management Opportunities
Standards and Procedures
Policy Manuals
Forms and Checklists 56

57. WAP Equipment Needs GPS IR Camera
Boroscope and Fiber Optic Scope and Video
Personal Safety Equipment 57

58. Performance measurement58

59. Performance Measurement Steps
Program Statistics
Refine
Pilot Changes
Quality Control
Hold Contractors Accountable
Assess Inputs and Outputs
Assess Inspection Results
Assess Energy Savings 59

60. Performance Measurement 1- Develop Baseline Stats
Pre-treatment usage
Major measure installation rates
Energy savings measured through billing analysis 60

61. Performance Measurement 2- Refine Procedures
Review and refine procedures
Ensure documentation reflects expectations and best practices
Train contractors on weak areas
Review contractor understanding of program requirements 61

62. Performance Measurement 3- Pilot Program Changes
Incremental changes
Pilot test innovative strategies
Examples of innovations
Contractor compensation plan
Procedures for treating different types of homes
Low usage
High baseload usage
Health and safety problems
Home previously treated 62

63. Performance Measurement 4- Conduct Quality Control
Observe work in the field
Frequently inspect completed jobs
Verify procedures are correctly implemented
Verify comprehensive work
Review all aspects of the work
Audits, work scope, installation, third party inspection
Require contractors to fix unacceptable work
Ensure all parties agree to specifications and procedures 63

64. Performance Measurement 5- Require Contractor Accountability
Require contractors to return to home to fix problems
Provide training in problem areas
Set goals for contractor performance
Conduct additional observations and inspections
Remove contractors who do not improve 64

65. Performance Measurement 6- Assess Inputs and Outputs
Pre-treatment usage
Measure spending distribution
Outputs
Measure installation rates
Are they improving enough to lead to better results? 65

66. Performance Measurement 7- Assess Inspection Results
Review rates of:
Comprehensive installation
High quality installations
Missed opportunities
Poor work quality
Early indication of energy saving expectations. 66

67. Performance Measurement 8- Assess Energy Savings
Conduct billing analysis on an annual basis
Needed to ensure expected results
Develop procedures to more easily extract data
Reduced evaluation cost 67

68. Performance Measurement Repeat
Compare Results Over Time
Assess What is Working
Refine Program 68

69. summary 69

70. Achieving High Savings Lessons Learned
It is challenging to meet savings expectations
Target high usage customers
Ensure major measures are installed where opportunities exist
Maximize use of proven home performance techniques
Conduct performance measurement 70

71. Contact Information
Jackie Berger
President
appriseinc.org
David Carroll
Managing Director
APPRISE
32 Nassau Street
Suite 200
Princeton, NJ 08542 71