1. Cost-Effectiveness Testing & Non-Energy Impacts in Program Evaluation
Jackie Berger
Home Performance Conference
April 25, 2018

2. NON-ENERGY IMPACT LITERATURE2

3. Types of Residential NEIs
Benefit Level
Societal
Household
Benefits To
General Population
Ratepayer
Participant
Economic 
Environmental
Health & Safety
Affordability
Collection Costs
System Reliability
Indoor Air Quality
Noise
Water Usage
Maintenance 3

4. Benefits to Ratepayers and Participants
Household level benefits may result in benefits to society
Program participant health improvements
May lead to reductions in Medicaid or Medicare costs
Benefits accrue to taxpayers
Household level benefits may result in benefits to ratepayers
Program participant affordability improvements
May lead to reductions in the cost of bill payment subsidy
Benefits accrue to ratepayers 4

5. Challenges in the Literature
Not applicable to current study
Utilizes past estimates
Out of date
Robustness of approach
Sample size
Statistical significance
Estimate quality not assessed
Methodology
Assumptions
Limitations
Lack of documentation 5

6. Common Approach To NEI Estimation
Review past studies
Select particular benefits for inclusion
Take an average of past studies or subset of reviewed studies 6

7. Important Questions Methodology Design Assumptions Sample size
Statistical significance
Robustness
Causality
Response rate
Impact of attrition
Comparison group quality
Factors
Population served
Program design
Program delivery
Weather
Fuels
Baseline
Other services available 7

8. MEASURING NEIs 8

9. Measurement Approaches: H&S, Comfort, Home Condition
Home Measurements
Direct on-site assessments
H&S: Mold, moisture, asbestos, CO, radon
Comfort: Temperature, humidity
Health Records
Health conditions
Medical utilization
Police and Fire Department Data
Incidence of fires & accidents
Not usually available at unit level
Occupant Reports
Home conditions, comfort, health, safety
Prior to and following treatment 9

10. National WAP Evaluation Measurement Framework
Design: Pre and post-treatment surveys
Measured housing unit and health status for participants and comparison group
Prior to weatherization
At least one year after weatherization
Treatment Group
Households scheduled for a WAP audit.
Comparison Group
Treated by WAP one year prior to survey.
Baseline Survey
Conducted summer 2011
Collected client self-reports of housing unit conditions and health status
Follow-Up Survey
Conducted summer 2013
Used same survey questions
Analysis
Restricted to homes that were weatherized and still occupying weatherized home in 2013
Compared changes for Treatment Group to changes for Comparison Group 10

11. National WAP Evaluation Measurement Framework
Quasi-Experimental Analysis Design
Pre
Post
Change
Measured
Treatment
Before Audit
12-18 Months After Services
Pre-Post
Program Impact + Other Factors
Comparison
After Services
One Year Later
Post-Post
Other Factors
Treatment - Comparison
[Treatment Pre – Treatment Post] – [Comparison Post1 – Comparison Post2]
Program Impact 11

12. National WAP Evaluation Design Limitations
Earlier participants instead of later participants
Less room for impact from external factors
Potential understatement of impact of exogenous factors
Comparison Group
Differential attrition
Very cold climate is over-represented
Hot climate is under-represented
Longitudinal Response Rate
Income (continuing recovery from recession)
LIHEAP receipt (referral to WAP following LIHEAP application)
Omitted Variables
>9% changed asthma response
Additional call for clarification
19% different respondent and could not verify, 28% not reached
Measurement Issues 12

13. National WAP Evaluation Affordability Impacts
Treatment Group
Comparison Group
Net Change
Pre
Post
Change
Very hard/hard to pay energy bills
76%
49%
-26%***
58%
52%
-6%**
-20%***
Could not pay energy bills at least every few months
17%
14%
-3%
13%
12%
-1%
-2%
Used short-term, high interest loan to pay for energy bills
15% 9%
Could not buy food at least every few months to pay for energy bills
10%
-4%** 8%
Could not fill prescription at least every few months to pay for energy bills
-6%*** 6%
-3%** 13

14. National WAP Evaluation Bill Payment Impacts
Treatment Group
Comparison Group
Net Change
Pre
Post
Change
Received disconnect notice almost every month
16%
12%
-4%**
13%
11%
-2%
-1%
Electricity or gas disconnected 3% 2% 0%
Fuel ran out 9% 5%
-4%*** 4%
-3%* 14

15. National WAP Evaluation Home Condition Impacts
Treatment Group
Comparison Group
Net Change
Pre
Post
Change
Extremely/very infested with cockroaches, spiders, other insects 5% 2%
-3%** 0%
Somewhat infested with cockroaches, spiders, other insects
19%
12%
-7%***
13%
15% 3%
-10%***
Extremely/very infested with rats or mice
-2%*** 1%
Somewhat infested with rats or mice 8% 6%
-2% 15

16. National WAP Evaluation Home Condition Impacts
Treatment Group
Comparison Group
Net Change
Pre
Post
Change
Mildew odor or musty smell
29%
21%
-8%***
15%
16% 1%
-10%***
Mold
24%
19%
-5%**
17%
-1%
-4%
Always or often observed standing water 5% 4% 3% 0%
Sometimes observed standing water 9%
-6%*** 7% 16

17. National WAP Evaluation Home Condition Impacts
Treatment Group
Comparison Group
Net Change
Pre
Post
Change
A great deal of noise
28%
17%
-12%***
12% 0%
Drafty all the time 2%
-10%*** 4% 3%
-1%
-9%***
Indoor temperature is comfortable in the winter
58%
82%
23%***
79%
83%
4%*
20%***
Indoor temperature is comfortable in the summer
57%
71%
13%***
72%
74%
12%***
Never have unsafe or unhealthy indoor temperature
81%
93%
91% 17

18. National WAP Evaluation Health Impacts
 Respondent Health
Treatment Group
Comparison Group
Net Change
Pre
Post
Change
Has ever had asthma
19%
21%
2%*** 0%
Allergies
28%
31% 3%
7%***
-4%
The flu
18%
-3%
16%
-1%
-2%
Persistent cold symptoms more than 14 days
20%
14%
-6%***
17%
Sinusitis
37%
33%
-4%*
34% 18

19. National WAP Evaluation Safety Impacts
Treatment Group
Comparison Group
Net Change
Pre
Post
Change
Has working smoke detectors
90%
97%
7%***
96%
-1%
8%***
Has working CO monitors
44%
80%
36%***
76%
73%
-4%*
40%***
Fire dept. called to put out fire 1% 0%
-1%**
Food poisoned in home and went to see a medical professional
-1%*
Poisoned by CO, and went to see a medical professional
Lead poisoned and went to see a medical professional
Burned from water coming out of a faucet or showerhead in home 19

20. Ideas for Improved Measurement
Difficult to track receipt of services using client response
At state level could use program records
Program Records
Assess measures installed in each home
Compare outcomes for households that received particular measures
Weatherization Services
LIHEAP participation
Better assessment of affordability issues
Additional Questions
Need to keep testing health status questions
Improved Questions 20

21. Challenge Monetizing the Impact
40% increase in homes with working CO detectors
20% increase in homes with comfortable winter temperature
10% reduction in homes with mildew odor or musty smell
Examples of Measured Impacts
Develop estimates of potential harmful effects and the cost of those
Ask participants to value compared to the energy savings
How much is that worth $? 21

22. New Jersey Natural Gas SAVEGREEN PROGRAMS22

23. SAVEGREEN Programs Goals
Increase energy efficiency opportunities for customers
Promote and enhance the use of the NJCEP offerings
Raise awareness of the whole house approach to energy efficiency
Increase customer awareness of energy efficient appliances and weatherization measures
Increase NJ employment in energy efficiency and conservation 23

24. SAVEGREEN Programs Overview
NJ Clean Energy Program
SAVEGREEN Project
Jan-June
July-Dec
Furnace/Boiler Enhanced Rebate
Furnace
$400
$250
$900
$500
Boiler
$300
NJNG Audit Required
Furnace and Water Heater $6500 OBRP
Up to $6,500 OBRP
0% interest over 5 years
HPwES
Tier
Savings
Rebate
Up to $10,000 OBRP
0% interest over 10 years
SAVEGREEN pays NJCEP rebate when OBRP is used II
10%-19.99%
$2,000
III
20%-24.99%
$4,000
>25%
$5,000
C&I Direct Install
70% of retrofit costs up to $125,000
Up to $53,571 OBRP
0% interest over 2 years 24

25. Rebate Impact Natural Gas Savings
Treatment
Matched Comparison
Net Savings #
Therms
Savings %
Pre
Post
Raw
3,168
1,028
1,099
-71**
-6.9%
1,036
1,178
-143**
-13.8%
72**
7.0%
Degree Day
1,102
1,026
75**
6.8%
1,098
>-1
>-0.1%
76**
6.9%
PRISM
2,697
1,110
1,008
101**
9.1%
1,103
1,089
14**
1.3%
88**
7.9% 25

26. HPwES Impact Natural Gas Savings
Treatment
Matched Comparison
Net Savings #
Therms
Savings %
Pre
Post
Raw
1,156
1,023
899
125*
12.2%
1,027
1,144
-117**
-11.4%
242**
23.6%
Degree Day
1,048
849
199**
19.0%
1,050
1,072
-22**
-2.1%
221**
21.1%
PRISM
1,068
1,039
814
225**
21.6%
1,037
-11**
-1.0%
235**
22.6%
**Denotes significance at the 99 percent level. 26

27. Environmental Benefits Methodology
Energy Savings
Reduction in Natural Gas Usage
Avoided Emissions
Tons of CO2, SO2, Nox, PM 2.5, and VOC
Value Avoided Emissions
$ value using APEEP Model 27

28. Environmental Benefits Avoided Emissions
Rebate
HPwES OBRP
2013 Participants
6,700
1,720
Natural Gas Savings Per Participant (Therms) 76
221
Total Natural Gas Savings (Therms)
506,631
380,316
Total Natural Gas Savings (MMBtu)
50,663
38,032
Natural Gas Emission Rate
(Tons CO2-eq/1,000 MMBtu)
Marginal Value of Avoided Emissions
(2015 dollars /Ton) 1
CO2-eq2 62
$41.4
SO23
$111,573
NOx3
0.046
$23,023
PM 2.53
$468,563
VOC3
$44,180 28

29. Environmental Benefits Value of Avoided Emissions
Rebate
HPwES OBRP
Total Savings
Avoided Emissions
(tons)
Marginal Value of Avoided Emissions
($ per ton)
Savings
CO2-eq
3,139
$41.4
$129,945
2,356
$97,547
$227,492
SO2
0.015
$111,573
$1,654
0.011
$1,242
$2,896
NOx
2.32
$23,023
$53,485
1.74
$40,150
$93,634
PM 2.5
0.047
$468,563
$22,002
0.035
$16,516
$38,518
VOC
0.136
$44,180
$6,005
0.102
$4,508
$10,513
Total
$213,091
$159,963
$373,054
Monetary values are given in 2015 dollars
Avoided emissions for CO2-eq are in metric tons. Avoided emissions for all other air pollutants are in short tons 29

30. Environmental Benefits Value of Avoided Emissions
Time Period
Rebate
HPwES OBRP
Total Benefit
Participants
6,700
1,720
2014 Total
$213,091
$159,963
$373,054
2014 per Participant
$32
$93
Lifetime
$1,909,622
$2,543,862
$4,453,491
Lifetime per Participant
$285
$1,479
Monetary values in 2015 dollars.
Lifetime benefits measured over 15-year measure life.
Avoided emissions for CO2-eq are in metric tons. Avoided emissions for all other air pollutants are in short tons 30

31. Economic Benefits Theory
SAVEGREEN spending replaces retail spending
Charge on energy bill would have been spent on retail goods
Retail spending replaces natural gas spending
Energy savings from program spent on retail goods
Increase in economic activity because replacement spending has
Higher labor intensity
Greater percentage spent in NJ 31

32. Economic Benefits Methodology
Regional Input-Output Modeling System (RIMS)
Produced by Bureau of Economic Analysis (BEA)
Output Change = Expenditures * (Output Multiplier with Program – Output Multiplier Without Program)
Employment Change = (1/$1,000,000) * Expenditures * (Employment Multiplier with Program – Employment Multiplier Without Program) 32

33. Economic Benefits Sources of Economic Impact
NJNG Administrative Spending
NJNG and NJCEP Incentives
Customer Net Costs
Customer Total Savings
Estimates based on 2013 expenditures and participants’ savings. 33

34. Economic Benefits Output Multipliers
Source of Economic Impact
Output Multiplier
With Program
Output Multiplier Without Program
Output Multiplier Increase
Sector
Multiplier
NJNG Admin Spending
Labor: Admin, Prog Dev.
Office admin
1.9212
Retail trade
1.7112
0.2100
General: Sales, Marketing
Business supply
1.7661
0.0549
Labor: Rebates, Inspect, QC
Prof, scientific, tech
1.9241
0.2129
NJNG & NJCEP Incentives
NJNG HPwES Incentives
Services to buildings/dwellings
1.7845
0.0733
NJNG Enhanced Rebates
NJCEP Rebates
Customer Net Costs
HPwES Net Costs
Rebate Net Costs
Customer Total Savings
Natural Gas Savings
Natural gas
1.2638
0.4474 34

35. Economic Benefits Employment Multipliers
Source of Economic Impact
Employment Multiplier
With Program
Employ Multiplier Without Program
Employ Multiplier Increase
Sector
Multiplier
NJNG Admin Spending
Labor: Admin, Prog Dev.
Office admin
Retail trade
General: Sales, Marketing
Business supply
Labor: Rebates, Inspect, QC
Prof, scientific, tech
NJNG & NJCEP Incentives
NJNG HPwES Incentives
Services to buildings/dwellings
22.062
5.3499
NJNG Enhanced Rebates
NJCEP Rebates
Customer Net Costs
HPwES Net Costs
Rebate Net Costs
Customer Total Savings
Natural Gas Savings
Natural gas
2.7693 35

36. Economic Benefits Output Impact
Source of Economic Impact
Base Amount
Output Multiplier
Economic Impact ($)
With Program
Without Program
Change
NJNG Admin Spending
Labor: Admin, Prog Dev.
$251,346
1.9212
1.7112
0.2100
$52,783
General: Sales, Marketing
$1,941,029
1.7661
0.0549
$106,562
Labor: Rebates, Inspect, QC
$1,224,056
1.9241
0.2129
$260,602
NJNG & NJCEP Incentives
NJNG HPwES Incentives
$23,694,720
1.7845
0.0733
$1,736,823
NJNG Enhanced Rebates
$4,804,500
$352,170
NJCEP Rebates
$2,178,460
$159,681
Customer Net Costs
HPwES Net Costs
$1,568,640
$114,981
Rebate Net Costs
$35,030,970
$2,567,770
Customer Total Savings
Natural Gas Savings
$10,086,713
1.2638
0.4474
$4,512,795
Total Economic Impact
$9,864,167 36

37. Economic Benefits Employment Impact
Source of
Employment Impact
Base Amount
Employment Multiplier
Economic Impact
(Job-Years)
With Program
Without Program
Change
NJNG Admin Spending
Labor: Admin, Prog Dev.
$251,346 -1
General: Sales, Marketing
$1,941,029
Labor: Rebates, Inspect, QC
$1,224,056 -4
NJNG & NJCEP Incentives
NJNG HPwES Incentives
$23,694,720
22.062
5.3499
127
NJNG Enhanced Rebates
$4,804,500 26
NJCEP Rebates
$2,178,460 12
Customer Net Costs
HPwES Net Costs
$1,568,640 8
Rebate Net Costs
$35,030,970
187
Customer Total Savings
Natural Gas Savings
$10,086,713
2.7693
141
Total Employment Impact
495 37

38. Economic Benefits Output and Employment
Type of Impact
Total Lifetime Impact
Per Participant Lifetime Impact
Output ($)
$9,864,167
$1,172
Employment
(job-years)
495
0.06 38

39. Health & Safety Benefits Methodology
EnergySavvy Data
Available for 2014 participants
Issues identified during NJNG Audits for Enhanced Rebates and $6500 OBRP
Percent who address issues is not known 39

40. Health & Safety Issues Identified
Homes With Issue
Rebate
$6,500 OBRP # %
Improper Dryer Venting
894
23% 61
25%
Improper Bath Venting
724
19% 62
Did Not Pass Oven Inspection
189 5% 6 2%
Moisture Issues
132 3% 10 4%
Gas Piping Leaks 90 4
Did Not Pass Depressurization Worst Case Test 76 --
Did Not Pass Water Heater Draft (Worst Case) 52 1%
Gas Piping Leaks at Dryer 28 0%
Did Not Pass Water Heater Draft (Natural) 15
<1%
Asbestos Issues
CO ≥100ppm by Water Heater 8
Structure Issues 7
CO ≥100ppm in CAZ 40

41. Health & Safety Benefits Participants with Any Issue
Customer Has Any Issue
Rebate
$6,500 OBRP # %
Yes
1,616
42% 99
40% No
1,969
51%
147
60%
No Health & Safety Data
283 7% 0%
Total
3,868
100%
246 41

42. USING NEI ESTIMTES IN COST-BENEFIT ANALYSIS42

43. Comparing Costs And Benefits
Analysis 1
Energy Costs
(exclude H&S costs)
Energy Impacts
Analysis 2
All Costs
Analysis 3
All Impacts
Need to track costs for energy and health and safety investments separately. 43

44. SUMMARY 44

45. Non-Energy Impacts Methodologies have been developed to measure NEIs
Current literature on NEIs has many challenges
Additional research is needed
Difficult to apply findings from previous studies
Factors specific to programs, jurisdictions, participants, and implementation can impact the NEIs
Additional challenges relate to valuing benefits relating to health, comfort, and safety 45

46. Contact Jackie Berger APPRISE 32 Nassau Street, Suite 200
Princeton, NJ 08540 46