Why Data Matters Building and Sustaining a Business Case NEAUC Conference June 18, 2014

Presentation Evaluation versus Performance Measurement Types of Performance Measures –Inputs –Outputs –Outcomes –Impacts Data Sources Process 2

EVALUATION VERSUS PERFORMANCE MEASUREMENT 3

Comparison 4 Evaluation PeriodicIn-DepthExternal Performance Measurement OngoingDevelopmentalInternal

Evaluation What are the goals? How is my program performing compared to goals or expectations? How does it compare to other programs? How can the program improve? 5

Performance Measurement How can I measure? –My organization’s efforts and inputs –Outcomes of those efforts –How we impacted clients –How we impacted utility How has this changed over time? How does my organization compare? What are higher performers doing? Are those actions related to results? Can I implement those actions? 6

TYPES OF PERFORMANCE MEASURES 7

Types of Measures 8 InputsOutputsOutcomesImpacts More powerful informationMore difficult to obtain data

Inputs Staff hours Equipment Travel costs Supplies 9

Input Example Delivery Costs 10 Agency Job Cost Year # of Jobs Mean Cost Percentiles P10P25P50P75P $2,744$764$1,297$2,423$4,105$5, $3,436$605$2,004$3,435$4,958$5, $4,324$904$2,622$4,310$6,039$7,068 Total1,661$3,413$763$1,679$3,314$4,912$6,070

Input Example Itemized Costs 11 UGIElectric Field Support$521,520$764,932$119,334$252,962 Administration$114,529$167,622$37,482$35,671 Inspections$19,840$12,830$7,385$11,390 No Measures$4,768$4,075$1,950$1,600 Rehab Pilot$94,774$68,742$0 Conservation Pilot$0$50,000$0 TOTAL$755,161$1,068,201$166,151$301,623

Outputs Number of customers applied Number of customers enrolled Service delivered Participant characteristics Services coordinated with other programs 12

Output Example Customers Served 13 Number Referrals51,868 Defaulted10,166 Cancelled17,006 Graduates1,011 Moved8,480 Re-certifications8,512 New Enrollments19,401

Output Example Customer Characteristics 14 Federal Poverty Level NumberPercentNumberPercentNumberPercent 0-50%89225%86028%73926% %1,25236%1,07935%90632% %83224%72523%63323% Unknown53315%45014%51719% Total3,509100%3,114100%2,795100%

Output Example Customer Characteristics NumberPercentNumberPercentNumberPercent Child2,13362%1,92362%1,74964% Elderly36911%31710%32512%

Output Example Service Type 16 EnrollmentsRecertifications Phone Mail14,7484,663 Office1, Home10-- Total16,6305,219

Output Example Payment Type 17 Payment TypeEnrollments Minimum Payment11% Percent of Bill Payment54% Percent of Income Payment16% Annualized Average Payment7% Agency Selected Payment12%

Output Example Grant Type 18 Number of Grants PercentPaymentsPercent Electric3,35087%$783,57685% Oil38010%$106,46911% Natural Gas722%$19,4182% Propane431%$9,9111% Kerosene211%$5,9821% Coal2<1%$7380% Total3,868100%$926,094100%

Output Example Measures Installed 19 Measures #%#% Audit74398%35295% LIURP Intake73997%35295% Repairs67589%31886% Blower Door Test58777%30382% Air Leakage Reduction53370%27675% Door Work52669%28076% Air Sealing49965%28276% Efficiency Testing44659%20556% HVAC Clean & Tune39552%21959% Pipe Insulation40453%21358% Energy Education34545%26271% Window Repair/Replace37850%18550% Measures #%#% Water Measure32743%19753% Attic Insulation32042%20455% Lighting24833%14840% Mileage33043%21759% Dryer Work23130%16545% Ventilation16021%11331% Wall Insulation15921%9526% Thermostat10013%10027% Insulation9613%4312% Header Insulation14018%6818% Attic Preparation8311%4412% Refrigerator567%205%

Outcomes Reduction in bill Reduction in energy burden 20

Outcome Example Burden Reduction National LIHEAP Survey Pre- LIHEAP Post- LIHEAP ≤ 5%13%37% 6% - 10%32%29% 11% - 15%19%17% 16% - 20%15%8% 21% - 25%7%4% >25%14%7%

Outcome Example Burden Reduction National LIHEAP Survey AllSeniorDisabled Child Under 18 Non- Vulnerable Pre-LIHEAP16%14%17%16% Post-LIHEAP11%9%11%12%10%

Outcome Example Heat Restoration National LIHEAP Survey Restored Heat Due to Electric/ Gas Shut Off Restored Heat Due to Fuel Running Out Restored Heat Due to Broken Equipment 61%69%58%

Impacts Bill payment coverage rates increased Service terminations declined Energy usage declined 24

Impact Example Customer Survey 25 Has the winter temperature improved, worsened, or stayed the same since receiving weatherization services? Change in Winter TemperatureNumber of Respondents Improved19 Worsened1 Stayed the Same0 Did Not Notice a Difference8 Total28

Impact Example Arrearage Impacts 26 Pilot Payment Program Treatment Group Net Change #PrePost Gross Change ALL GRAD561$276$251-$26**-$119** 1 – Graduated Discount213$257$254-$3-$96** 2 – Discount &Audit170$332$289-$43*-$136** 3 – Discount & Counseling178$246$210-$36**-$129**

Impact Example Service Terminations 27 Service Terminations Utility 1Utility 2Utility 3Utility 4 Pre5%14%15%4% Post7%8%4%2% Gross Change1%-6%**-10%**-3%** Net Change-1%-6%**-10%**0%

Impact Example Usage Impacts 28 Obs. UsageSavingsNet Savings PrePost kWh/ ccf % % Electric Baseload 62 14,10813, %9156.5% Electric Heating 4327,80826,3431,465 * 5.3%1,2954.7% Gas Heating 4711,5881, ** 11.2%156 ** 9.8%

Impact Example Major Measures 29 # Major Measures Obs. Usage (ccf)Savings PrePostccf% 0581,5751, ** 10.6% 11981,5551,47580 ** 5.1% 22541,5431, ** 10.5% 31671,5941, ** 14.2% 4 or more531,6101, ** 16.8% All7301,5651, ** 10.4%

Impact Example Cost-Effectiveness 30 ccf Saved Mean Cost Payback Years $/ccf Saved Lifetime Air Sealing/ Leakage Reduction 7.68$50913$1.25 Attic Insulation4.68$91521$2.06 Wall Insulation8.27$1,52012$1.16 Misc. Insulation6.81$1,19015$1.41 Window Work51$83416$1.58 Thermostat57$1422$0.24

DATA SOURCES 31

Agency Records Most accessible Should be put in a database May not be needed if good program database Data –Customers served –Characteristics – income, poverty level, elderly, children –Services provided 32

Public Use Data Available for free download Characterize eligible population in service territory Programming skills needed Data –Number eligible –Geography –Characteristics – income, poverty level, elderly, children, language –Energy costs 33

Customer Survey Real time feedback Requires staff time Document methodology Data –Customer characteristics –Satisfaction –Self-reported impacts 34

Program Database Program manager – state or utility Canned reports Queries Data –Customers served –Characteristics – income, poverty level, elderly, children –Services provided 35

Utility Data Difficult to obtain Easier for utility managed program Requires software and programming skills Data –Customer type – heating, water heating, baseload –Energy usage –Energy bills –Customer payments –Energy assistance 36

PROCESS 37

Process Start with available data Identify performance measures Determine additional data sources Collect additional data Develop additional performance measures 38

SUMMARY 39

Summary Performance measurement overlaps with evaluation Start with program goals Work with available data Identify ways to enhance data Measure performance over time Identify areas for improvement Impact measures require more data and analysis 40

Contact Jackie Berger, Ph.D. President and Co-Founder APPRISE 32 Nassau Street, Suite 200 Princeton, NJ