1. Comprehensive Usage Reduction Program Evaluation Affordable Comfort 2007 Jacqueline Berger

2. Evaluation Objectives 1.Determine the cost-effectiveness of WRAP 2.Develop standard questions so PPL can measure the same criteria in future evaluations 3.Comply with the PUC Order 2

3. Evaluation Questions 1.What are the goals and how are they met? 2.What are the admin costs? How can they be lowered? 3.How effective is program recruitment? 4.Is the audit mechanism effective? 5.Is the list of program measures comprehensive? 3

4. Evaluation Questions 6.Is the education process cost-efficient and effective? 7.What is the level of post-inspection and is it appropriate? 8.Does PPL coordinate WRAP with other weatherization programs? 9.What are the energy savings? 4

5. Evaluation Questions 10.What is the cost-effectiveness of the various agencies? 11.Does PPL provide adequate support and training for contractors? 5

6. Evaluation Design 1.Background Research 2.Review of Specifications and Procedures 3.Contractor Survey 4.Baseload Observations 5.Full Cost Observations and Inspections 6

7. Evaluation Design 6.Customer Survey 7.Usage Impacts 8.Payment Impacts 7

8. Background Research Goal – develop a complete understanding of the WRAP procedures and implementation. Activities –Interview WRAP managers and staff –Review program documentation –Review program statistics Outputs –Understanding for research foundation –Documentation 8

9. Background Research Key Findings –PPL has five managers who oversee WRAP and other low-income programs in their geographical area. Advantage: ability to focus on needs in a particular geographic area Disadvantage: many responsibilities – difficult to oversee work of contractors –PPL spends much effort to come within 4% of expenditure goal (based on PUC requirement). 9

10. Background Research Key Findings –Affordability customers are prioritized. –Otherwise, jobs sent to contractors on a first come first served basis. –Program coordination barriers Long waiting lists for WAP Long waiting lists & requirements for gas programs Customers who use gas and electric may not have high enough usage to qualify for either program 10

11. Background Research Recommendations –Prioritize customers based on usage. –Track program coordination and provide incentives for contractors to coordinate services with other programs. –Continue to introduce technological improvements, such as the web-based measure reporting form. 11

12. Review of Specifications and Procedures Goal – Determine potential effectiveness of measure selection, measure installation, and energy education Activities – Assess procedures and forms: –Education specifications –Education forms –Written technical procedures and manuals –Measure installation rates 12

13. Review of Specifications and Procedures Outputs –Recommendations for modifications to: Education procedures Education forms Measure selection guidelines Procedures manual 13

14. Review of Specifications and Procedures Education Findings –All participants receive at least one on-site education visit –Follow-up education is provided at the time of the inspection or by phone –Remedial education provided to customers whose usage increases by at least 10% six months after service delivery 14

15. Review of Specifications and Procedures Education Recommendations –Education should be provided at the time of the audit. The homeowner should be present. –Customer profile should collect information on potentially large opportunities for saving – use of dehumidifiers, use of second refrigerators, lights/appliances left on at all times. –Action form should prioritize actions by the potential for energy saving in the individual home. Should list top 3-5 actions with estimated $ savings. 15

16. Review of Specifications and Procedures Technical Findings and Recommendations* –Review cost-effectiveness calculations in audit decision trees to reflect current estimates of costs and savings. Refrigerators CFLs –Water heater wraps and pipe insulation may be more cost effective than water heater replacement. 16 * Blasnik & Associates.

17. Review of Specifications and Procedures Technical Findings and Recommendations –Duct sealing in basements should be focused on safety and comfort. –Blower door guided air sealing: investigate why only done in 60% of full cost jobs. –WRAP standards and field guide: more concise program field guide with separate specifications for specific areas may be useful. 17

18. Contractor Survey Goal - Assess contractor compliance with program procedures, and assess problems in program administration. Activities –Develop contractor survey instrument –Determine survey sample –Send survey to contractors –Review completed surveys and contact respondents for additional information/clarifications 18

19. Contractor Survey Outputs - Understanding of: Contractor background and experience Support and training provided to contractors Usefulness of program forms Program implementation procedures Joint service delivery with WAP Health and safety problems found in homes Contractors’ quality control Inspection issues 19

20. Contractor Survey Outputs - Recommendations related to: –Program procedures –Contractor training and support –Inspection procedures 20

21. Contractor Survey 21 # of Contractors Complete16 No Response2

22. Contractor Survey 22 Staff Training# of Contractors Observing other service delivery staff12 Being observed while delivering services 10 Classroom training8 Affordable Comfort6 PA WX Classes3 WX Training Center Classes2 Testing staff1

23. Contractor Survey 23 Staff Assessment# of Contractors Field observation of WRAP jobs11 Practical exam7 Professional certification7 Written exam5 Inspection of WRAP jobs5 Pass Wx Training Center class1 Web training1

24. Contractor Survey 24 PPL Training Ratings # of Contractors Who Provided Each Rating Mean Rating 1-234-5 Quality14103.7 Focus1493.6 Level1493.8 Amount2753.2 Overall13103.6

25. Contractor Survey 25 Joint Delivery of WRAP Services # Who do Joint Delivery % of Jobs MinMaxMean PA WX80%100%16% Gas Utility40%100%7% County WX20%10%1%

26. Contractor Survey 26 Provide Evening and Weekend WRAP Services EveningsWeekends Yes87 No78 No Answer11

27. Contractor Survey 27 Baseload Audit Procedures AlwaysSometimesNever Describe WRAP 1010 Discuss bill with customer 1100 Discuss H&S with customer 1010 Conduct walkthrough with customer 821 Provide measure saving estimate 650 Provide actions savings estimate 560

28. Contractor Survey 28 Quality Control Methods # of Contractors% Of Jobs Review data collection forms 1175% Contact customers by telephone 1040% On-Site Inspection1241% Observation1232%

29. Contractor Survey 29 PPL Inspectors % of Inspections Resolution Inspector Fixed Action Sheet None Needed Invoicing mistake 7%17%47%36% Insulation 6%0%83%17% Dryer Venting 15%39%61%0% Education 11%100%0%

30. Contractor Survey 30 PPL Program Ratings # of Contractors Who Provided Each Rating Mean Rating 1-234-5 WRAP Specs03124.1 PPL Communication 03124.3 Invoicing04114.1 Overall03123.9

31. Contractor Survey Recommendations –Revisit audit forms and determine whether they can be consolidated. –Require home walkthrough on all jobs. –Formalize a process to respond to action sheets. 31

32. Baseload Observations Goals - Understand how well contractors address opportunities for baseload usage reduction and whether education is effectively provided. Activities –Sample design and selection –Observation protocols –Conduct observations –Review findings and synthesis 32

33. Baseload Observations Outputs - Recommendations for: –Additional contractor training –Additional quality control 33

34. Baseload Observations 34 Visit Introduction # Of Observations Comments YesNo Visit expected100 WRAP explained64 Usage reviewed462 did later in the visit H&S discussed462 did later in the visit Comfort discussed372 did later in the visit

35. Baseload Observations 35 Home Walkthrough # Comments YesNo Inspected every room64 1 auditor did not do the walkthrough Systematic inspection64 Discussed electric uses82 Estimate costs of uses464 did later in visit Reinforced costs later82

36. Baseload Observations 36 Home Walkthrough # Comments YesNo Discussed actions731 later in visit Estimated savings553 later in visit Discussed willingness to take actions 731 later in visit Obtained customer commitment 642 later in visit

37. Baseload Observations 37 Home Walkthrough Minutes Comments MinMaxAvg. Length of walkthrough137941 2 without not included in average Part on education 134526

38. Baseload Observations 38 Refrigerator Replacement # Comments YesNoNA Monitored refrigerator622 2 new, 1 couldn’t be moved, 1 broken Replacement 451 Explored 2-for-1 127

39. Baseload Observations 39 Compact Fluorescent Light Bulbs # YesNoNA Discussed all inside lights46 Discussed all outside lights 73 Installed CFLs 802 Asked if satisfied with lighting 802 Left extra bulbs for customer 010

40. Baseload Observations 40 Energy Education # YesNoNA Energy education visit10 Engaged customer 91 Reviewed measures 721 Analyzed electric bill 82 Discussed appliances 82

41. Baseload Observations 41 Visit Summary Minutes MinMaxMean Length of visit70180119 Rating ExcellentGoodFair Not Acceptable Overall rating 4222

42. Baseload Observations Recommendations –Review WRAP requirements and expectations with contractors. –Provide education to contractors on the importance of 2-for-1 swaps in refrigerator replacement. –Conduct observation of baseload service delivery. 42

43. Full Cost Observations and Inspections Goals - To understand: –How well contractors address opportunities for electric usage reduction –Whether correct measures were selected –Extent to which energy education is provided. Activities –Sample design and selection –Observation protocols –Review WRAP technical protocols –Conduction observations –Review findings and synthesis 43

44. Full Cost Observations and Inspections Outputs – Recommendations for: –Procedures –Training 44

45. Full Cost Observations and Inspections 45 Visit Introduction # Of Observations Comments YesNo Visit expected60 One of the customers was not present for most of the visit. WRAP explained42 Usage reviewed42 H&S discussed60 Comfort discussed60

46. Full Cost Observations and Inspections 46 Diagnostic Testing # Of Observations Comments YesNoNA Blower door testing51 One of the observations was a mobile home. Pressure diagnostics141

47. Full Cost Observations and Inspections 47 Missed Opportunities 1House still connected to garage and attic. Solar hot water. 2 Remove dropped sealing and install sheetrock for an air barrier. 3None. 4 Insulate entire attic. Would have been difficult, but possible.

48. Full Cost Observations and Inspections 48 Overview Very Good GoodFairComments Data collection accuracy 112 Unable to duplicate results in one. Not all forms used in another. Measure selection 13 Attic sealed shut in one so work could not be inspected. Appropriateness of measures 22 One hardship case and contractor told to do everything to reduce usage.

49. Full Cost Observations and Inspections 49 Overview ExceptionalGoodSatisfactory Comments Effort 31 Hard to assess one home because customer had moved and home unoccupied. Evaluator’s tests did not match up in another case. Quality 121 Appropriateness 31 Overall rating 13

50. Full Cost Observations and Inspections Comments –They were on the right track. They were not afraid of working hard. –What he did was correct, but he should have done more air sealing. –He looked at all the right things and asked the homeowner to clarify things that were not obvious. 50

51. Full Cost Observations and Inspections Comments –He inspected the entire home and included the homeowner in the audit. –He did not fill out all the forms and did not do testing. –He was very friendly and carefully explained everything to the customer. –He did everything according to the protocol. 51

52. Full Cost Observations and Inspections Recommendations –Develop one set of forms that is required for all jobs. –Provide instructions for each form on the back of the form. –All applicable diagnostic tests should be required at the audit visit. 52

53. Customer Survey Goals - Understand the participants’ –Demographics –Reasons for participation –Understanding of the program –Actions taken to save electricity –Bill payment difficulties –Perceived program impact on electric bills –Perceived program impact on comfort –Satisfaction with the program 53

54. Customer Survey Activities –Develop the survey instrument –Test the survey instrument –Develop a sample plan –Select the survey sample –Conduct surveys –Analyze the data 54

55. Customer Survey Outputs - Recommendations for –Program procedures –Customer education –Contractor training 55

56. Customer Survey 56 Demographics Does anyone in your home have a medical condition that requires the use of additional electricity? Yes22% In the past 12 months, was any member of your household unemployed and looking for work? Yes34%

57. Customer Survey 57 Why did you want to receive WRAP? Reduce electric bills 64% Improve comfort of the home20% Reduce electric usage9% Difficult financial situation6% Told to enroll3% Received new appliances2%

58. Customer Survey 58 What are the benefits of WRAP? (Unprompted) Energy education 36% Lower electric bills35% Lower electric usage18% Safer or more comfortable home11% New appliances9% Improvements to home6%

59. Customer Survey 59 What are the benefits of WRAP? (Prompted) Lower electric bills88% Lower electric usage91% Energy education95% New appliances86% Safer or more comfortable home92%

60. Customer Survey 60 What is the most important benefit of WRAP? Lower electric bills27% Energy education19% Safer or more comfortable home11% Lower electric usage10% New appliances10%

61. Customer Survey 61 How difficult is it for you to pay your PPL bill? Very difficult23% Somewhat difficult36% Not too difficult25% Not at all difficult13%

62. Customer Survey 62 Were you home for the service provider’s visit? Home for visit93% Home for entire visit85%

63. Customer Survey 63 Actions to Save Electricity Provider recommended actions83% Provider gave savings estimates63% Provider gave written plan of actions64% Provider left information80%

64. Customer Survey 64 Reduced Usage Of Lights72% Dishwasher72% Electric heat68% Electric hot water heater62% Air conditioner56% Electric dryer55% Dehumidifier39%

65. Customer Survey 65 Actions Taken (Unprompted) Use CFLs37% Turn off lights not in use18% Keep doors/windows closed11% Purchase energy efficient appliances8% Use cold water for clothes washing6% Add insulation, air sealing, other measures6% Turn off television2% Turn off computer2% Wash only full loads2% Reduce hot water usage2%

66. Customer Survey Other Actions –Clean/replace ac filters –Heat fewer rooms –Use less electric heat –Stop using an appliance –Reduce length of showers/ reduce baths –Raise refrigerator temperature –Reduce use of dishwasher 66

67. Customer Survey 67 Change in Comfort WinterSummer Improved40%32% Worsened1% No Change58%67%

68. Customer Survey 68 Program Satisfaction Very satisfied71% Somewhat satisfied22% Somewhat dissatisfied4% Very dissatisfied2%

69. Usage Impacts 1 Goal - estimate the actual impact of the program on customer electric usage Activities –Obtain program measure data –Obtain electric usage data –Obtain weather data –Weather normalize the data –Compare change for treatment and comparison groups 69 Usage impact analysis done by Blasnik and Associates.

70. Usage Impacts Outputs –Estimate of the impact of the program on energy usage –Estimate of the impact of particular program measures –Estimate of the effectiveness of different providers –Estimate of cost effectiveness of service delivery 70

71. Usage Impacts 71 Major Measure Installation Rates Baseload Low Cost Full Cost Refrigerator replacement50%62%44% Water heater replacement1%59%10% Air conditioner replacement18%36%13% Attic insulation0%1%26% Other insulation0%1%8% Air sealing with blower door0%1%15% HVAC work0% 11%

72. Usage Impacts 72 Electric Savings Results UsageSavings #PrePostGrossNetNet % Baseload6599,6619,3942678368.7% Low Cost11210,86910,6332365004.6% Full Cost1,01917,91217,1297831,7679.9%

73. Usage Impacts 73 Measure Savings Results Baseload Program SavingsCost$/kWh Refrigerator replacement777$662$0.85 Air conditioner replacement172$546$3.17

74. Usage Impacts 74 Measure Savings Results Full Cost Program SavingsCost$/kWh Refrigerator replacement532$606$1.14 Attic insulation766$882$1.15 Other insulation887$999$1.13 Windows & doors457$1,206$2.64 Air sealing w/Blower door378$288$0.76

75. Usage Impacts 75 A (75) B (102) C (27) D (108) E (22) F (56) G (95) H(107) I (36) -100001000200030004000 Net Savings (mean ± 90% confidence intervals) Net Savings by Provider Baseload Program

76. Usage Impacts 76 Net Savings by Provider Full Cost Program A (166) B (129) C (281) D(99) E (58) F (59) G (122) H (73) 01000200030004000 Net Savings (mean ± 90% confidence intervals)

77. Usage Impacts Recommendations –Reassess air conditioner replacement targeting strategy and water heater replacement. –Refrigerator replacement, insulation, and blower door guided air sealing should be pursued and perhaps expanded. 77

78. Payment Impacts Goal - To estimate the impact of the program on customer bills and payments. Activities –Obtain customer bill and payment data –Add up bills and payments in the pre and post period –Compare change for treatment and comparison groups 78

79. Payment Impacts Outputs – Estimate of the impact of the program on bills and payments. 79

80. Payment Impacts 80 UsageSavings PrePostGrossNetNet % Total Bill$1,214$1,194-$21-$118-9.7% Total Payments$1,124$1,179$54-$58-5.2% Bill Coverage Rate93%100%8%12%12.9% There were 1,873 customers in the treatment group and 1,228 customers in the comparison group.

81. Conclusions Several types of evaluation activities Each research activity brings a different set of information Do they tell the same story? Synthesis Recommendations that can be implemented 81