1. Quantitative On-Site Evaluation of Energy Efficiency Programs Jackie Berger August 16, 2010

2. Motivation Quotes from summary of on-site visit: The core finding was that the auditors and crews engagement with the client and with one another increased as a function of their familiarity with their tools and their tasks. To some degree there was more engagement within the crew when the logistical challenges were greater, causing crew members to turn to one another to work out solutions. 2

3. Motivation Quotes from summary of on-site visit: The agency routinely uses good-quality silicone caulk for air sealing interior and exterior cracks, particularly around windows. While this is probably satisfactory, an easier-to-apply caulk with a long lifetime and substantially lower cost is in widespread use in weatherization and other retrofit work. Its an acrylic latex with silicone. After dispensing (using the pull or push method), it can be easily worked with a wet sponge, rag, or finger and has a lifetime rated at 35 to 50 years… 3

4. Motivation Quotes from summary of on-site visit: … It is available in a number of standard colors including white and clear. (The clear comes out of the tube white and gradually turns clear during curing, a process that takes at most a day unless humidity levels are particularly high.) It cleans easily with soap and water and costs about $2.25 per tube in case quantities at Home Depot and other home supply stores... 4

5. Motivation Quotes from summary of on-site visit: … Finally, since installing air sealing measure on the inside of the conditioned envelope is usually more effective in limiting convective losses than is installation on the exterior, use of an acrylic latex product with silicone usually results in a more aesthetically appealing result than does pure silicone. When fully cured, it also takes paint well (Figures 22 and 23). 5

6. Motivation 6 Figures 22 and 23. All caulks have their place and matching the right one to the right job is part of the weatherization craft.

7. Motivation Goals for on-site observation and inspections –Collect the targeted information. –Summarize findings over all visits. –Understand how findings relate to the program overall. –Make recommendations for program improvement. 7

8. Session Outline Introduction Process Evaluation Research On-Site Evaluation Sample Selection Data Collection Instruments Findings from On-Site Evaluation Summary 8

9. INTRODUCTION 9

10. Impact evaluation –How much energy is saved? –Is the program cost-effective? –Which measures are cost-effective? Process evaluation –Why succeeding or not meeting goals? –How can the program be improved? Documenting service delivery –Anecdotal – cannot be generalized. –Quantitative – how pervasive are the issues. 10

11. PROCESS EVALUATION RESEARCH 11

12. Research Tasks Background research purpose –Understand program policies and procedures. –Identify potential improvements. Background research activities –Review program documentation. –Conduct in-depth interviews with program designers and managers. –Review program performance statistics. 12

13. Research Tasks Contractor survey purpose –Understanding of program –Program implementation issues –Barriers with program procedures Contractor survey activities –Review delivery statistics. –Review contractor characteristics. –Select sample. –Conduct survey and analyze data. 13

14. Research Tasks Client survey purpose –Program experiences –Energy education delivered –Energy education impact –Safety and comfort impact –Satisfaction 14

15. Research Tasks Client survey activities –Identify key segments. –Select sample. –Conduct survey and analyze data. 15

16. ON-SITE EVALUATION 16

17. On-Site Importance Were protocols followed? How well did they work? Were important savings opportunities missed? 17

18. On-Site Importance Protocols – provider compliance –Program information, diagnostic tests, measure installation, client education. Protocols – applicability –How well protocols work, consistent application. Use of equipment –Tools available, adequately maintained and used correctly. 18

19. On-Site Importance Provider adaptability –Protocols properly adapted to individual circumstances. Comprehensiveness –Extent to which cost-effective procedures are addressed, potential modification of procedures. Quality of Work –Conformance to program standards, respect for client home, safety. 19

20. On-Site Importance Client Education –Focus on high potential areas, ability to motivate clients. Client Interaction –Explanation of program and work, use of client feedback as information input. 20

21. SAMPLE SELECTION 21

22. Sample Selection Challenges Limited sample size. Many contractor/job/home/client characteristics to consider. Difficult to draw inferences about program implementation. 22

23. Sample Selection Methods Probability sampling –Random selection techniques are used. –Each job has a known probability of selection. –Results can be weighted to represent the program. –Confidence intervals can be developed. Purposive sampling –Random selection techniques are not used. –But can allow for assessment of program performance. 23

24. Sample Selection Utility Program Evaluation 5 service delivery areas, as specified by utility. Baseload jobs – customers without electric heat or hot water. Full cost jobs – customers with installed electric heat and >3,600 kwh seasonal heating or cooling usage. 16 contractors provide services. 24

25. Sample Selection Utility Program Evaluation One dominant contractor in each service area for each job type. Budget only allowed for observation/inspection of one contractor in each service area for each job type. Dominant contractor for each area was chosen. 25

26. Sample Selection Utility Program Evaluation 26 RegionContractor # Base load # Full Cost RegionContractor # Base load # Full Cost 1A62723I135102 1B0233/4J46128 1C11073/4K1531 2D811154L3827 2E36584M2106 2F441024N630 2G5305O4412 2H28465P570 Previous Year Service Delivery

27. Sample Selection Utility Program Evaluation Selected contractors represent 60 percent of Baseload jobs and 68 percent of Full Cost jobs. Contractors use only 1-3 auditors. Observed auditors do 30 to 40 percent of the audit work. Findings represent significant percentage of job experiences. 27

28. Sample Selection National WAP Evaluation Select 20 local agencies in 20 different states 28 Stratification Factors Climate Region Crews or Contractors Urban/Rural/Suburban Agency Size Client Education Community Action Agency /Other Type Baseload Measures Training DOE Funding/ Substantial Other Resources QA Dominant Fuel Computerized Audit/Priority List

29. PROCEDURES AND FORMS 29

30. Procedures and Forms Development Understand –Program goals –Program design and implementation –Roles/responsibilities of providers Source of problems –Protocols –Division of responsibilities –Implementation 30

31. Procedures Utility Program Evaluation Baseload Observations, Full Cost Observations, Full Cost Inspections –Procedures: how to conduct visit, forms to complete, visit write-up, forms to collect from contractor. –Data Collection Form: home, electric usage, services provided. –Potential Big Users: lights and appliances that could lead to high baseload usage. 31

32. Baseload Procedures Utility Program Evaluation 1.During visit –Record auditor work and customer interactions. –Every 10 minutes record time and auditor actions. –Record if auditor addresses the Potential Big Users. 2.During/after visit –Complete Baseload Observation Form. –Complete Potential Big Users Form. 3.Immediately following visit –Debrief customer and auditor. 32

33. Baseload Procedures Utility Program Evaluation 4.Evening or next day –Complete customer debriefing. 5.Narrative –Chronologically document audit. –Estimate length of key parts of audit. –Assess interaction between auditor and client. 33

34. Baseload Procedures Utility Program Evaluation 6.Forms – obtain copies of all forms used during the visit. –Customer usage history –Program application –Core assessment form –Refrigerator data form –Water heater checklist –Customer profile –Your Electric Bill form 34

35. Forms Utility Program Evaluation Selected parts of baseload observation form 35 VISIT INTRODUCTION 1.Did the customer expect the visit?YESNO 2. Did the auditor describe the program to the customer?YESNO 3. Did the auditor review and explain the electric bill?YESNO 4. Did the auditor discuss whether there were any health and safety issues? YESNO 5. Did the auditor discuss whether there were any comfort issues? YESNO 6. Did the auditor discuss whether there were any problems with energy usage? YESNO

36. Forms Utility Program Evaluation Selected parts of baseload observation form 36 PRIORITY LIST – AIR CONDITIONERS 1.Did the auditor inspect all window air conditioners?YESNO 2. Did the auditor do sealing around the air conditioner?YESNO 3. Did the auditor talk about seasonal storage?YESNO 4. Did the auditor determine that the air conditioner had EER of <6 or was in poor condition? YESNO 5. If yes, did the auditor offer to replace air conditioner?YESNO 6. If yes, did customer accept AC replacement?YESNO 7. If no, why did the customer refuse

37. Forms Utility Program Evaluation Selected parts of full cost inspection form 37 THERMOSTATS (FILLED OUT FOR EACH ONE) LocationProgram Temp System controlledDay Heat ProgrammableNight Heat ConditionDay Cool AccuracyNight Cool Exceptional (Ex): No improvements identified Good (Gd): Minor improvements possible Satisfactory (Sat): Some improvements recommended Fair (Fr): Extensive improvements necessary Poor (Pr): Substandard in all respects Same as recorded Effort_________Quality _______Appropriateness ___________

38. Procedures WAP Evaluation Observe audit, measure installation, and final inspection. Observe job from start to finish in one home. Conceptual home – observe all aspects. 38

39. Challenges WAP Evaluation Observe up to 480 homes from 20 providers in 20 different states. WAP implemented differently in every state. –Audit procedures –Eligible measures –Type of education provided Develop general forms and procedures that collect detailed quantitative data. Assess work based on whose standards? 39

40. Forms WAP Evaluation Audit observation form Audit write-up assessment form Measure installation observation form Final inspection observation form Client debriefing Weatherization staff/contractor debriefing 40

41. Forms - WAP Evaluation Selected parts of audit observation form 41 Combustion Safety Tests Test done? In agency standards? Should have been done? Performed correctly to obtain needed info? Ambient CO in CAZ YNYNYNYN Ambient CO outside CAZ YNYNYNYN Gas/propane/ fuel oil leaks YNYNYNYN Spillage testing YNYNYNYN Heating system CO YNYNYNYN Water heating CO YNYNYNYN

42. Forms - WAP Evaluation Selected parts of audit observation form 42 Combustion Safety Tests Test done? In agency standards? Should have been done? Performed correctly to obtain needed info? Heating system draft YNYNYNYN Water heating draft YNYNYNYN CAZ – combustion air YNYNYNYN Range CO YNYNYNYN Stove CO YNYNYNYN Addressed immediate safety issues YNNA

43. Forms - WAP Evaluation Selected parts of measure installation observation form 43 Air Sealing Mark NA if not in work scope and skip section Mark NA if not done this visit and skip section Blower door used to guide air sealing Y NNA Sealing at top and bottom of envelope prioritized Y NNA In Work Scope Done This Visit Rate Quality Attic floor penetration sealed YNYN12345 Knee walls sealed YNYN12345 Second floor rim joist sealed YNYN12345

44. Forms - WAP Evaluation Selected parts of measure installation observation form 44 Air Sealing In Work Scope Done This Visit Rate Quality Basement penetrations sealed To outsideYNYN12345 To conditioned spaceYNYN12345 Crawlspace penetrations sealed To outsideYNYN12345 To conditioned spaceYNYN12345 Garage penetrations to conditioned space sealed YNYN12345

45. Forms - WAP Evaluation Selected parts of measure installation observation form 45 Air Sealing All major opportunities sealedYN LowestHighest Overall quality of air sealing work12345 Overall cleanliness of air sealing work12345 Comments 5 – Exceptional: No possible improvements identified 4 – Good: Minor improvements possible 3 – Satisfactory: Some improvements recommended 2 – Fair: Extensive improvements necessary 1 – Poor: Substandard in all respects

46. FINDINGS FROM ON-SITE EVALUATION 46

47. Utility Program Visit Introduction 47 # of Observations YesNo Customer expected visit100 Explained Program64 Reviewed electric usage46 Discussed health and safety issues46 Discussed comfort issues37 Discussed any problems with energy usage 37

48. Utility Program Baseload Visit Summary 48 Length of Visit Minimum MaximumAverage Visit length (minutes) 70180119 Rating ExcellentGoodFairNot Acceptable Overall rating (#of jobs) 4222

49. Utility Program Baseload Recommendations Review requirements and expectations with contractors. Reinforce importance of the walkthrough for baseload jobs with all contractors. Provide additional training to contractors on the importance of 2-for-1 swaps in refrigerator replacement, and train contractors to work with customers to obtain their acceptance of this measure. 49

50. Utility Program Baseload Recommendations Review CFL replacement procedures with contractors. Review customer education requirements with contractors. Observe baseload service delivery to ensure that contractors meet program standards for service delivery. 50

51. Utility Program Inspections Overview 51 # of Observations Very GoodGoodFair Data collection accuracy112 Measure selection13 Measure appropriateness22 Exceptional GoodSatisfactory Effort31 Quality121 Appropriateness31 Overall Rating13

52. Utility Program Full Cost Recommendations Develop one set of required forms. Provide instructions on the back of each form. Require all applicable diagnostic tests at the audit visit. 52

53. SUMMARY 53

54. Recommendations On-site work is an important component of technical process evaluations. Select sample that accurately represents program. Design tools to capture what is observed in the field. Train staff to systematically record data and information. Analyze findings in conjunction with other research. 54