1. 1Managed by UT-Battelle for the Department of Energy Jackie Berger APPRISE National WAP Evaluation Field Process Study

2. 2Managed by UT-Battelle for the Department of Energy Presentation Overview  Process Field Study Design  Challenges and Resolution  Overview of Findings – Audits – Measure Installation – Final Inspections  Opportunities - Training, Equipment, and Management

3. 3Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY DESIGN

4. 4Managed by UT-Battelle for the Department of Energy Process Field Study Focus  Audit approach and implementation  Energy education delivery  Measure installation/operation of crews  Post-weatherization quality assurance inspection approach and implementation  Client interaction  Interactions among agency staff  Implications for management, procedures, and training

5. 5Managed by UT-Battelle for the Department of Energy Process Field Study Approach  Go beyond anecdotes  Quantify findings across all visits – Develop check lists and rating scales – Train weatherization experts to implement consistently  Enrich data with descriptive information – Examples used to illustrate and clarify – Not used to draw conclusions  Make recommendations for program improvement based on prevalent issues

6. 6Managed by UT-Battelle for the Department of Energy Observation Team  Social Scientists – One week observation at agency  Weatherization manager and staff interviews  Observations – client intake, audit, measure installation, final inspection – Written agency summary report  Weatherization Experts – 3-Week Observations at Agency  Observations – audit, measure installation, final inspection – Post Observation Data Entry  Training  Monthly update calls  Final debriefing

7. 7Managed by UT-Battelle for the Department of Energy CHALLENGES AND RESOLUTION

8. 8Managed by UT-Battelle for the Department of Energy Study Challenges (#1)  Program is different in every state and even within states – Approximately 1,000 local agencies – Varying guidelines and standards – Different level of access to training and equipment – On what standards should the jobs be assessed? – Decision: Weatherization best practices as agreed on by a national team of weatherization experts – Assessment of where there are opportunities – standards, equipment, training

9. 9Managed by UT-Battelle for the Department of Energy Study Challenges (#2)  How to pick 20 agencies to represent the program

10. 10Managed by UT-Battelle for the Department of Energy Agency Selection  WAP Agencies – Almost 1,000 deliver services across the country – 400 comprised agency billing data sample – Field Process Study – 20 agencies in 20 different states  Selection based on: – Number of states per region based on WAP funding – State selection based on part of region and program size – Agencies selected to represent: Urban/RuralEducation } Based on agency survey PY 2008 planned jobsTraining Measure selectionQuality Control

11. 11Managed by UT-Battelle for the Department of Energy Study Challenges (#3)  How to pick staff/jobs to observe at each agency – “Best foot forward” approach – Agency managers to select staff/jobs – Where are there opportunities for improvement for even the best teams at each agency

12. 12Managed by UT-Battelle for the Department of Energy Study Challenges (#4)  Obtaining consensus among weatherization experts  Number of issues, tests, measures to consider  Where to start… ….and where to stop? (still trying to figure out)

13. 13Managed by UT-Battelle for the Department of Energy Field Study Flow Chart Draft Forms and Procedures Test Draft Forms and Procedures Revise Forms and Procedures Convene Expert Panel Revise Forms and Procedures Select Agency Sample Conduct Interviews and Observations (3/11-9/11) Wx Expert and Social Scientist Debriefing Analyze Data and Summaries Report Test Forms and Procedures Revise Forms and Procedures Wx Expert Training Social Scientist Training Revise Forms and Procedures

14. 14Managed by UT-Battelle for the Department of Energy Audit Observations Check Lists  Audit preparation  Audit introduction  Audit home walk through  Heating assessment  Air conditioning assessment  Ventilation assessment  Water heater assessment  Air leakage and insulation diagnostics  Diagnostic testing*  Combustion safety testing*  Refrigerator assessment  Lighting assessment  Exit interview *Should have been performed, was performed, performed correctly **Described on write-up, recorded thoroughly and accurately Ratings  Audit introduction  Auditor attempt to engage client  Client engagement in audit  Auditor communication skills  Quantity of education  Quality of education  Exit interview  Completeness of inspection  Completeness of testing  Quality of testing  Auditor efficiency  Auditor professionalism  Auditor respect of client’s home  Auditor safety practices  Audit comprehensiveness*  Planned work comprehensiveness*  Audit write up *Excellent, good, poor

15. 15Managed by UT-Battelle for the Department of Energy Measure Installation Observations Check Lists  Measure installation preparation  Client interaction  Introduction  Air sealing procedures  Attic insulation  Thermostat installation  CFLs Ratings  Windows – sealed, interior storm, exterior storm, sash kit, repair  Doors – weather stripping, treated, replacement, installation  Air sealing (8 different locations rated)  Overall air sealing quality, cleanliness  Attic insulation quality, cleanliness  Attic hatch work quality  Walk-up attic work quality, cleanliness  Wall insulation quality, cleanliness, siding preservation/interior wall finishing  Basement insulation quality, cleanliness  Crawl space - vapor barrier insulation, perimeter wall insulation, vents sealing, ceiling insulation and air barrier installation  Ventilation – existing fans ducted to outside, whole house, attic, kitchen, bathroom

16. 16Managed by UT-Battelle for the Department of Energy Measure Installation Observations Ratings (continued)  Heating system – tune-up, repair, replacement, installation, vent correction  Thermostat installation  Air conditioning – sealing around, tune-up, replacement  Evaporative cooler – repair, replacement  Duct sealing, duct insulation, duct improvement cleanliness  Hot water temperature adjustment, heater wrapped, pipes insulated, heater repaired, replaced, heater venting work  Subcontractor treatment of refrigerator and home  Dryer venting  Low cost – AC cooling coils cleaned, HVAC filters cleaned or replaced, faucet aerators, showerheads, CO detector, smoke detector, fire extinguisher  Crew efficiency, professionalism, cleanliness, safety practices  Communication, attempt to engage client, client engagement, education quantity, education quality

17. 17Managed by UT-Battelle for the Department of Energy Final Inspection Observations Check Lists  Home walkthrough  Diagnostic tests*  Combustion safety tests*  Draft tests*  Occupant interaction *Should have been performed, was performed, performed correctly Ratings  Inspection completeness  Testing completeness  Testing quality  Inspector attempt to engage client  Client engagement  Inspector communication skills  Education quantity  Education quality  Inspector efficiency  Inspector professionalism  Inspector cleanliness  Inspector safety practices

18. 18Managed by UT-Battelle for the Department of Energy Study Challenges (#5)  Consistent implementation of assessment tool – Designed to maximize consistency – Training of weatherization experts – Extensive review of data entry

19. 19Managed by UT-Battelle for the Department of Energy Check List Example Audit Air Leaking and Insulation Diagnostics Applicable Observations Action Taken #% Measured surfaces 1009494% Inspected windows 1009696% Inspected all accessible attics 786988% Measured insulation in all accessible attics 797089% Created access to inaccessible attics 33310% Measured insulation in exterior walls 954749% Measured insulation in basement/crawlspace 745574% Inspected for all typical bypasses 1006262% Visual inspection for air sealing opportunities 1008383% Used blower door while inspecting for leaks 966467%

20. 20Managed by UT-Battelle for the Department of Energy Rating Design Rating1234*5* % of Points needed0%-49%50%-74%75%-89%90%-99%100% Total pointsNumber of Points needed for each rating 40-123--4 50-234--5 60-23-4--56 70-34567 8 4-5678 90-45-6789 100-45-78910 110-56-78-91011 120-56-89-101112 130-67-910-111213 140-67-910-121314 150-78-1011-131415

21. 21Managed by UT-Battelle for the Department of Energy Rating Example Attic Insulation Quality Quality of Attic Insulation 1 All air sealing work completed first 2 Exhaust fans vented to exterior as needed 3 Heat producing devices or systems protected from insulation contact 4 Attic checked for knob and tube wiring 5 Workers wore respirators, safety glasses, gloves, and hard hats while insulating attic 6 Insulation installed in sufficient quantity (bags per ft 2 ) to meet R-value requirement 7 Proper insulation material chosen for attic conditions 8 Open blow insulation is level and of consistent depth 9 Attic ventilation maintained 10 Confined areas blown to dense pack 11 Proper containment used to protect client and belongings Needs Improvement Excellent Mean Rating# Rated Rating12345 Total Points needed 0-56-78-91011 Bold Points needed00066 Attic Insulation Quality4%18%42%19%18%3.357

22. 22Managed by UT-Battelle for the Department of Energy Study Challenges (#6)  Remaining observers, rather than trainers, coaches, mentors – Weatherization experts are used to coaching  Told only to intervene in case of serious health and safety issue – Weatherization staff expect to receive coaching  Asked weatherization experts for feedback and advice

23. 23Managed by UT-Battelle for the Department of Energy Study Challenges (#7)  Connection between work quality and impacts  Not enough data to assess  Assumption is if… – Conduct comprehensive and accurate testing – Correctly diagnose problems in the home – Effectively engage clients about energy use practices and potential changes – Accurately transfer audit data and information to installers – Effectively implement weatherization measures – Inspect for quality and comprehensiveness – Then …results will be meet or exceed expectations

24. 24Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY AUDIT OBSERVATION FINDINGS

25. 25Managed by UT-Battelle for the Department of Energy Process Field Study Observations Completed Number of Observations Social Scientist Weatherization Expert Total Audit 43112155 Measure Installation 45114159 Final Inspection 3791128 Total 125317442

26. 26Managed by UT-Battelle for the Department of Energy Audit Strengths  Inspection – 99% inspected every accessible room  Heating system – 94% inspected  Filters – 72% inspected  Ventilation – 72% inspected the kitchen – 78% inspected the bathroom ventilation.  Water heater – 93% inspected  Air conditioning – 82% inspected outside unit.  Insulation – 89% measured all accessible attics.  Testing – 97% conducted blower door test (84% correctly)

27. 27Managed by UT-Battelle for the Department of Energy Audit Opportunities for Improvement  Bathroom ventilation flow – 24% assessed  Hot water – 39% checked temperature at the faucet  Shower flow – none of the auditors measured flow  Insulation – 49% measured in exterior walls.  Blower door – 67% used while inspecting for leaks  Zonal pressure diagnostics – 42% done (87% correctly)  IR camera – 49% used (70% correctly)

28. 28Managed by UT-Battelle for the Department of Energy Communication of Audit Findings  Audit write-ups do not convey educational opportunities  Work order not always clear or specific enough for installer to understand the desired outcome  Tools that improved information transfer – Pre-work walk through with installer – Attaching photos of critical areas to the work order – In process check-ups with the installers  Air sealing is a particular area for improved communication  Need for flexibility to change work order when problems are found during installations

29. 29Managed by UT-Battelle for the Department of Energy Audit Findings Opportunities  Utilize energy bills  Increase client engagement  Identify client-specific opportunities for reducing energy use  Auditor understanding of pressure boundaries  Auditor understanding of testing purpose and procedures  Conduct worst case draft testing Strengths  Respect and concern for clients  Explanation of WAP program and process  Use of data collection forms  Auditors meet with contractors to explain audit findings and work scope

30. 30Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY MEASURE INSTALLATION OBSERVATION FINDINGS

31. 31Managed by UT-Battelle for the Department of Energy Crews and Contractors Observed NumberPercent Crews 3531% Contractors 7465% Both 54% TOTAL 114100%

32. 32Managed by UT-Battelle for the Department of Energy Measure Installation Strengths  Air sealing prioritized the right areas – 77 percent prioritized sealing at the top and bottom of the envelope  Attic insulation followed best practices – 77 percent completed attic floor sealing prior to insulation – 88 percent had no gaps or voids – 95 percent used appropriate materials.

33. 33Managed by UT-Battelle for the Department of Energy Measure Installation Mid-High Ratings AreaRatings Window work 3.5 – 4.0 Door work 2.8 – 3.5 Attic insulation 4.0 Wall insulation 2.9 Basement insulation 3.6 Crawl space work 3.1 – 4.0 Ventilation work 4.1 – 4.8 Thermostat installation 4.1 Air conditioning work 3.0 – 4.0 Water heater work 3.1 – 4.0

34. 34Managed by UT-Battelle for the Department of Energy Measure Installation Opportunities for Improvement  Air sealing diagnostics – 22 percent used the blower door to guide air sealing – 11 percent used the zonal pressure test to affirm appropriate pressure boundaries.  Air sealing – 57 percent sealed all major opportunities. AreaRatings Air sealing 2.4 Heating system work 2.0 – 2.5 Smoke detector installation 2.6 CO detector installation 2.0 Low Ratings

35. 35Managed by UT-Battelle for the Department of Energy Measure Installation Findings Opportunities  Increased use of blower door when air sealing  Respect for clients’ homes (booties, covering furniture)  Crew member safety  Increased assessment of HVAC contractors  Explain CFLs when installing  Client education Strengths  Crew flexibility and dedication (above and beyond work scope)  Crew problem solving  Some very high quality work  Experienced HVAC contractors

36. 36Managed by UT-Battelle for the Department of Energy PROCESS FIELD STUDY FINAL INSPECTION OBSERVATION FINDINGS

37. 37Managed by UT-Battelle for the Department of Energy Final Inspection Strengths  The inspectors discussed the work that was performed – 75 percent discussed the work that was performed with the client.  Inspections were complete – 90 percent examined all accessible rooms.  Blower door testing was usually conducted – 85 percent conducted this test (85 percent of those performed correctly)

38. 38Managed by UT-Battelle for the Department of Energy Final Inspection Opportunities for Improvement  Zonal pressure test – 33 percent performed the test (87 percent done correctly)  IR camera – 44 percent used the camera (95 percent done correctly)  Combustion safety – 31 percent performed this test on the gas stove (100 percent done correctly)  Worst case draft test – 60 percent performed this test on the heating system (92 percent correctly)

39. 39Managed by UT-Battelle for the Department of Energy Final Inspection Findings Opportunities  Increased client education – Explain measures installed – Reinforce client action plan  Improved testing quality  Increased assessment of installation quality  Reduced use of auditor to conduct final inspection – additional perspective Strengths  Good job when following work order closely  Time saved when contractor attends  Information to clients – who to call if they have problems  Referrals to additional assistance programs

40. 40Managed by UT-Battelle for the Department of Energy SUMMARY

41. 41Managed by UT-Battelle for the Department of Energy Average Ratings By Agency and Overall

42. 42Managed by UT-Battelle for the Department of Energy Training Needs  Energy bill utilization – to assess needs and educate clients  Building science fundamentals  Critical thinking – assessing unique situations  Combustion safety testing  Zonal pressure testing  IR camera use  Ventilation assessment  Safe work practices – worker safety and lead safe work  Interviewing skills – how to understand the client’s needs  Client education

43. 43Managed by UT-Battelle for the Department of Energy Equipment Needs  GPS  IR camera  Boroscope and fiber optic scope and video  Personal safety equipment

44. 44Managed by UT-Battelle for the Department of Energy Management Opportunities  Standards and procedures  Policy manuals  Forms and checklists