1. Where did it go? Lost savings found in real-world data SEEM Calibration for the Northwest Power and Conservation Council’s Regional Technical Forum (RTF) Josh Rushton, Rushton Analytics Adam Hadley, Hadley Energy Engineering

2. Motivation 2 Model parameters Floor area Foundation Heating equipment Duct tightness Attic R Wall R ⋮ Thermostat setting Internal gains ⋮ Location/weather SEEM simulation Physics stuff ⋮ Heating energy Some things pretty well- known Others not so much Some shortcuts ? Calibration

3. Calibration overview Basic idea: Based on a sample of audited homes, identify and estimate systematic differences between SEEM-based and bill-based heating energy estimates. 3 Object of study is difference between two reasonable estimates

4. Calibration overview Basic idea: Based on a sample of audited homes, identify and estimate systematic differences between SEEM-based and bill-based heating energy estimates. Two calibration phases... 4 Phase I. Differences in total heating energy associated with… Envelope characteristics Heating equipment Occupant behavior(?) Phase II. Differences in electric heating energy associated with non-electric fuels Our analysis doesn’t identify / separate out root causes Estimates coarse but straightforward

5. The data Residential Building Stock Assessment (RBSA) Sponsored by Northwest Energy Efficiency Alliance (NEEA) Survey of 1404 homes in WA, OR, ID, MT – Physical building characteristics – Site-level billing data summaries – Occupant interview data It’s public! Google “NEEA RBSA” 5

6. Calibration phase I (Total heating energy) Sample filter: “Clean” sites only – No off-grid fuels – Clear heating energy signatures in bill data – No major confounding loads (hot tub, irrigation, …) Analysis: Regression-based comparison – SEEM.69 kWh. SEEM output; some inputs from audit, others conventional (T-stat = 69⁰ day / 64⁰ night ) – VBDD kWh. Billing data heating estimates calculated with variable-base degree-day (VBDD) algorithm Basic finding: The difference SEEM.69 – VBDD tends to be greater among homes that are expensive to heat. 6

7. Calibration phase I (Total heating energy) Conclusion Need to adjust SEEM.69 to align with VBDD. Increase SEEM.69 for homes that are cheap to heat Decrease for homes that are expensive to heat 7 Phase I: Adjustment Factor vs Uo Uo

8. Calibration phase I (Total heating energy) Example Electric-resistance, heating zone 1, varying Uo values Intent: Phase I-calibrated SEEM estimates should align with VBDD (on average) for “clean” homes. 8 SEEM.69 (kWh) Uo Phase I adjustment factor Phase I calibrated (kWh) Site A8,0000.0950.977,744 Site B6,7000.0801.026,840

9. Calibration phase II (Electric heating energy) Sample filter: “Program-like” homes – Off-grid fuels okay, but must have permanent electric heat too And we need decent VBDD estimates… – Clear heating energy signatures in bill data – No major confounding loads (hot tub, irrigation pump, …) – Not too much missing data Analysis: Regression-based estimate of how non-electric fuels affect electric heating energy (VBDD). 9

10. Calibration phase II (Electric heating energy) Basic finding: Just what you’d expect 10 Zone 1 summary KWh adjustment (affected homes) Percent of Zone 1 homes affected Net adjustment “Wood” (kWh) 6K to 12K-20%30%-6.2% “Wood” (kWh) Over 12K-39%11%-4.3% Gas heat (kWh) Over 8K-69% 8%-5.2%

11. Calibration phase II (Electric heating energy) Phase II breaks Phase-I-calibrated estimates (of total heating energy) into components Intent: Expect to see about 83% of Phase-I kWh on the grid. 11 Non-electric fuel components Non-energy component Electric heat component Off-grid fuelsNatural gasPoor-VBDDNA 10.5%5.2%1.6%82.8%

12. Conclusions Uncertainty about inputs would affect any simulation model, not just SEEM Observational nature of Phase I is a weakness – Matters when we estimate savings – Looking into pre-/post- program billing data to corroborate savings values Phase II results pretty straightforward 12

13. Additional slides 13

14. Phase I 14 Have to do something about heteroskedasticity…

15. Phase I 15 Need to do something about heteroskedasticity… We look at percent differences between SEEM.69 and VBDD, numerically similar to ln(SEEM.69 – VBDD) If you try this at home, consider this perspective: Goal is to get SEEM-based estimates that align with VBDD (on average). In modeling terms, SEEM.69 is a known variable; use it however you like… – Normalize the target variable (VBDD) – Build explanatory variables Might be more sensible to reverse the sign… ln(VBDD – SEEM.69)

16. The Uo Pattern 16 SEEM.69 < VBDD for lower-Uo homes SEEM.69 increases relative to VBDD as Uo increases Average difference roughly constant across high Uo values y-axis: percent difference SEEM 69/64 – VBDD y > 0 means SEEM.69 > VBDD

17. The Uo Pattern 17 y > 0 means SEEM 69/64 > VBDD SEEM 69/64 < VBDD for lower-Uo homes SEEM increases relative to VBDD as Uo increases Average SEEM – VBDD difference roughly constant across high Uo values Pre-1992 Post-1992 y-axis: percent difference SEEM 69/64 - VBDD

18. Someone asked if the pattern might be due to the fact that a lot of homes didn’t get blower door tests (we imputed infiltration values for these in SEEM input). 18 Doesn’t appear to be the case. If it were, then the pattern should go away for homes that had actual blower door tests.

19. Someone asked if the pattern might be due to the fact that a lot of homes didn’t get duct tests (we imputed duct leakage values for these in SEEM input). 19 Doesn’t appear to be the case. If it were, then the pattern should go away for homes that had duct blaster tests. (We already knew about the zonal heat effect) Duct blaster