SEEM Calibration: Phase-2 Adjustments for Failed VBDD Fits RTF Calibration Subcommittee July 31, 2014

Agenda 1.Problem overview and background 2.Proposal A 3.Proposal B 4.Discussion

1. Problem Overview

Calibration Review Phase I: Estimating total heating energy. – Align SEEM with billing data for homes with strong and clear heating energy signatures and no off-grid fuels. Phase II: Estimating electric heating energy in “typical” program homes. – How is electric heating energy affected by the presence of natural gas and off-grid fuels? – What can we say about electric heating energy in homes with weak or unclear heating energy signatures? 4

Phase II general approach 5 Phase I gives total heating energy estimates for homes with clear VBDD signatures. RTF measure savings needs average electric energy savings for all program homes. Phase II uses regression to find out… – How the presence of non-electric fuels affects electric heating energy, – How heating energy differs in homes with unclear VBDD signatures. Regression focuses on TMY-normalized (VBDD) estimates derived from electric billing data.

Example: SF Phase II regression 6 VariableDefinitionEstimateStd. ErrorP-value Intercept Natural log of UA x HDD Natural log of the area of conditioned floor space Indicator: Has electric forced-air furnace Indicator: Has heat pump Indicator: Non-utility fuel over 40,000 kBtu/yr Indicator: HZ1 and non-utility fuel between 5,000 and 40,000 kBtu/yr Indicator: Gas heating energy over 5,000 kWh/yr Indicator: Failed SEEM Calibration billing analysis filter

Example: MH Phase II regression 7 VariableEstimateStd. Errort valuep value Intercept ln(UA × HDD65) Heat pump Gas Ht. (kWh) 4K to 8K Gas Ht. (kWh) Over 8K Wood (kWh) 6K to 12K Wood (kWh) Over 12K Bad VBDD fit Adjusted R-square: 40%

The issue Regressions show that all else equal, VBDD heating energy estimates tend to be lower in homes with “poor” VBDD fits. But VBDD fits are “poor” for these sites! Should we really conclude that actual heating energy is lower?

What’s at stake? In homes with poor VBDD fits, VBDD estimates are around... o 1 - exp(-0.42) = 34% lower in SF homes; o 1 - exp(-0.33) = 28% lower in MH homes. About 20% of SF homes and 28% of MH homes have poor VBDD fits. Net result is 6% to 7% less heating energy across the full population.

Additional Notes In addition to VBDD-normalized estimates, the RBSA also provides “annualized” energy estimates. Annualized estimates are… Simple sums of 1 year’s billing records (maybe a couple months imputed if necessary) Very reliable (no regression fit, no modelling error) All end uses together – no separate heating energy estimate.

Additional Notes All else equal, homes caught by “poor VBDD” filter have lower annualized energy values. Regression-based quick estimate (accounts for UA*HDD, square footage, etc.) puts them around 2800 kWh lower kWh is about 14% of average total energy; 2800 kWh is about 28% of average heating energy (per VBDD estimates with good fits);

The big question So “poor-VBDD” homes have around 2800 kWh less than their “good-VBDD” peers. How is the missing 2800 kWh distributed across end-uses? Mostly missing heating energy? From all end-uses equally? Something else?

One more bit of data Annualized estimates available for full RBSA but don’t separate out heating energy. RBSA Meter Study has a lot of detail for a small number of sites. – For sites in the meter study, we have pretty good information on actual heating energy. – 40 of these are in the SF Phase II sample. 12 with “poor” VBDD fits 28 with “okay” VBDD fits

2. Proposal A

Summary Proposal A: Take the existing Phase-II regression results at face value. Supporting claim: Homes with poor VBDD fits really do have much lower heating energy (on average) than those with okay fits. It’s mostly heating energy that’s missing from these homes’ annualized energy values. Reasons…

VBDD estimates mostly agree with our “truth set”, except for a few sites. VBDD is actually too high in these sites. Suggests that for bad-VBDD sites, actual heating energy is even lower than what VBDD says. Warning: Note mismatched time periods. Reason 1, VBDD vs “Truth”

Meter study sample probably not representative in all respects we care about. – In the broader RBSA sample, VBDD estimates tend to be lower for poor-VBDD-fit sites than for okay-VBDD-fit sites. – Reverse holds in previous plot. Discrepancy partially explained by confounding variables (UA*HDD, wood heat, etc.); After accounting for these, t-test for difference in means has p = In any event, meter study sample gives no evidence that VBDD is artificially low in poor-VBDD sites; If anything, it suggests actual heating energy is lower still in these sites. Reason 1, continued

Graph uses meter- study data; values are “real”. 9 of the 12 bad-VBDD sites lie below the okay-VBDD trend. Meaning: Relatively small fraction of their total energy is heat- related. Two main exceptions, and 10877; VBDD- and meter- based estimates agree for these. Reason 2, Heating kWh vs Total kWh

Among sites in the meter study, most of those with “poor” VBDD fits use less heating energy, as a fraction of total energy, than those with okay fits. There are two serious exceptions, but VBDD appears to have gotten these sites “right.” (For whatever that’s worth.) As support for Proposal A, Reason 2 is sort of a mixed bag. Reason 2, continued

3. Proposal B

Summary Proposal B: Treat the “poor” VBDD fit sites’ missing 2800 kWh as coming from all end uses equally, so everything is reduced by about 14%. This leads to a net reduction of about 3% to 4% in population-average heating energy (instead of 6% to 7%). Supporting claim: VBDD is totally detached from reality for some of these homes so we shouldn’t trust it at all. We have no clear basis for dividing the missing kWh among end uses, so the conservative move is to assume a uniform allocation. Reasons…

Sometimes a “poor” VBDD fits are total nonsense. VBDD-based estimates in such cases may as well be taken directly from a random number generator. Reason 1, total VBDD failures

Reason 2, heating fractions This is another perspective on heating kWh versus total kWh. We saw that this comparison is a mixed bag as support for Proposal A. This supports Proposal B by highlighting how little we really know.

4. Discussion

Subcommittee notes

Subcommittee recommendation

Supplementary slides

CoefficientDescriptionEstimate*Standard Errorp-val β0β0 Intercept <.001 β1β1 Floor Area (ft 2 ) <.001 β2β2 UA * HDD65 (kWh/yr) <.001 β3β3 Heat Pump (TRUE/FALSE) β4β4 Failed Filter (TRUE/FALSE) < ft 2 higher area  360 higher annualized kWh 1000 kWh higher UA * HDD65  120 kWh higher annualized kWh Presence of Heat Pump  60 kWh lower annualized kWh (not significant) Failed Bill Filter  2800 lower annualized kWh *Estimated w/ RBSA survey weighting Raw kWhAdjusted kWh Passed Filter Failed Filter Difference

Summary of VBDD failures Site IDReason FailedDiagnosisMeter heating kWhVBDD vs Meter 10887Tbal 70F, Low R^2Malfunctioning HP (?)HighAgree 11418Tbal 48FProbably actual TbalNormalAgree 12507Tbal 70FSpaNormalVBDD too high 13912Tbal 48FNo utility heatingLowAgree 14140Tbal 70FShop (?)NormalVBDD much too high 20020Tbal 48FSpaNormalVBDD much too high 20230Tbal 70FPossibly actual TbalNormalAgree 20469Tbal 48FLarge cooling loadNormalAgree 20998Tbal 48FCooling, Spa, & Well PumpNormalAgree 21143Tbal 48FBad ReadLowAgree 23960Low R^2Bad ReadNormalAgree 24203Tbal 48FProbably real TbalNormalAgree 24375Low R^2Vacations & SpaLowAgree 24684Low R^2Possible Bad ReadsNormalVBDD too high

Annotated VBDD vs “Truth” Shop, sump Spa Bad meter read Among “poor” VBDD fit sites in the meter study sample, the sites with large VBDD errors mostly have large additional loads unrelated to space heat.

Annotated Heating kWh vs Total kWh HP Malfunction Possible real T-Bal ≈ 70 Cooling, spa, well pump Cooling Among “poor” VBDD fit sites in the meter study sample… Large heating fractions are associated with a lot of heating equipment usage; Sites with small heating fractions have different causes (little heat or lots of other stuff). No utility heat