1. SEEM 94 Calibration to Single Family RBSA Data Analysis and proposed actions RTF SEEM Calibration Subcommittee May 7, 2013

2. Goals for today’s Subcommittee Meeting Review the following presentation in detail – Consensus on next steps Are there any needed changes to the analysis? Is there subcommittee consensus that the RTF should make a decision stating SEEM94 is calibrated? – Receive suggestions from the subcommittee for improvements in the presentation Does it adequately tell the full story? Is it the appropriate tool present to the RTF (assuming previously covered sections will be skimmed over)? 2

3. SEEM 94 Calibration to Single Family RBSA Data Analysis and proposed actions Regional Technical Forum May 21, 2013

4. Overview Background – Purpose – History Methodology – Data – Regression – Calibration Discussion Proposal Overview - 4

5. Overview Background – Purpose – History Methodology – Data – Regression – Calibration Discussion Proposal Overview - 5

6. Purpose of Calibration 6

7. Align SEEM with Measured Energy Use The SEEM model is used to estimate energy savings for most space-heating-affected residential UES measures using the “calibrated engineering” estimation procedure (see section 2.3.3 of guidelines)guidelines – Heat Pumps and Central AC (ASHP, GSHP, DHP) – Weatherization – New Homes – Duct Sealing – Space Conditioning Interaction Factor Goal: Ensure SEEM94’s results are grounded in measured space heating energy use of single family homes. Use RBSA as source of measured data. 7 Background - Purpose

8. RTF Savings Guidelines 8 Background - Purpose

9. Overview Background – Purpose – History Methodology – Data – Regression – Calibration Discussion Proposal Overview - 9

10. RTF Decision History 10 Background - History Date RTF Decision Summary Housing TypeT-stat ResultsData Sources Used in Calibration Nov-2009 SEEM 92 model is calibrated. Single Family HP & Gas FAF 70°F Day ; 64°F Night Electric FAF and Zonal 66°F Day & Night 1. Res New Const. Billing Analysis (RLW 2007) 2. SGC Metered Data 3. NEEA Heat Pump Study (2005) Note: Very limited representation of Zones 2 & 3 Apr-2011 SEEM 93 model is calibrated. (implicit decision) Single Family with GSHP 70°F Day ; 64°F Night 1. Missoula GSHP Study (1996) Dec-2011 Use updated SEEM94 model Single Family, Manufactured Home n/a Ecotope updated SEEM code to model the physics of the house infiltration, rather than rely on a constant stipulated infiltration rate input in previous versions of SEEM. Dec-2011 SEEM 94 model is calibrated Manufactured Home 69.4°F Day 61.6°F Night 1. NEEM 2006 2. NEEA Heat Pump Study (2005) 3. MAP 1995 4. RCDP (manufactured homes) Sep-2012 SEEM 94 model is calibrated Multifamily Walk-up and Corridor 68°F Day& Night Townhouses 66°F Day & Night 1. Multifamily MCS (SBW 1994) 2. MF Wx Impact Evaluation for PSE (SBW 2011) 3. New Multifamly Building Analysis (Ecotope 2009) 4. ARRA Verification for King County (Ecotope 2010)

11. RTF Decision History (Continued) For “model is calibrated” decisions… Calibration Methodology: 1.Use available house and operation characteristics data from billing/metering studies to develop inputs to SEEM runs; 2.Adjust SEEM thermostat setting input to achieve a good match (on average) between SEEM output (annual heating energy use) and billing/metering study results. Note: The data sources used were free of (or mostly free of) supplemental fuel usage (wood, propane, oil, etc.) Collection of reliable electric and gas usage data for space heat consumption is relatively easy compared to other fuels. 11 Background - History

12. DateForumTopicOutcomeLinks 1/23/13Full RTF Proposal to adopt calibration: Send staff back to assess calibration needs related to climate and measure parameters; and engage subcommittee. Presentation Minutes 3/20/13 Sub- committee Status update and check in. Presentation Minutes 5/7/13 Sub- committee Review staff’s proposal in detail. Decide whether to recommend RTF adoption. Today Presentation Minutes SF Calibration to RBSA - Recent History History - 12

13. Overview Background – Purpose – History Methodology – Data – Regression – Calibration Discussion Proposal Overview - 13

14. Methodology – Overview 14

15. Two Sources of Heating Energy Estimates RBSA-PRISM. Estimates of annual “space heating use” for each house determine by using PRISM – PRISM is a “change-point” regression model that uses billing data to estimate temperature-sensitive use – PRISM analysis based on monthly billing data (at least 2years) SEEM. Estimated annual space heating energy use for each house based on SEEM engineering model – RBSA individual home characteristics (e.g., thermal envelop, heating system type, duct tightness) used as model inputs; – Initial model runs use thermostat set to 68°F day & night SEEM is a one-zone model, so t-stat setting input represents the average setting for the entire house Actual t-stat settings are not well documented (occupant reported settings are unreliable, especially for “zonal systems”) Thermostat setting will be used (step 2, below) as the “calibration knob” Methodology - Overview 15

16. Step 1 (Regression) Use regression techniques to identify building characteristics that drive systematic differences between SEEM(68°F) and PRISM space heating energy use estimates. 16 Methodology - Overview

17. Step 2 (Calibration) Use regression results to determine thermostat set- point that will align (i.e., “calibrate”) SEEM with PRISM annual space heating use – Calibration based on comparing average of all SEEM (68) annual estimates to average of all PRISM annual estimates – Calibration is based on building characteristics identified in regression. – SEEM run for each house at varying “day-time” thermostat settings, with “night-time” thermostat settings based on occupant reported setbacks 17 Methodology - Overview

18. Overview Background – Purpose – History Methodology – Data – Regression – Calibration Discussion Proposal Overview - 18

19. Data Sources Data Source used in this calibration: Underlying database* for the Single Family Residential Building Stock Assessment (2012) – RBSA study’s database offers recent billing analyses results and detailed house characteristics on 1404 single family houses in the Region. – RBSA data allows inputs for SEEM runs to be well defined for individual homes. Methodology - Data 19 * Using a pre-release version of the database for this analysis.

20. Key Model Input Parameters RBSA Data Availability UAAvailable for each house. WeatherZip code (available for each house) linked to nearest TMY3 weather station. Gas Heating EfficiencyAvailable for some houses; used average for remaining houses. HP Operation & EfficiencyNot readily available. Used ARI control & 7.9 HSPF. Duct System Leakage and Surface Area Available for some houses; used average for remaining houses with ducts. Duct System Insulation and Location Available for each house. InfiltrationAvailable for some houses; used a floor area-scaled average (by foundation type) for remaining houses Mechanical VentilationNot available. Assumed 2 hours /day at 50 cfm. Non-Lighting Internal GainsNot available. See next slide for details. Lighting Internal GainsLPD available for each house; assumed 1.5 hours/day. T-stat SettingAvailable based on interviews, but used this as the “calibration knob”. 20 Methodology - Data

21. Detail: Non-Lighting Internal Gains 21 *Hendron, Robert. "Building America Research Benchmark Definition, Updated December 20, 2007." NREL/USDOE EERE. January 2008. NREL/TP-550-42662 Methodology - Data

22. Realistic SEEM Simulations Not Feasible/Possible for All Homes in RBSA IssueCount More than one foundation type331 25% > Ceiling Area to Floor Area > 200%, or Missing Ceiling U-value36 Footprint Area to Floor Area < 20%36 30% > Wall Area to Floor Area > 200%, or Missing Wall U-value24 Missing Floor U-value for Crawlspace Foundation5 Window Area = 03 Window u-value = 03 Resulting House Count: 1011 – These issues overlap on some houses, so the sum of the counts cannot be subtracted from 1404 to get 1011. 22 Methodology - Data

23. Data Filters Excluded Some RBSA Homes VariableFilter Value(s) NotesCount (filtered out) SEEM RunValidSEEM run must be valid (> 0 kWh/yr).4 Billing Energy Use> 1,500 kWh/yr Intends to screen out partially used or unused houses 38 eRsq and gRsq= 0 or ≥0.45 Screens out houses with poor billing analysis results (0.45 per David Baylon) 398 Non-natural-gas & non-electric Fuel Use 0Screens out houses with wood, oil, propane, etc. consumption because billing analysis not performed. 352 Primary Heating System eZonal, eFAF, gFAF, HP Removes gas boilers, wood stoves, etc.216 Secondary Heating System Fuel Electric or Gas Removes wood stoves, propane heaters, etc.274 Gas Billing converted to kWh/year using reported AFUE Resulting House Count: 293 (The counts for each item overlap here, too) 23 Methodology - Data

24. Additional Data Filter for PRISM Excluded Additional Homes Exclude any home that had an out-of-range PRISM T-balance for one or more components. – The PRISM analysis restricted balance point temperatures to be between 48 and 70 ⁰F. T-balance below 48⁰ is plausible. T-balance above 70⁰ is not physically plausible. We filter these out since such values are evidence of a poor PRISM fit. Our 293 sites’ T-bal values include… – 10 that defaulted to 70⁰ when PRISM’s initial fit exceeded the max. (Excluded from analysis.) – 16 that defaulted to 48⁰ when PRISM’s initial fit was below the minimum. (Kept) – 267 whose PRISM balance points were within the acceptable range. (Kept) This leaves 283 sites for the present analysis. 24 Methodology - Data

25. Final Data Set SEEM values calculated with t-stat = 68°F (constant) 25 Methodology - Data Seem (68) Heating Energy (kWh)

26. Overview Background – Purpose – History Methodology – Data – Regression – Calibration Discussion Proposal Overview - 26

27. Methodology - Regression 27

28. Regression Overview (1) Analysis Identify and quantify any systematic patterns (trends) in the differences between SEEM(68°F) and PRISM savings estimates ( ∆ kWh = SEEM(68°F) kWh ‒ PRISM kWh. Systematic means “explained by known variables.” (Example: SEEM(68°F) kWh tends to exceed PRISM kWh in cooler climates.) Tacit assumption: PRISM estimates roughly unbiased. Definitions “PRISM kWH” = Heating energy use via billing analysis; from RBSA SF dataset. “SEEM(68°F) kWh” = Heating energy use via SEEM runs using house-specific characteristics data from the RBSA SF dataset with thermostat set to 68°F 28 Methodology – Regression

29. Regression Overview (2) Problem is multivariate… – A single underlying trend (example: ∆ increasing with heating use) may appear in multiple guises (∆ increasing with HDD, or with U-value, or with building heat loss) Approach is multiple regression… – Compare PRISM kWh with SEEM kWh when SEEM is run with a constant T-stat setting (68°F day, 68°F night.) – Y-variable is the percent difference between SEEM kWh and PRISM kWh (when SEEM uses T-Stat=68°F). – X-variables are physical characteristics known through RBSA. (Specifying the x-variables is a large part of the work.) 29 Methodology – Regression

30. Setting up the Regression (1) Primary interest is in differences between SEEM(68) kWh and PRISM kWh—the Y-variable must capture these differences. – Heteroskedasticity. The SEEM(68) /PRISM differences generally increase in magnitude in proportion to SEEM(68) kWh (or PRISM kWh). (See earlier graph.) – Measurement error (random noise). As estimates of heating kWh, SEEM(68) and PRISM both have substantial standard errors. 30 Methodology – Regression

31. Setting up the Regression (2) 31 Methodology – Regression

32. Dividing by PRISM kWh magnifies differences where PRISM’s random error happens to be negative (since these values get artificially small denominators). This biases the percent differences upwards. 32 Methodology – Regression

33. Upward bias (mostly) goes away when we divide by the value halfway between SEEM(68°F) and PRISM. 33 Methodology – Regression

34. Upward bias (mostly) goes away when we divide by the value halfway between SEEM(68°F) and PRISM. 34 Methodology – Regression

35. Upward bias (mostly) goes away when we divide by the value halfway between SEEM(68°F) and PRISM. 35 Methodology – Regression

36. Upward bias (mostly) goes away when we divide by the value halfway between SEEM(68°F) and PRISM. 36 Methodology – Regression

37. Building the Regression Model Goal is to identify variables that lead to systematic differences between SEEM(68°F) and PRISM. – “Lead to” is only seen in rough trends (think: correlation). Looking to capture unknown effects – not a physical model. Model development is iterative. – A variable may be weakly correlated with raw y-values but strongly correlated with y’s that have been adjusted to account for some other variable’s influence. 37 Methodology – Regression

38. Important Limitations Avoiding Colinearity - When a potential x-variable closely tracks some combination of variables that are already included. – Example – Including both heat loss rate and vintage – This redundancy leads to unstable model fits. – Threshold for “tracks too closely” gets low when the usual suspects are around: High noise / faint signal / small sample. Pursuing Parsimony. General principle: Don’t over-fit the data (by including too many explanatory variables). Incomplete data variables. Some variables (e.g., duct tightness and infiltration) aren’t known for very many houses. Methodology (Regression) - 38 Methodology – Regression

39. Prominent x-variable candidates Characteristics that likely influence differences between SEEM(68°F) and PRISM estimates of use – Thermal efficiency drivers (U-values, duct tightness, infiltration, …) – Heating system type – Climate (i.e., HDDs) Following graphs illustrate “influence” of several variables (separately) on percent difference between SEEM(68°F) and PRISM 39 Methodology – Regression

40. 40 Methodology – Regression

41. 41 Methodology – Regression

42. 42 Methodology – Regression

43. Insulation Variables The big surfaces: Wall, ceiling, and floor. Express in terms of heat loss (U-values, weighted by surface area as appropriate) We separate out Floor U because of different foundation types. – One variable accounts for ceiling and wall heat loss. – Another variable accounts for floor heat loss in crawlspace homes. 43 Methodology – Regression

44. Variable for Wall/Ceiling U Applies to all homes (regardless of foundation type). A simple indicator variable: “Wall/Ceiling Insulation is Poor” if Wall u-value > 0.25, OR Ceiling u-value > 0.25, OR Both u-values > 0.25. This variable captures the main effect of the weighted average. (See next slide) 44 Methodology – Regression

45. 45 Methodology – Regression

46. Variable for Floor U Particularly interested in crawlspace heat loss since crawlspace insulation is a common measure. Variable definition: “Yes/No” indicator for uninsulated crawlspace. Note: Sites with basements, slabs, and insulated crawlspaces all have Uninsulated Crawl = “No” 46 Methodology – Regression

47. Do these indicator variables really capture the insulation effects? The next two slides compare various u-values’ relationships with – Unadjusted (raw) percent differences; – Percent differences that have been adjusted for the two insulation variables included in the regression. 47 Methodology – Regression

48. 48 Methodology – Regression

49. 49 Methodology – Regression

50. Heating System Variable Four distinct heating systems in the sample: Electric zonal Electric FAF Gas FAF Heat pump After controlling for insulation, heating system effect appears to be captured with just two groups: “Electric Resistance” = Electric zonal / Electric FAF “Gas/HP” = Gas FAF / Heat Pump Parsimony: two is better than four! 50 Methodology – Regression

51. 51 Methodology – Regression

52. 52 Methodology – Regression

53. Model 1 fit summary: Est. s.e. p-value (Intercept) -0.01 (0.04) 0.80 elec. resistance 0.27 (0.05) 0.00 poor.ins.ceil.wall 0.42 (0.08) 0.00 uninsulated.crawl 0.15 (0.07) 0.04 Adjusted R-square = 0.212 …and with an interaction term for insulation: Est. s.e. p-value Intercept -0.18 (0.04) 0.62 elec. resistance 0.27 (0.05) 0.00 poor.ins. ceil.wall 0.49 (0.09) 0.00 uninsulated.crawl 0.21 (0.08) 0.01 poor.ins.c.w*unins.crawl -0.21 (0.16) 0.19 Adjusted R-square = 0.214 53 No strong recommendation either way because of low p-value, but proposal is to drop the interaction term. Methodology – Regression

54. Climate variable HDD effect is not very pronounced. Next slide shows percent differences (adjusted for effects in the previous regression), versus HDDs Standard HDDs with constant (65⁰) base. Plot shows (slight) positive correlation between HDDs and adjusted y-values. Group means (x-mean, y-mean) lie very near the overall trend line. 54 Methodology – Regression

55. 55 (Black line indicates OLS linear regression fit.) Methodology – Regression

56. Climate Variable A modest linear trend is clear from the plot. Could either use indicator variables, or the actual HDDs values (a single continuous variable). – Group means agree with overall linear trend almost perfectly, so little practical difference. We use the continuous variable, x = HDDs. 56 Methodology – Regression

57. Previous fit: Estimate s.e. p-value (Intercept) -0.01 (0.04) 0.80 elec. resistance 0.27 (0.05) 0.00 poor.ins.ceil.wall 0.42 (0.08) 0.00 uninsulated.crawl 0.15 (0.07) 0.04 Adjusted R-square = 0.212 And now with HDDs: Estimate s.e. p-value Intercept -0.40 (0.15) 0.01 elec. resistance 0.27 (0.05) 0.00 poor.ins.ceil.wall 0.44 (0.07) 0.00 uninsulated.crawl 0.13 (0.07) 0.07 Base-65 HDDs 7.3e-5 (2.7e-5) 0.01 Adjusted R-square = 0.230 57 Methodology – Regression

58. Interpreting the HDD Coefficient 58 Methodology – Regression

59. So far, so good… The next two slides compare four variables’ relationships with – Unadjusted (raw) percent differences; and – Percent differences that have been adjusted for all four variables included in the regression. HDDs and heat source show zero relationship with adjusted differences. Square footage and internal gains relationships went from weak to weaker (even though they are not included in the model – that’s good!). 59 Methodology – Regression

60. 60 Methodology – Regression

61. 61 Methodology – Regression

62. Percent difference versus midpoint has also improved… (And midpoint isn’t in the model either) 62 Methodology – Regression

63. And the insulation variables’ plots still look good… 63 Methodology – Regression

64. 64 Methodology – Regression

65. 65 Methodology – Regression

66. What else should we consider? Next slide indicates several variables’ correlation with adjusted percent differences. 66 Methodology – Regression

67. 67 Observations Duct leakage has the largest apparent correlation, but this variable is sparsely populated. (We’ll look at it next.) PRISM HDDs have moved up – these had almost no correlation with unadjusted differences. (We discuss at the end.) RBSA (reported) t-stat values have a slight negative correlation with % differences. (This sign makes sense, but we’d expect more.) Methodology – Regression

68. Duct Leakage Have direct RLF and SLF measurements for 33 homes; Also, 87 homes have no ducts (zero leakage); Another 38 (excluded from analysis) have ducts entirely inside of conditioned spaces. – Some of these spaces are basements designated “conditioned” simply because they contain ducts. – 26 of the 38 have “heated basements” Not much basis for calibration here… 68 Methodology – Regression

69. 69 Methodology – Regression

70. Duct Leakage (continued) Visually, there’s not much correlation in the range containing most of the data. Numerically… – Correlation is 53% when only the 33 measured values are included; – Drops to 15% when 4 right-most points are omitted; – Values drop to 31.5% and 5.3% when we include homes without ducts (zero leakage). Weak (and ambiguous) basis for recommending adjustment specific to duct-leakage. 70 Methodology – Regression

71. Infiltration Have direct infiltration measurements for 95 homes; But even less reason for calibration here… 71 Methodology – Regression

72. 72 Methodology – Regression

73. Infiltration (continued) Visually, the relationship is null. Numerically… – Correlation is -11.8% when all points are included; – Changes to 2.3% when single right-most point is omitted. No basis for adjustment for infiltration once other variables are included. 73 Methodology – Regression

74. PRISM Balance Point A question was raised at the May 20 subcommittee meeting regarding whether the regression should take into account the house balance point determined by PRISM. 74 Methodology – Regression

75. 75 Methodology – Regression

76. PRISM HDDs Definitely a trend, but is it unexpected? Does it require action? – We know that unobserved variables drive a portion of SEEM-PRISM deviations that we treat as noise. – Consider a home where an unobserved variable yields an effective balance point that is lower than we would expect based only on observed variables. This home will tend to satisfy both: SEEM(68) kWh > PRISM kWh and TMY HDD > PRISM HDD – In other words, the presence of unobserved variables causes a positive correlation between kWh differences and HDD differences. Conclusion: The presence of unobserved variables should yield a trend like the one seen on the previous slide. So the trend is what we would expect. 76 Methodology – Regression

77. Proposed Final Model Variable Estimated Standard p-value coefficient error Intercept -0.40 (0.15) 0.01 elec. resistance 0.27 (0.05) 0.00 poor.ins.ceil.wall 0.44 (0.07) 0.00 uninsulated.crawl 0.13 (0.07) 0.07 HDDs (Base 65) 7.3e-5 (2.7e-5) 0.01 Adjusted R-square = 0.230 77 Methodology – Regression

78. Overview Background – Purpose – History Methodology – Data – Regression – Calibration Discussion Proposal Overview - 78

79. Final Step: Interpreting Results From the fitted model, we obtain adjustment factors that apply to SEEM output to align SEEM with the RBSA-PRISM data. A given site’s adjustment factor depends on the values of the explanatory variables for that site. Group HDDs by climate zone. Then for each zone, there are 8 possible configurations of the three other variables. This yields 24 distinct adjustment factors in all. 79 Methodology – Calibration

80. Calibration Factors SEEM(68) differs from PRISM by these factors (on average) 80 Methodology – Calibration

81. T-Stat Calibration Translate percent kWh adjustments into adjustments in daytime t- stat setting (from 68 °F). No data limitations here: we can directly observe SEEM’s sensitivity to t-stat settings. Method: 1.Run SEEM for each house at multiple temperature settings in 2 degree increments – Daytime Settings: … 58, 60, 62, … – Nighttime Setback: Daytime setting - setback » Setback: Use average difference between reported daytime and nighttime t-stat settings in RBSA dataset; by heating system type: 2.Determine relationship of calibration factors to temperature settings for each of the 24 scenarios. 3.Interpolate to determine “calibrated” t-stat settings. (need to add a graph to help explain this) 81 Methodology – Calibration

82. Calibrated Thermostat Settings 82 Methodology – Calibration

83. Overview Background – Purpose – History Methodology – Data – Regression – Calibration Discussion Proposal Overview - 83

84. Discussion 84 Discussion

85. Next Steps If the RTF agrees it’s calibrated, the RTF will be able to use SEEM94 to help estimate energy savings for residential single family – Heat Pump Conversions Upgrades Commissioning, Controls, and Sizing – Weatherization Insulation Windows Infiltration reduction – Duct Sealing – New Home Efficiency Upgrades “Help” is used here because we will still need to deal with “non- electric benefits” for these measures. – This topic is out of scope for today’s discussion. The goal today is simply to determine whether SEEM has been calibrated to provide reliable results. 85 Discussion

86. Overview Background – Purpose – History Methodology – Data – Regression – Calibration Discussion Proposal Overview - 86

87. Proposed Motion “I _______ move that the RTF consider SEEM94 calibrated for single family houses.” 87 Proposal