2015 Multifamily Programs Focused Impact Evaluation

2015 Multifamily Programs Focused Impact Evaluation
April 10, 2017

Agenda Multifamily Programs Impact Evaluation
Multifamily Program Overviews Whole Building Evaluation Overview Methods & Findings Conclusions and Recommendations MFEER

Reporting Schedule 4 / 1 Draft Final report released 4 / 10 Webinar
4 / 21 Public Comments due

PROGRAM OVERVIEW: Multifamily Whole Building

Program Overview: MF-WB
Multifamily Whole Building (MF-WB) IOU implemented Initial implementation of program in Offers technical and financial assistance Target larger retrofit projects Savings based on building simulations Flexibility with measures Must achieve >10% savings to qualify

Completed 36 projects statewide in PY2015, representing over 4,500 tenant units IOU Projects Tenant Units Savings (ex ante gross) kWh kW Therms PG&E 19 1,324 919,887 480 47,273 SCE/SoCalGas 6 1,080 567,155 180 54,155 SDG&E 11 2,111 1,919,050 382 49,384 Totals 36 4,515 3,406,092 1,042 150,813 *SoCalGas and SCE jointly implement the MF-WB program. SoCalGas did not claim savings for their joint projects in 2015. Source: CPUC Tracking Database and IOU-provided claim IDs

Savings increased by more than 400% since initial implementation Program Years Savings (ex ante gross) kWh kW Therms 2013–2014 Claimed Savings 594,942 152 23,069 2015 Claimed Savings 3,406,092 1,042 150,813 Percent Increase 473% 586% 554% Sources: 2015 CPUC Tracking Database and MF Impact Evaluation

Spent less than planned Also fell short of savings goals IOU Budget Spent % Spent % Savings Achieved (ex ante gross) (% of goal) kWh kW Therms PG&E $5,400,000 $2,217,369 41% 29% 75% 53% SCE/SoCalGas $2,500,000 $1,809,819 72% 40% 13% 47% SDG&E $5,882,655 $5,802,203 99% 76% 28% 39% Totals $13,728,655 $9,829,391 71% 48% 31% 46% *SoCalGas and SCE jointly implement the MF-WB program. SoCalGas did not claim savings for their joint projects in 2015. Savings goals provided by the IOUs through EEstats data requests Budgets as provided by the IOUs through EEstats data requests. Source: CPUC Tracking Database and IOU-provided claim IDs and IOU-provided goals and budgets through EEstats data requests

PROGRAM OVERVIEW: Multifamily Energy Efficiency Rebate Program

Program Overview: MFEER
Multifamily Energy Efficiency Rebate (MFEER) Statewide, Core Program – IOU Administered Self-Install and DI measures Prescriptive rebate amounts $ $1,500

Measure Group Example Measures IOU PG&E SCE SoCalGas SDG&E Cooling Central Air Conditioner, Variable Speed Fan, Duct Sealing, Evaporative Coolers  Pool Equipment Variable Speed Pool Pump, Pool Heater Appliance Clothes Washers, Refrigerators Space Heat Natural Gas Furnaces, Boilers Lighting Light-Emitting Diode (LED) Fixtures and Bulbs, Compact Fluorescent Lamp (CFL) Fixtures, Occupancy Sensors Shell Insulation, Windows Other Vending Machine Controls, Smartstrip Small Domestic Hot Water (DHW) Faucet Aerators, Low-Flow Showerheads, Shower Start Large DHW Tankless Water Heaters, Storage Water Heaters, Water Heating Boilers, Boiler Controls

Electric Measures Lighting dominated the ex ante electricity savings 69% for PG&E, 88% for SCE, and 64% for SDG&E Gas Measures Large DHW contributed most gas savings 98% for PG&E, 79% for SoCalGas, and 52% for SDG&E Compared to the evaluation, the proportion of lighting claimed in all electric measures has gone down at PG&E (it was 84%) and SDG&E (it was 100%). However, it has gone up slightly for SCE (it was 83%)

Rebated >600,000 measures in 2015 SCE claimed most measures (69%) and electric savings (92%) PGE claimed most natural gas savings (85%) IOU Measures Savings (ex ante gross) kWh kW Therms PG&E 145,845 100,779 6 405,357 SCE 457,016 18,598,493 1,190 –98,850* SoCalGas 10,268 19,035 13 147,740 SDG&E 50,819 1,560,477 670 21,642 Totals 663,948 20,278,785 1,879 475,889 *Negative therm savings represent interactive effects.

Across all IOUs, spent more than planned Fell short of savings goals IOU Budget Spent % Spent % Savings Achieved (ex ante gross) (% of goal) kWh kW therms PG&E $1,839,507 $1,522,581 83% 50% 32% 135% SCE $11,100,651 $12,912,471 116% 69% 53% N/A SoCalGas $1,328,972 $851,267 64% 31% SDG&E $2,183,742 $2,711,606 124% 359% 65% Totals $16,452,872 $17,997,925 109% 76% 70% Notes: These ex ante totals are without any evaluation adjustments. Goals and budgets are from PIPs.

PROGRAM COMPARISONS: Spending & Savings

Program Comparison: Measures
Substantial overlap between MFEER and MF-WB measures, also some unique WB measures: Attic/Roof insulation Cool Roof DHW Demand Control Dishwashers Electric baseboards Heat pump Recirculation pump Provides evidence on if WB programs are incenting measures above and beyond those offered through MFEER. WB = flexibility (?) Overlap includes AC, aeraters and showerheads, lighting, boilers, furnace, Water heaters and windows

Program Comparison MFEER program savings considerably larger than MF-WB MF-WB programs represented only 14% of the multifamily energy (kWh) and 24% of gas (therm) savings

Program Comparison MF-WB program more expensive
MF-WB: $749 spent per MMBtu saved MFEER: $243 spent per MMBtu saved IOU MF-WB MFEER Spending $/MMBtu PG&E $2,217,369 $614 $1,522,581 $346 SCE/SoCalGas $1,809,819 $731 $13,763,738 $215 SDG&E $5,802,203 $824 $2,711,606 $489 Totals $9,829,391 $749 $17,997,925 $243 MMBTU (ex ante/gross), per $ spent (As provided by PAs) . $ per MMBTU has gone up since last evaluation. Data from evaluation: MF-WB: $651 spent per MMBTU saved MFEER: $86 spent per MMBTU saved

EVALUATION OVERVIEW: MF-WB

Data Collection: MF-WB
Decision-maker survey Participant site visits Property energy consumption (billing) data IOU EnergyPro models IOU informational data requests

Data Collection: MF-WB
Decision Maker Survey – Objectives Confirmation/verification of installed measures Anticipated actions in the absence of program intervention Importance of program education and incentives on the decision to install high efficiency equipment Working status and estimated age of replaced units Timing for building maintenance/upgrades Recruitment for site visits Site Visits – Objectives Collect meter numbers Verify measure installation Collect high-level building and dwelling characteristics IOU Completed Projects Completed Surveys Completed Site Visits PG&E 19 8 6 SCE/SoCalGas 3 1 SDG&E 11 2 Totals 36 13 9

Evaluation Activities & Outcomes: MF-WB
Baseline Assessment Freeridership (FR) Estimate Consumption Analysis Simulation Modeling

Baseline Assessment

Baseline Assessment: Methods
Compared tracking database baselines Participant phone survey to assess: Working status of prior equipment Age of prior equipment Expected remaining life of equipment Regularly schedule/government-mandated upgrade schedule and policy Looked at baselines used in models

Baseline Assessment: Findings
Tracking database-assigned baseline conditions SDG&E assigned 100% of projects as ER SCE assigned 100% of projects as ROB PG&E designated projects between ROB and ROBNC Baseline Assigned in Tracking Database PG&E SCE SDG&E Grand Total Retrofit (ER) 105 ROB (Code baseline) 6 ROBNC (New construction – code baseline) 32 38 149 RET = Retrofit (pre-program existing conditions, or ER) ROB = Code baseline ROBNC = ROB new construction (code baseline)

Energy models assumed existing conditions (ER) Inconsistency with what was in the tracking system RET = Retrofit (pre-program existing conditions, or ER) ROB = Code baseline ROBNC = ROB new construction (code baseline)

Baseline Assessment: Survey Methods
ER logic for lighting, small DHW, Roofing and Shell Measures

Baseline Assessment: Survey Methods
ER logic for all other surveys measures

Evaluation team measured baselines Measure Category Measures % ER n = Shell Windows, Insulation (Attic, Roof) 18% 13 Lighting Indoor and Outdoor CFLs and LEDs 70% 10 Large DHW Storage/Tankless/Boiler Water Heaters, Hot Water Demand Control 50% 6 All Others Appliances, Pool, Faucet Aerator, Low-Flow Showerhead 57% 7

Response categories that determined ROB Baseline Determining ROB Aspect Measure Count (n = 36) ROB Measure was part of a scheduled upgrade 19 Replaced equipment RUL <2 years 1 ER N/A 16 RET = Retrofit (pre-program existing conditions, or ER) ROB = Code baseline ROBNC = ROB new construction (code baseline)

Program measures are a mix of ROB and ER Only 33% of measures qualify for ER when logic is applied Not all measures are ER (which is the baseline used in the model)

Free-ridership Estimation

Free-Ridership Estimation: Objective
Assess influence of program on decision to install measures Capture complex decision-making process associated with MF programs Explored company policy Investigated and attempted to reach the true decision maker Investigated the use of outside agency funding Based on the CPUC ED methodology

Free-Ridership Estimation: Methods
Three equally weighted components (program attribution index, PAI): PA-1: influence of the program and non-program related factors PA-2: perceived importance of the program relative to non-program factors PA-3: likelihood of actions the customer might have taken if program had not been available (counterfactual)

Free-Ridership Estimation: Methods
Only free-ridership (FR), did not include spillover Option to differ by measure Consistency checks FR estimated individually for each respondent, then MMBtu savings weight for composite value

Free-Ridership Estimation: Findings
Overall NTFR: 44.8% PA-1 [Influence]: 40.6% More respondents rates non-program > program influences PA-2 [Relative Importance]: 40.6% 31% learned of program after deciding to install equipment PA-3 [Install Same Equipment]: 47.3% PAI-1 (Influence) PAI-2 (Relative Importance) PAI-3 (Install Same Equipment) Overall Net of FR FR Precision (90%) 40.6% 47.3% 44.8% ±10.8% NTFR MFEER = 51.6%

Consumption Analysis

Consumption Analysis: Objective
Replace EnergyPro models simulated/assumed usage with actual billing data Allow more accurate energy modeling by using actual billing data

Consumption Analysis: Methods
Primary steps Leverage onsite data collection to log meter numbers (in- unit and common areas) Use meter numbers to establish billing accounts Locate additional premise IDs associated with the buildings Summarize: billing data using meter and premise IDs total # of meter and premise accounts vs expected Compare consumption with Energy Pro

Consumption Analysis: Findings
Electric and gas meter linkages incomplete Earlier billing months incomplete (2013/2014) Common area assignment uncertainty Extrapolating to full occupancy # Buildings # Electric Units # Gas Units Number of Meters Collected Onsite Number of Meters Linked to Each Site # Electric Meters Collected (Unit / Common) # Gas Meters Collected (Unit / Common) # Electric Meters Linked (Unit / Common) # Gas Meters Linked (Unit / Common) 69 713 326 317 / 6 278 / 14 621 / 12 285 / 20

Consumption Analysis: Findings
Opted to rely on existing EPro model usage due to: No sites showed complete tenant and common linkages based on meter and premise lookups Common area uncertainty Incomplete 2013/2014 (pre-installation) billing

Simulation Modeling

Simulation Modeling: Objective
Why did the Evaluation Team choose Simulation Modeling? Allows review/verification of program inputs Can update to actual site visit characteristics Incorporates blend of ER and ROB baselines Billing analysis would not have whole building data available, small sample size, and could not incorporate various baselines. Why not a billing analysis? Difficult to get billing data for the entire site Small samples Cannot incorporate multiple baselines

Simulation Modeling IOUs using EnergyPro Version 5.0 Two modules
Non-residential (NR PERF) Residential (RES PERF) Key differences Simulation engines Methods for lighting/appliances HVAC capacity input Operating schedules

Simulation Modeling *Low-rise buildings are three stories or less

Simulation Modeling Evaluation Team used NR PERF only
Desire to maintain consistency in tool used for savings analysis NF PERF also offers more flexibility for calibration to utility billing data RES PERF considered inaccurate for SF Dropped from approved list of simulation software for CA SF Whole Building Program Overestimates heating and cooling energy consumption

Simulation Modeling Summary of evaluation updates to model
Convert to NR PERF for projects using RES PERF for ex- ante savings estimates Building characteristics updates Exterior surface areas and window areas changed if greater than 10% difference HVAC system types Measure quantities and /or measure performance metric updates ER ROB baseline adjustments For discussion: We were also going to calibrate to pre-program consumption, but were not able to do so because of concerns over the comprehensiveness of the consumption data.

Simulation Modeling: Findings
Largest changes came from switch to NR PERF and changes to baseline Some smaller changes to NR PERF even when used in Ex Ante assumptions Ex ante modeling had assumed existing conditions for all sites, Ex post modeling used Title 24 prescriptive code requirements for alterations to determine code baseline for ER and ROB

Simulation Modeling: Findings
Savings at each step in the modeling process Site level details presented in the report

PROGRAM FINDINGS: MF-WB

Ex Post Savings Gross Savings:
Applied RR from simulation model findings to all projects Gross Realization Rate, First Year Savings – by IOU IOU kWh kW Therms Ex Ante Ex Post RR PG&E 919,887 171,158 19% 480 42 9% 47,273 28,667 61% SCE/SoCalGas 567,155 105,527 180 16 54,155 32,840 SDG&E 1,919,050 357,067 382 33 49,384 29,947 Totals 3,406,092 633,753 1,042 91 150,813 91,454 MF-WB programs represented only 14% of the multifamily energy (kWh) and 24% of gas (therm) savings

Ex Post Savings Net Savings:
Applied the ex post NTG estimate (44.8%) to the ex post gross savings Net Realization Rate, First Year Savings – by IOU IOU kWh (net) kW (net) Therms (net) Ex Ante Ex Post Net RR PG&E 869,759 76,679 9% 464 19 4% 45,122 12,843 28% SCE/SoCalGas 342,256 47,276 14% 103 7 7% 34,857 14,712 42% SDG&E 1,631,192 159,966 10% 325 15 5% 41,977 13,416 32% Totals 2,843,208 283,922 891 41 121,956 40,972 34% MF-WB programs represented only 14% of the multifamily energy (kWh) and 24% of gas (therm) savings

Conclusions & Recommendations

C&R: MF-WB Conclusion 1: Although the IOUs have assumed ER savings for all multifamily measures, this research indicated that a substantial portion of projects may not qualify for ER because of planned improvements, installation of new equipment, or replacement of equipment that was in poor condition. For example, only 18% of program shell measures and 50% of water heater installations qualified as ER measures. Recommendation 1: The IOUs should set up a survey for multifamily participants at intake to better determine the appropriate baseline for each project and measure. The intake survey can follow a similar logic as that used in this report or that from the CPUC early retirement guidance document. The baseline assumptions for a sample of projects should then be verified by an independent third-party evaluator.

C&R: MF-WB Conclusion 2: This research found a NTG ratio of 44.8%. This value is slightly lower than the 2013–2014 REN MF-WB NTG value and significantly less than the IOU provided ex ante value of 85%. These NTG values reduce savings from measures that would have been installed without program intervention. Recommendation 2: IOUs should consider using the researched NTG ratio from this study and update this information as future evaluation results become available. Because the program is still relatively new, the composition of participants may change over time, so the NTG ratio may change as the program matures. In addition, the NTG ratio should be updated if there are changes in the implementation strategies that might reduce or alter the free-ridership (e.g., increasing incentive levels or changing the measure mix).

C&R: MF-WB Conclusion 3: The consumption analysis did not result in comprehensive energy use for many of the sampled properties. This is due to challenges linking the meter numbers to IOU billing data and considerable time periods with zero energy use during the pre-program period. As such, the evaluation team could not calibrate the simulation models to the estimated consumption as planned, and relied upon the consumption estimates calculated in the simulation models. Recommendation 3: Program administrators need to access and calculate whole building consumption for projects prior to approving project application and have this information readily available for evaluators to justify savings claims. Program administrators should access at least 12 months of gas and electric use prior to potential program upgrades, and 12 months of use after the upgrades occur. These data need to encompass all common area and dwelling units within the participant property and should be a prerequisite of participation. These data will allow savings assumptions and models to be calibrated and/or verified through actual customer bills and will be imperative to support future claims for projects utilizing an existing conditions baseline.

C&R: MF-WB Conclusion 4: IOUs should discontinue use of the EnergyPro RES PERF model for their savings estimates because concerns about the accuracy of this software have led to it to be dropped from the CPUC list of approved simulation model software for the California single-family whole building programs. Recommendation 4: Consider the use of the EnergyPro NR PERF model with inputs that reflect building and use characteristics of multifamily projects in future program cycles. Conclusion 5: The IOU data collection and tracking systems were greatly improved from the 2013–2014 multifamily evaluation, with near complete information on property and measure details. For several projects, however, the energy estimates and savings from energy models submitted by the IOU did not match to the tracking data. Recommendation 5: Continue to review tracking data and energy model results before submitting IOU models to the evaluation team to ensure they match one another.

C&R: MF-WB Conclusion 6: Some projects had incentivized measures that did not exceed Title 24 prescriptive requirements. For example, Title Standard Section 150.2(b)1B requires replacement fenestration to meet prescriptive requirements in Table A and some projects installed windows that did not meet these, according to project documentation. These projects were negatively impacted when adjusting the baseline to the proper code. Additionally, one project did not install a measure according to the evaluation surveyor and the IOU documentation did not include photos of the measure claimed as installed. Recommendation 6: Require project submittals to include Title 24 compliance documentation for project retrofits to building envelope and mechanical systems to demonstrate that the project at least meets the required prescriptive Title 24 Code. Additionally, the certified performance rating certificates for windows (NFRC), HVAC (AHRI), and DHW (AHRI) equipment documenting the efficiencies at least meet code requirements should be included in project documentation. IOU staff should take photos of the NFRC ratings affixed to manufactured windows during the IOU test-out QC inspections. This may require closer coordination with the construction schedule so the labels are not removed prior to the inspection. Additionally, IOU staff should include a site measurement of solar transmission for verification of low-e glazing when NFRC labeling data is not available. Photo documentation of all installed measures should be included in the IOU final documentation.

Questions?

EVALUATION OVERVIEW: MFEER Lighting

Evaluation Activities
Verify gross and net savings inputs Compare MFEER Lighting Tracking Database savings values to DEER or work paper Assess accuracy of NTG assignments (Re)assign EUL & interactive effects, as needed Calculate Realization Rates (RR)

Data Sources MFEER Lighting Tracking Database
Database for Energy Efficient Resources (DEER) IOU Workpapers

Savings Prioritization
Installed measures should first be consistent with the applicable version of DEER (in the case of the program year, the DEER 2014 update). For those measures that are not in DEER or vary from DEER assumptions because of installation (e.g., common area installation and only in-unit installation were available in DEER for the same measure) or other notable differences, the IOUs should use Lighting Disposition values. If savings and EUL’s are not available from either of these two sources, the IOUs may default to approved workpapers, but they need to provide documentation explaining why they diverged from DEER. The evaluation team learned to use the following hierarchy of data sources in assigning savings values: 1) DEER 2) Lighting disposition 3) IOU workpaper

Program Overview There were 485,742 lighting measures (bulbs, fixtures, sensors, etc.) claimed SCE had the largest lighting contribution (93% of units, 94% of kWh savings) IOU Number of Units kWh Savings kW Savings n= 485,752 17,478,263 827 PG&E <1% SCE 93% 94% SDG&E 7% 6% Total 100% Per above earlier slide, Lighting dominated the ex ante electricity savings 69% for PG&E, 88% for SCE, and 64% for SDG&E

Savings Verification

Savings Verification The Evaluation Team looked at what data sources the IOUs were using to assign saving values Version Source PG&E Ex ante kWh SCE Ex ante kWh SDG&E Ex ante kWh D13 v1.0 (DEER) 69,560 822,064 306,872 Lighting Disposition 379,002 IOU Workpaper 15,557,191 317,599 Not Provided (NULL) 372 25,604 Total 69,932 16,404,858 1,003,472 The evaluation team learned to use the following hierarchy of data sources in assigning savings values: 1) DEER 2) Lighting disposition 3) IOU workpaper

Savings Verification The Evaluation Team was able to assess the majority (87%) of claimed savings values 13% of claimed savings could not be evaluated Updated 80% of the claimed savings that the Evaluation Team was able to assess (70% overall)

Net-to-Gross Verification

The Evaluation Team reviewed accuracy of the NTG assignments The majority of claims were using inaccurate NTG assignments

The weighted average NTG value dropped from 80% to 55% after The Evaluation team made NTG assignment adjustments NTG Assignment NTG Value Percent of Savings Assignments (kWh) Ex Ante Ex Post Residential Default 55% 16% 98% Commercial Default 60% 2% Constrained-Area Program 85% 78% <1% Emerging Technology 3% 0% Hard to Reach Weighted Average 80%

EUL Evaluation

EUL Evaluation The review resulted in a low-proportion of measures receiving updated EULs The overall difference between ex ante and ex post EUL was 2.4 years IOU Ex Ante EUL Ex Post EUL PG&E 14.3 12.4 SCE 13.0 10.5 SDG&E 13.9 Overall 13.1 10.7

Interactive Effects

Gross Interactive Effects (Therms)
Negative therm savings were updated to reflect DEER-based values The overall interactive effects realization rate was 127% IOU Gross Interactive Effects (Therms) Ex Ante Ex Post Realization Rate PG&E -281 -287 102% SCE -103,711 -133,005 128% SDG&E -5,580 -5,578 100% Overall -109,572 -138,870 127%

Gross Savings Findings

Low proportion of measures receiving updated savings values; however, these measures represented high-proportion of savings claims Overall kWh RR was 105%, demand RR was 142% IOU Gross Energy Savings (kWh) Gross Demand Savings (kW) Ex Ante Ex Post Realization Rate PG&E 69,932 71,172 102% 1.31 1.46 118% SCE 16,404,858 17,283,242 105% 778.27 1,126.49 145% SDG&E 1,003,472 1,003,484 100% 46.93 Overall 17,478,263 18,357,898 826.51 1,281.16 142%

The majority of lighting records received a 100% electric energy (kWh) RR Overall kWh RR was 105%, demand (kW) RR was 142%

Net Savings Findings Overall net kWh RR was 73% Net IOU Ex Ante kWh
Net IOU Ex Ante kWh Ex Post kWh kWh RR PG&E 40,015 39,144 98% SCE 13,224,829 9,551,070 72% SDG&E 669,315 569,303 85% Total 13,934,160 10,159,518 73%

Conclusions & Recommendations

C&R: MFEER Conclusion 1: A number of measures are included in the DEER database, yet the IOUs used workpaper savings values. Recommendation 1: IOUs should use DEER savings values for all applicable measures to make the ex ante savings more closely align with the ex post values. For measures not included in DEER, IOUs can continue to use approved workpaper values. Conclusion 2: SCE incorrectly assigned the vast majority (78%) of measures to a constrained area NTG. Constrained area NTG is only applicable for approved zip codes and must show an increased incentive to qualify. Recommendation 2: IOUs should apply the residential default NTG value unless the measures qualify for this increased NTG value.

Questions?

Thank you!

Contact Information Apex Analytics: Katie Parkinson (303) , x102 Scott Dimetrosky (303) , x101 CPUC: Tory Francisco (415)

2015 Multifamily Programs Focused Impact Evaluation

Similar presentations

Presentation on theme: "2015 Multifamily Programs Focused Impact Evaluation"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

2015 Multifamily Programs Focused Impact Evaluation

Similar presentations

Presentation on theme: "2015 Multifamily Programs Focused Impact Evaluation"— Presentation transcript:

Similar presentations

About project

Feedback