Reliability Coordination Task Force

Reliability Coordination Task Force
Report to Finance and Audit Committee December 5, 2012 Presented by Members: Tom Botello - Southern California Edison Terry Baker - Platte River Power Authority

Report Agenda FAC Assignment Summary of the Issue/Finding/Conclusions
Real-time Operations Operations Support Western Interconnection Synchrophasor Program (WISP) Compliance Training Recommendations

Reliability Coordination Task Force (RCTF) Members
Terry Baker Tom Botello John Cupparo Graham Edwards John Meyer

RCTF 2012 Working Sessions July Kick-off meeting in Portland, Oregon August In-person meeting in Denver August Meeting by Webinar October In-person meeting at the Vancouver RC November Meeting by Webinar November Meeting by Webinar November Meeting by Webinar December In-person meeting in Salt Lake City

FAC Assignment RCTF to examine the following:
Original scope of RC duties and expenses Current scope of duties including expanded duties since 2009 Future scope of duties and expenses Analyze key drivers for operational and budgetary change, WECC’s response to those drivers, and provide recommendations

Existing WECC RC Functions and Current Staffing
Operations Planning Current - day/Real-time Support Next-day Study Data Validation and Mapping Seasonal SOL Coordination and Validation Emergency Plan Coordination Real-time Operations Monitor and analyze real-time system conditions Coordinate system restoration activities Proactively respond to and mitigate potential adverse impacts to the BES Communicate with and direct corrective actions of registered entities during actual SOL exceedances Identify and mitigate potential/actual IROLs in real- time operations horizon Compliance Document and Evidence Management Audit Preparation Event Analysis Standards Development Compliance Reporting Disturbance Reporting Operations Support Application Architects Application Support DTS Support EMS Modeling RAS Engineer Architecture Support Communications IT Infrastructure Systems Administrator WISP Broaden deployment of phasor measurement units throughout the Western Interconnection Implement telecommunication network necessary to support a large- scale production synchrophasor system Install synchrophasor applications for Western Interconnection Increase ability to integrate variable generation Enhance power system analysis capability/model validation Training Develop/Deliver in-house training to new RC trainees and current RC staff to ensure compliance with mandatory standards and retention of NERC RC System Operator certification Conduct just-in- time training for new tools, applications, procedures, and processes Update job task analysis Develop/deliver NERC-certified restoration training to BAs, TOPs, and GOs Other Direction and Oversight Administrative Support WECC and NERC Committee Work WECCRC.org Development Membership Request Management Existing WECC RC Functions and Current Staffing Current FTE – Filled Current FTE Approved – Unfilled 82 16 Totals

Summary of the Issue FERC/NERC Requirements
Highest level of authority responsible for the reliable operation of the BES Authority to prevent or mitigate emergencies in next-day analysis/real-time Wide Area view of BES (situational awareness) Tools, processes, and procedures Broad purview to enable IROL calculation(s) that may be based on parameters of transmission systems beyond any TOPs vision Taken from the NERC Glossary of Definition for an RC

Summary of the Issue Reliability Coordination in the Western Interconnection Past – Reliability through collaboration Hosted by utilities – CAISO, BPA, WAPA Use of facilities, tools, and personnel Present – WECC’s Responsibility Independent facilities, tools, and responsibilities Compliance obligations questioned Reliability vulnerabilities exist Future – Implement RCTF Recommendations Increase staffing in critical areas Meet regulatory compliance requirements Reduce reliability vulnerabilities To look forward we must look to the past. Since 2009, the RC’s roles and responsibilities have increased dramatically, driving the need for additional staffing. WECC RC is expected to meet changing and emerging regulatory obligations, requirements, and stakeholder expectations. Past RCs: Each RC had its own model and limited awareness of operating conditions outside of its RC Area (highly susceptible to seams issues). Limited monitoring outside of 345-kV and 500-kV systems Limited awareness of RAS State estimators: while key applications for maintaining situational awareness, have not always been reliable Very limited number of high voltage contingencies defined, certainly not contingencies outside the RC footprint Current RC: Loveland and Vancouver Reliability Coordination Offices operating as “One RCO” with a single model and identical set of tools. Evolution of the RC Function: Monitoring of SOL/IROL Former RCs: Western Interconnection had no IROLs under normal, studied conditions. Primarily focused on major WECC Transfer Paths identified in TOP-SDT (approximately 40 paths). Current RC: Monitors established SOLs for all equipment > 100 kV and select lower-voltage facilities. Summary data from LNLIM table: Among the LNLIM records, there are 8661 records with KV>=100, excluding all in series CAP banks. 8520 of them are set “Eligible” for monitoring. Among 430 ZBR records, there are 142 ZBR records with the limits. Among 4506 XFLIM records, there are 1422 being set “Eligible” for monitoring, including a few 92-kV XF in Coachella area. Only the question is not sure how many are three-winding XFMR. IROLs exist within the West and must be identified in the operations horizon – including real-time, current-day, and next-day operations. Future State: Improved processes for determining IROLs; additional awareness of sub-100-kV facilities; and greater dependency on RTCA to determine acceptable pre- and post-contingency operating conditions. Monitoring of Disturbance Control Standard (DCS) Former RCs: Limited role in Disturbance Control Performance. Primarily focused on RSG rather than BA performance. Present RC: Monitors individual BA parameters to evaluate Disturbance Control Performance and identify reserve adequacy issues. RC takes a proactive approach to identifying DCS events, determining viable mitigation and directing corrective action as required. Future State: additional awareness of reserve deliverability and sustainability and improved processes for determining viability of mitigating actions. Monitoring of System Frequency and BA Performance Former RCs: Monitored system frequency and BA performance to identify system events or performance issues that could adversely impact the BES. Present RC: Monitors system frequency to identify exceedance of Frequency Trigger Limits established in RBC field trial. Monitors BA performance to identify large ACE that may be contributing to SOL/IROL exceedance and/or Unscheduled Flow. RC takes a proactive approach to identifying BA performance issues that could adversely impact neighboring BAs and BES operations (RBC field trial – RC has unquestioned authority to identify operating issues that may be attributed to BA performance) Future State: improved tools and processes for identifying BA performance issues that should be corrected to limit adverse impact to adjacent entities.

Summary of RCTF Findings
Additional personnel are recommended in the following areas: Real-time operations Broad purview to enable IROL calculations (granularity, specific focus) Additional real-time tools, processes, procedures Operations support Creation of operating plans, procedures (seasonal and next-day studies) System Modeling and model validation IT application support requirements Compliance support Training

Summary Conclusions WECC has a high level of responsibility and is at risk for reliability and compliance issues Staffing is a major factor Tools are a major factor Emergent tools and Standards will add to these risks WISP-related displays, data, training PER-005 and EOP-005 WISP will provide the RC with a more granular view of the Western Interconnection but will also significantly increase the amount of data that the on-shift RC System Operator must assess. This poses a risk if the current staffing for the RC does not increase to implement a three-region approach. To do this we need to add two 24/7 RC System Operators and one additional 24/7 Model Engineer for a total of 18 FTEs. The Enhanced Curtailment Calculator (ECC) is not budgeted nor is the budget included in this analysis. The WECC Operations Committee is working with the WECC RC to determine the scope and potential cost of implementing the ECC and that will be put before the Board in a separate request.

Real-Time Operations RC Contingency Analysis
Four RC System Operators on shift on September 8, 2011 Prior to loss of North Gila-Hassayampa 500-kV line, there were 287 harmful contingencies identified by RTCA RTCA run following line loss identified 632 harmful contingencies Between start of event and SONGS units tripping (less than 11 minutes) 1,185 alarms were generated Of the 287, roughly 180 were INTRFC violations (INTRFC violations = Path Limit Exceedance) Key Takeaway – The RC System Operator must evaluate each harmful contingency to determine whether corrective action is required. The RC System Operator must remain aware of expected post-contingent conditions to be prepared for the next credible contingency. Risk – Identifying and addressing such issues reduces the RC’s vulnerability to major system disturbances and potential non-compliance. WISP impacts system awareness New data from the synchrophasor Relevant training and tools will be required, as well as consideration of the effect on the support side More than 400 new or upgraded Phasor Measurement Units (PMU) have been or will be installed as part of the WISP project. The sub-second measurements supplied by PMUs will provide the RC with additional data to analyze real-time operating conditions. Phasor data will enhance the RC’s ability (or in some cases, provide it for the first time) to: Monitor inter-area modes for dangerous low system damping conditions, and identify high energy oscillations that could impact the BES Monitor phase angles, reactive reserves, and voltage stability to; More accurately estimate the state of the system by allowing the use of phase angle measurements directly in the state estimator algorithm Identify abnormal or emergency conditions that require corrective action to; Reduce potential for major system disturbances and widespread outages. The RC must introduce new applications and enhance existing tools to integrate phasor data into the RC purview. Real-time Operations and Operations Support staff must be trained to take full advantage of this new grid-monitoring technology.

Real-Time Operations Similar events to SW Outage reveal RC real- time staffing concerns Ability to monitor current system specifics questioned by regulators Increase in required monitoring of RAS Emergent tools such as WISP Sub-100-kV facilities Specific interconnection area knowledge by RC System Operators needed to meet requirements RC phone calls per quarter have increased to over 37,000

Real-Time Operations Key Takeaway - Additional FTEs will help identify and address reliability issues that require RC System Operator action. Risk –Identifying and addressing such issues reduces vulnerability to major system disturbances and potential non-compliance. 2012 – totals include December projections

Vancouver RC Total Load: 104506 MW Total BAs: 21 Total TOPs: 29
Loveland RC Total Load: MW Total BAs: 16 Total TOPs: 25 Compare the load area responsibility for the VRC and LRC. While the number of TOPs are close for both control centers, the VRC has responsibility for more WECC-Rated Paths and those with the most loading issues.

Vancouver RC Total Load: 49666 MW Total BAs: 16 Total TOPs: 17
CA/MX RC Total Load: MW Total BAs: 6 Total TOPs: 13 Loveland RC Total Load: MW Total BAs: 15 Total TOPs: 24 The WECC RC is transitioning from a two sub-region model to a three sub-region model. This transition is needed: to accommodate the significant increase in workload tied to the inclusion of sub-100-kV facilities to accommodate the increase in SOL and IROL responsibilities; and to respond to concerns expressed by FERC and NERC with regard to the adequacy of existing real-time staffing under the current workload. Transitioning to a three sub-region model will require twelve additional RC System Operators and the addition of a modeling engineer. This transition will also require a corresponding increase in IT and Applications support. Additional facilities will be required to house the FTE increase; however, the renovation of the control rooms in Vancouver and Loveland will be accomplished with grant funding and the only additional budget requirement is for more office space in existing facilities. This model (the three sub-region approach) balances the workload.

Real-Time Operations - Documentation
The RC must create and maintain operating plans, procedures, and processes This documentation must: Identify RC System Operator actions taken to address specific system conditions and events Reflect compliance to mandatory NERC, FERC, and Regional requirements, policies, and procedures Currently no specific position to perform this function Key Takeaway – The RC does not have an FTE focused on maintaining, updating, and publishing operational documentation. This documentation is being accomplished by Managers and Directors working extra hours. Risk – Failure to create and maintain operating plans, procedures, and processes poses a potential reliability and compliance risk. Talking Points: 22 Operating Procedures, 16 are in need of revision to reflect system changes that are pending. Two Operating Plans, both are in need of updating. Three Operating Guidelines, all in need of revision.

Real-Time Operations − Findings
Additional personnel are recommended in order to: Monitor the Western Interconnection to the level of granularity expected by FERC, NERC, and the NPCC Respond effectively to the volume of alarms and RTCA results during an event to determine appropriate corrective action Create and maintain Operating Plans, Procedures, and Processes as required by NERC Standards NERC, in their compliance audit of the WECC RC, and FERC/NERC in their Joint Report on the September 8, 2011 event, indicated an expectation that the WECC RC will have more in-depth knowledge of the Transmission Operators’ systems. For example: know the impact of re-dispatch on line loading, know the TOP’s mitigation plans for contingencies and if those plans are viable, know the impact of removing lines from service, even at the sub-BES voltage operating level. This is challenging, especially for the Vancouver RC with the number of TOPs being monitored.

WECC RC Function and Recommended FTE Increase
Real-Time Operations Monitor and analyze real-time system conditions Coordinate system restoration activities Proactively respond to and mitigate potential adverse impacts to the BES Communicate with and direct corrective actions of registered entities during actual SOL exceedances Identify and mitigate potential/actual IROLs in real-time operations horizon 33 14 FTE Totals FERC and NERC have clearly stated their expectation that the WECC RC will establish a broader and deeper real-time view into the Interconnection, including sub-100-kV elements that may impact reliability. This impacts more than the study and assessment processes as RC System Operators will now be required to monitor current and expected post-contingent states for these elements, as well as associated increases in potential contingencies. This increases alarm management, logging, and notification and information dissemination responsibilities as well as increasing the data points of which the RC System Operators must be aware. The WECC RC is transitioning from a two sub-region model to a three sub-region model. This transition is needed: to accommodate the significant increase in workload tied to the inclusion of sub-100-kV facilities to accommodate the increase in SOL and IROL responsibilities; and to respond to concerns expressed by FERC and NERC with regard to the adequacy of existing real-time staffing under the current workload. Transitioning to a three sub-region model will require twelve additional RC System Operators and the addition of a modeling engineer. This transition will also require a corresponding increase in IT and Applications support. Additional facilities will be required to house the FTE increase; however, the renovation of the control rooms in Vancouver and Loveland will be accomplished with grant funding and the only additional budget requirement is for more office space in existing facilities. FERC, NERC, and NPCC expect that upon the occurrence of an event, disturbance, or limit violation that impacts reliability in the Western Interconnection, the WECC RC will have adequate information and analysis capabilities to evaluate potential mitigating activities and direct specific, detailed responsive action as necessary to return the system to an acceptable operating state as soon as possible. As a result of a system disturbance on February 14, 2008, FERC and the WECC RC entered into a settlement agreement that requires implementation of Reliability Enhancement Measures (REM). This includes REMs targeted specifically at meeting FERC expectations with regard to WECC RC directives. To enable the WECC RC to take timely action that not only meets regulatory expectations but also supports TOPs and BAs in mitigating emergency or abnormal conditions, the WECC RC is developing additional tools, such as the Line Outage Distribution Factor Calculator. To fully implement the REMs the WECC RC added a shift lead Reliability Coordinator position. These positions, while not currently fully staffed, will provide the RC System Operators with guidance and direction for actions taken on shift, particularly during system disturbances and unusual events. The RC shift lead will ensure that internal and external notifications are made in accordance with established procedures and will work closely with the RC management team to support the rollout of new procedures and re-enforce training for newly implemented tasks. Implementing the REMs also resulted in the addition of a shift operations engineer who is primarily responsible for ensuring the quality of the state estimator and contingency analysis solutions, review of IROL conditions, and review of the viability of mitigation plans. Current FTE – Filled Current FTE Approved – Unfilled Future FTE Projection

Operational Planning (Seasonal)
SW Outage event and regulatory findings revealed seasonal planning gaps Not all WECC paths studied seasonally TOPs coordination weak Incomplete WECC contingency analysis Mitigating reliability vulnerabilities With an enhanced seasonal process and more thorough assessments, the event may have been predicted Key Takeaway – TOPs are waiting for the RC to take the lead. The RC needs to establish processes and coordination techniques supported by additional FTEs to improve the seasonal study process. Risk – Without new FTEs, coordination, and processes; seasonal studies will continue to miss issues similar to September 8th that can be mitigated.

Operational Planning (Next-Day)
Why did the RC next-day study not see the problem that caused SW Outage event? Only study next-day peak Study automation incomplete Need to assess sub-100-kV facilities Forecast data quality inputs need improvement and are too numerous Number of scheduled outages continues to increase A million rows of unit commitment data daily Key takeaway – The RC needs an improved process to ensure the next-day study is high quality. Need adequate software and FTEs to improve processes and data quality. Risk – Without adequate software and FTEs, the RC will be no better equipped to mitigate in the day-ahead time frame for September 8th-type issues.

Operations Planning - RCTF Findings
Additional personnel are recommended to perform the following activities: Seasonal Planning: Take lead in implementation of an enhanced process with more thorough assessments Participate in the resolution of identified gaps in the WECC Regional process Next-Day Study: Complete study automation Assess sub-100-kV facilities Ensure quality of data inputs

Operations Planning Current -day/Real-time Support Next-day Study Data Validation and Mapping Seasonal SOL Coordination and Validation Emergency Plan Coordination 9 5 11 FTE Totals Current FTE – Filled Current FTE Approved – Unfilled Future FTE Projection

Operations Support – System Modeling
Model Impacts to Situational Awareness Three ways to improve accuracy of RTCA: Increase accuracy of facility ratings and SOLs Increase number of RAS modeled Expand modeling of lower-voltage networks that impact the BES Sub-100-kV systems that parallel higher voltage networks Sub-100-kV systems with both automatic and manually-switched reactive devices Key Takeaway – Modeling resources are required to ensure proper model validation with TOPs as well as processes to ensure improved sub-100-kV modeling. Risk – TOPs know their system well and without a process to review and validate the model, the RC System Operators won’t have awareness of the impacts of the lower-voltage facilities. The RC is working to improve its ability to access sub-100-kV systems for the network model. More FTEs to review/validate the model translates to an increased awareness of impacts of lower-voltage networks and other facilities that impact the BES.

RAS Engineer and EMS Modeling – WECC RC System Operators are required to improve awareness of the impacts of RAS Continuing increase in the number of WECC RAS A key component of improved awareness – RAS modeling in the EMS and real-time contingency analysis (RTCA) Only 45 of 206 RAS modeled in RTCA Key Takeaway – Lack of RAS modeling requires RC System Operators to manually study expected RAS actions and decreases RTCA accuracy. Risk – RC System Operators may misinterpret post-contingency issues identified by RTCA due to a need for additional RAS modeling. RTCA results can be improved with an increase in RAS modeling. RAS modeling is critical to ensure the accuracy of the next-day studies because RAS may only be armed for certain operating conditions, RAS operation can significantly impact real-time operations as was the case of the IID RAS when it ran back the CFE generation.

Key Takeaway – Lack of RAS modeling requires RC System Operators to manually study expected RAS actions and decreases RTCA accuracy. One quarter of the total have been modeled. Risk – RC System Operators may misinterpret post-contingency issues identified by RTCA due to a need for additional RAS modeling. RTCA results can be improved with an increase in RAS modeling. There are two charts shown: RAS modeled vs. Identified WECC RAS – this chart shows the number of RAS that are modeled in the WECC RC contingency analysis application. RAS Real-time Data Points in EMS – this chart shows the number of real-time ICCP data points associated with RAS. What is the difference? Both of the charts use the WECC RASRS RAS list that at the time of slide development had 206 RAS on the list (Terry mentioned that had since increased). The data in chart #1 shows RAS modeled in contingency analysis. For example, if there is a RAS that drops generation X for loss of line Y, when the contingency runs that takes out line Y, contingency analysis will automatically drop generation X. That is what is meant by a RAS model in contingency analysis. The WECC RC has 45 of these scenarios modeled in our contingency analysis application. The data in chart #2 shows how many RAS data points we are bringing into our EMS via ICCP. For example, a RAS that has multiple generators associated with it for gen dropping might have multiple ICCP points – typically an arming status point for each generator associated with the RAS. The WECC RC has just over 100 RAS (roughly half of the RASRS-listed 206) that the RC is receiving ICCP data for. The “Minimum Expected RT Points” reflects the number of ICCP points the RC should expect to have in the EMS when it has data for all 206 RAS. The RC currently has just under half so the chart reflects just under half of the total RAS points.

Key takeaway – model has improved significantly over time and is helping the RC be much more predictive of potential problems. Need to continue and improve ability to correct identified problems. Risk – Unaddressed issues can negatively impact the RC’s ability to monitor the power system.

Addressing Model Deficiencies WECC RC needs to be more proactive in the modeling process by increasing the frequency of model reviews with TOPs West-wide System Model and Base Case Reconciliation Task Force (WBRTF) Develop process for periodic review and sign off of model accuracy Add more power system measurements Share data and application results Key Takeaway – Modeling resources are required to ensure proper model validation with TOPs as well as processes to ensure improved sub-100-kV modeling. Risk – TOPs know their system well and, without the process to review and validate the model, the RC System Operators won’t have awareness of the impacts of the lower-voltage facilities. The RC is working to improve its ability to access sub-100-kV systems for the network model. More FTEs to review/validate the model translates to an increased awareness of impacts of lower voltage networks and other facilities that impact the BES.

Operations Support - Applications and IT
Applications and IT areas of responsibility include, but are not limited to: coordinated outage scheduling EMS system support load and interchange forecasting operating system maintenance change management cyber security administration networking

Application Architects and Support areas of responsibility include, but are not limited to: support planning of IT initiatives to evaluate and determine new tools, technology, and integration with external drivers such as SAFNR or data requests attend working groups review new software releases set system direction upgrade paths ensure a timely response for application support ensure continued compliance with new CIP Standards RC startup did not include FTEs to plan the direction of applications after initial start Backup staff or cross-training is not optimal for supporting the 30-minute compliance response times required for application support Taking the necessary actions for ensuring compliance with new CIP standards will require additional FTEs To meet standards compliance imposed timeframes, more FTEs are required

IT and Applications increases since 2009: Number of applications increased from seven to 16 Number of users increased from 15 to 34 As a cost savings measure, RC tool applications have been developed in-house Creating a burden: volume of custom applications requiring on-going maintenance have increased Number of developed applications increased from two to 53 Support staff increased from five to six An early decision was made to have all applications windows based thus reducing the number of operating platforms that had to be supported. While this proved to be an effective strategy, it also meant that some applications had to be developed in-house. Overall it is more efficient in terms of staffing to use one operating platform even though there is a side-effect of needing to support custom applications. The number of users (increased from 15 to 34) only counts the actual number of RC staff on duty. It does not count the Reliability Coordinators who are off shift. This count includes: Reliability Coordinators on-shift, Modeling Engineers, EMS Engineers, Planning Engineers, Study Engineers and Training Staff. Application Support Number of Users Supporting 2009 — 4 On-Shift RC, 7 Engineers, 4 RC Mgr-Lead 15 2010 — 4 On-Shift RC, 14 Engineers, 2 Trainer, 4 RC Mgr-Lead 24 2011 — 4 On-Shift RC, 16 Engineers, 4 Trainer, 4 RC Mgr-Lead 28 2012 — 4 On-Shift RC, 21 Engineers, 5 Trainer, 4 RC Mgr-Lead 34

Operations Support - IT Infrastructure

Operations Support - IT Infrastructure
In 2009 average was 40 servers/IT employee Today, it is over 112 servers/IT employee WISP represents cutting-edge technology The RFP responses did not include development of some of the applications because they were not commercially available Requires WECC staff to develop and maintain custom software for the WISP program The total workload increase in hardware, software, and applications is about 35 percent The WECC RC databases are all Microsoft SQL technology and have grown in complexity and size There will be a 40 percent increase in the volume of data storage in SQL databases with the Synchrophasor systems Adding an FTE to this function will allow proper database maintenance as well as provide a resource for internal and external data mining requests Adding an FTE will assist in: Maintaining the model analysis and oscillation detection systems, provide model validation analysis, state estimator analysis, and Voltage Stability Analysis Voltage Stability Analysis will be integrated with 1500 points of synchrophasor data that will be maintained and tuned

WISP − IT Infrastructure
WISP is cutting edge technology WECC staff to develop and maintain custom software for the WISP program There will be a 40 percent increase in the amount of data storage in SQL databases with the synchrophasor systems Voltage Stability Analysis will be integrated with 1500 points of synchrophasor data that will be maintained and tuned Key Takeaway WISP includes an entirely new suite of tools for the RC to use when making decisions regarding grid security. The wide area view (WAV) tool and the wide area situational awareness tool (WASA Alstom e-terravision). This is cutting edge, non-tested technology for which the West is the front runner on installing and using for BES security. Two new data centers were built to support WISP and, with the addition of PI displays along with WAV and WASA, increased IT and application support person is a necessity. This is critical because the WAV and historical tools are also accessible to all the BAs, TOs, and TOPs who have signed the Universal Data Sharing Agreement – which means not only the RCs are relying on the data to make operating decisions or conduct forensic analysis. The WISP total cost of ownership includes 11 FTEs that support the technical components of WISP but do not cover all the components.

Operations Support − RCTF Findings
Additional personnel needed in the following in order to: Identify and address model deficiencies Make necessary improvements to the model Ensure proper model validation with TOPs Model all WECC Region RAS and real-time RAS data points in the WECC RC EMS Provide maintenance and support for an increasing number of users, applications, and servers Ensure compliance What is missing is a process where TOPs and BAs have reviewed data and WECC has their buy-in.

Operations Support Application Architects Application Support DTS Support EMS Modeling RAS Engineer Architecture Support Communications IT Infrastructure Systems Administrator 20 3 12 FTE Totals IT infrastructure, applications, and hands-on compliance support has significantly increased without a commensurate increase in FTEs to manage the work load. In addition, to avoid large initial and on-going contract maintenance vendor costs, the IT and Applications staff have developed required applications in-house. These applications require FTEs to maintain, upgrade, and integrate into other RC tools. In addition, the hardware and software systems the RC purchased in 2008 have a three-to-five-year upgrade/refresh cycle. This work requires FTEs to manage. To implement changes to the applications in a “live” system that is an identified Critical Cyber Asset, there are significant requirements for CIP compliance and the need to install upgrades in stages, verify each change, and implement the changes without adversely impacting real-time operations, which is resource intensive. Detailed documentation is required for CIP from the IT and Applications staff every time they work on the IT or applications systems, whether it is installing new systems, upgrading existing systems, testing systems per required maintenance schedules or trouble shooting problems, has grown since the RC went live on Jan 1, 2009. Current FTE – Filled Current FTE Approved – Unfilled Future FTE Projection

Compliance WECC RC Compliance staffing has experienced high turnover in the last three years As of 11/26/12 only one staff position is dedicated to Compliance Compliance monitoring and documentation undertaken by upper-level RC management resulting in additional work often outside normal duty hours March 2012 WECC BOD Resolution: Resolved, That the Western Electricity Coordinating Council (WECC) Board of Directors (Board) strongly believes in a culture of compliance, and that WECC should set the highest standard for this culture in the Western Interconnection. This culture and associated performance starts at the Board and should include every employee, consultant, and supplier to the company. The Board directs the Chief Executive Officer to lead WECC in exemplifying and instilling this culture of compliance. Proving compliance is as critical as being compliant consequently there is a need for adequate support to maintain documentation for compliance records. This is currently being undertaken by the managers and directors with assistance from WECC legal. For example, post event analysis requires the gathering of all data such as event logs, screen shots, voice recordings and other evidence necessary to first analyze compliance and second to document non-compliance or compliance. All of this falls currently falls to the control center managers and is typically accomplished by working long hours. This also takes the managers focus away from real-time operations and puts them into a reactionary mode which isn’t conducive to good management practices. RC is a Registered Entity subject to applicable reliability standards Increased regulatory scrutiny and expectations – FERC, NERC, and NPCC (Sept. 8, Report and NPCC Audit) The RC’s expanded scope and responsibilities demand enhanced compliance programs: developing compliance expertise enhancing capability to meet reliability and compliance responsibilities adding FTEs with compliance expertise for both CIP and O&P Risks Fail to meet Board expectations Subject to civil penalties and sanctions for non-compliance (Violations) with standards Damage credibility with FERC, NERC, and NPCC -- and with other Entities in West In preparation for the compliance audit of the RC a contractor performed a mock audit of the RC for CIP compliance. Following the mock audit, the WECC CEO chose to perform an Analysis on the RC Compliance Program to be accomplished by WECC Compliance Staff. As a result of that analysis a new position was added in WECC corporate and Tom Schneider was selected to lead WECC Corporate Compliance reporting to Steve Goodwill, General Counsel. At that time it was decided to move two vacant positions and one encumbered position from the WECC RC into corporate compliance reporting to Schneider. As the need for compliance support for the RC was analyzed further the decision was made to move two of the three positions from Corporate Compliance back to the RC. On November 26, 2012 a Lead Regulatory Compliance Specialist was hired within the RC. Currently there is one filled position, one unfilled position and one FTE being recommended for the RC. The budget for the three positions was moved from the RC to corporate and has not been moved back to the RC as of November 27, 2012.

Compliance - RCTF Findings
The WECC RC increased staffing will improve the ability to: Meet the March 2012 WECC Board Resolution that established clear compliance expectations for WECC Meet increased compliance and reliability responsibilities Meet increased regulator expectations Mitigate increasing regulatory and financial risks

Compliance Document and Evidence Management Audit Preparation Event Analysis Standards Development Compliance Reporting Disturbance Reporting 1 FTE Totals The compliance staff is responsible for: maintaining evidence of compliance, typically in the form of RSAW’s (Reliability Standard Auditor Worksheets). They will prepare the documentation (tapes, screen shots, logs, etc.) for analysis of compliance following each event and directive. Preparation of mitigation plans, self reported violations, self certifications and audit preparation, and management Serve as subject matter experts advising other RC staff when implementing changes such as replacement of CCAs, new facilities for CIP-006 compliance, drafting of policies and procedures to ensure that they are compliant. Current FTE – Filled Current FTE Approved – Unfilled Future FTE Projection

Training Reliability Coordination Trainers DTS Engineers
Training responsibilities are identified in NERC Reliability Standards DTS Engineers Support increase in system restoration drills, exercises, and simulations Assist in developing implementation of new TOP and region-specific restoration training scenarios Key Takeaway – FERC, NERC, and Regional Standards Internal procedures, processes, and plans CIP Training RC System Restoration Training (includes external BA/TOP System Operators, GOP personnel) Support the increase in system restoration drills, exercises, and simulations that will include TOP and GOP staff to meet the new NERC Standard EOP requirements, effective July 2013 The RC has experienced difficulty in recruiting and retaining highly experienced RC System Operators. To attain appropriate staffing levels in this area, it will be necessary to implement a training program designed to bring in operators with a base-line level of knowledge and train them to a level where they are qualified to operate independently as an RC System Operator. In 2012, the WECC BOD approved four FTE for trainee-level positions but these positions have not been filled due to the lack of ability to provide the requisite training. Additional staffing is necessary to develop an evaluation process and training program, and to provide the classroom instruction necessary to fully implement an in-house development program.

Internal -Training Hours
NOTE: Includes projected hours for remainder of 2012

Training - RCTF Findings
Additional personnel recommended to: Provide required training to RC staff Provide required training to regional reliability entities Ensure training documentation is maintained as a provider Ensure compliance with NERC Reliability Standards

Training Develop/Deliver in-house training to new RC trainees and current RC staff to ensure compliance with mandatory standards and retention of NERC RC System Operator certification Conduct just-in-time training for new tools, applications, procedures, and processes Update job task analysis Develop/deliver NERC certified restoration training to BAs, TOPs, and GOs 4 5 6 FTE Totals This slide includes FTEs for four trainees that were approved by the Board in 2012 (as part of the 20 unbudgeted but approved FTEs). The goal is to develop and implement a training program to develop high potential candidates to fill vacant RC System Operator positions. This has not been accomplished due to the turnover in the training department. As the Training Department is staffed, the trainee positions will be filled. The need for the trainee positions will be analyzed annually and, based on expected attrition (due to retirement or unexpected vacancies), may be adjusted down. Should staffing stabilize then no trainees would be needed. Since it will likely take two-to-three years to fully train the trainees, that will be taken into consideration in the staffing level decisions. Current FTE – Filled Current FTE Approved – Unfilled Future FTE Projection

Conclusions WECC has a high level of responsibility and is at risk for reliability and compliance issues Staffing is a major factor Tools are a major factor Emergent tools and Standards will add to these risks WISP related displays, data, training PER-005 and EOP-005 The base budgets for all include a 3 percent inflationary increase. This is in alignment with merit, benefit, license and maintenance agreement cost adjustments and lease contract increases. For the year beginning 2012, the RC hardware and software equipment that was purchased in 2008 began either a three- or five-year refresh and or upgrade cycle. For example, in 2012 the RC installed/upgraded their entire EMS system. The 2012 budget will overrun by $1.5 million dollars. Overrun costs are due to the unbudgeted but approved FTEs (approved by the Board in December 2, 2011). At one point 11 positions had been filled. The RC lead positions were pulled from the on shift RC employees, which has resulted in $300,000 in unbudgeted overtime. Cost includes benefits, salaries, space, equipment, and licenses associated for each FTE. In addition, overrun includes the cost of software purchases required to meet the FERC February 14, 2008 mitigation settlement. The 2013 budget has been approved and includes the total cost of the 20 approved FTEs approved by the Board in December of 2011 as well as the total cost of ownership for WISP (synchrophasor project) beginning the second quarter of 2013 when the DOE grant funding ends. WISP total cost of ownership is for nine months in 2013. The total cost of ownership for the WISP project has been approved by the Board and is $3.5 million per year starting in 2014 through 2016. Hardware refresh costs begin for the WISP data centers in 2017, which escalates the total cost of ownership

RCTF Recommendations Increase monitoring areas from two to three
Add 47 WECC RC FTEs to enhance and improve the capability of the RC as follows: FTEs FTEs FTEs Acquire additional tools as identified in the projected budget

Operations Planning Current-day/ real-time Support Next-day Study Data Validation and Mapping Seasonal SOL Coordination and Validation Emergency Plan Coordination 9 5 11 FTE Totals Real-time Operations Monitor and analyze real-time system conditions Coordinate system restoration activities Proactively respond to and mitigate potential adverse impacts to the BES Communicate with and direct corrective actions of registered entities during actual SOL exceedances Identify and mitigate potential/actual IROLs in real- time operations horizon 33 14 Compliance Document and Evidence Management Audit Preparation Event Analysis Standards Development Compliance Reporting Disturbance Reporting 1 IT Applications, Engineering Support, & IT Support Application Architects Application Support DTS Support EMS Modeling RAS Engineer Architecture Support Communications IT Infrastructure Systems Administrator 12 20 3 WISP Broaden deployment of phasor measurement units throughout the Western Interconnection Implement telecommunication network necessary to support a large- scale production synchrophasor system Install synchrophasor applications for Western Interconnection Increase ability to integrate variable generation Enhance power system analysis capability/model validation 2 Training Develop/Deliver in-house training to new RC trainees and current RC staff to ensure compliance with mandatory standards and retention of NERC RC System Operator certification Conduct just-in- time training for new tools, applications, procedures, and processes Update job task analysis Develop/deliver NERC certified restoration training to BAs, TOPs, and GOs 4 6 Other Direction and Oversight Administrative Support WECC and NERC Committee Work WECCRC.org Development Membership Request Management Current FTE – Filled Current FTE Approved – Unfilled Future FTE Projection 82 16 47 Totals

Budget Projections 1 BASE BUDGET 2012 2013 2014 2015 2016 2017 2 Includes WISP, RC tools, approved FTE's and an across the board 3% inflation increase $14,438,927 $19,024,411 $20,400,136 $21,010,320 $21,588,266 $23,169,891 3 TOTAL DOLLAR INCREASE FROM PREVIOUS YEAR $1,490,803 $3,585,484 $2,375,725 $610,184 $577,946 $1,581,625 4 PERCENT INCREASE FROM PREVIOUS YEAR 10% 19% 12% 3% 7% 5 FUTURE REQUIREMENTS 6 Approved but unbudgeted FTE salaries, loading and set up cost. Not budgeted, staggered hiring in 2012 but included in 2013 base budget. And, over-runs for Operating Expenses and Fixed Assets. $1,520,569 7 18 unbudgeted requested 2013 FTE salary & loading costs. Staggered hiring in Full cost in 2014 and beyond. Includes 3% inflation increase on salaries. $1,648,480 $1,702,759 $1,757,038 $1,811,317 $1,865,596 8 Build out unbudgeted construction cost for 2013 requested 12 FTE. (6 FTE are in real-time control room) $157,500 9 22 requested 2014 FTE salary & loading costs. Staggered hiring in Full cost in 2015 and beyond. Includes 3% inflation increase on salaries. $2,098,286 $2,164,627 $2,230,968 $2,297,309 10 Build out construction cost for 2014 requested 14 FTE (8 FTE are in real-time control room) $183,750 11 8 requested 2015 FTE salary & loading cost. Staggered hiring in Full cost in 2016 and beyond. Includes 3% inflation increase on salaries. $854,394 $878,518 $902,642 12 Build out construction cost for 2015 requested 5 FTE (3 FTE are in real time control room) $65,625 13 We will need to acquire Sq feet of space in 2013 as it becomes available in LRCC and VRCC so we don't lose the option in the current buildings. This is an added $17,000 per month. Assuming full cost beginning in 2013 $204,000 14 Increase each year $2,009,980 $4,188,795 $5,045,684 $5,124,803 $5,269,547 15 Totals for base & future budgets (line 14 plus line 2) $15,959,496 $21,034,391 $24,588,931 $26,056,004 $26,713,069 $28,439,438 16 PERCENT INCREASE FROM BASE BUDGET 17% NOTE DOE expansion of control rooms is $3.5 million dollars, anything outside of the $3.5 million will be a WECC cost. NOTE -- the Enhanced Curtailment Calculator is in 2014 WECC Business Plan but not in the budget because it has not been approved by the WECC OC. NOTE budget is projected to exceed base approved budget by $1,520,569 due to cost overrun in Personnel, Operating Expenses and Fixed Assets. NOTE and 2014 Total Dollar Increase From Previous Year (Line 4) is due to change in WISP funding from DOE to WECC NOTE Total Dollar Increase From Previous Year is due to hardware refresh for WISP support The base budgets for all include a 3 percent inflationary increase. This is in alignment with merit, benefit, license and maintenance agreement cost adjustments and lease contract increases. For the year beginning 2012, the RC hardware and software equipment that was purchased in 2008 began either a three- or five-year refresh and or upgrade cycle. For example, in 2012 the RC installed/upgraded their entire EMS system. The 2012 budget will overrun by $1.5 million dollars. Overrun costs are due to the unbudgeted but approved FTEs (approved by the Board in December 2, 2011). At one point, 11 positions had been filled. The RC lead positions were pulled from the on shift RC employees, which has resulted in $300,000 in unbudgeted overtime. Cost includes benefits, salaries, space, equipment and licenses associated for each FTE. In addition, overrun includes the cost of software purchases required to meet the FERC February 14, 2008 mitigation settlement. The 2013 budget has been approved and includes the total cost of the 20 FTEs approved by the Board in December of 2011 as well as the total cost of ownership for WISP (synchrophasor project) beginning the second quarter of 2013 when the DOE grant funding ends. WISP total cost of ownership is for nine months in 2013. The total cost of ownership for the WISP project has been approved by the Board and is $3.5 million per year starting in 2014 through 2016. Hardware refresh costs begin for the WISP data centers in 2017, which escalates the total cost of ownership.

Questions?

Back Up Slides

Control Room Facilities Comparison
MISO WECC RC Vancouver

Acronym Guide BA – Balancing Authority BES – Bulk Electric System
CIP – Critical Infrastructure Protection DTS – Dispatcher Training Simulator EMS – Energy Management System FTE – Full-time Equivalent IROL – Interconnection Reliability Operating Limit NERC – North American Electric Reliability Coordination RAS – Remedial Action Scheme RC – Reliability Coordinator RTCA – Real-time Contingency Analysis SOL – System Operating Limit SONGS – San Onofre Nuclear Generating Station TOP – Transmission Operator USF – Unscheduled Flow WBRTF – West-wide System Model and Base Case Reconciliation Task Force WISP – Western Interconnection Synchrophasor Project WSM – West-wide System Model

Real Time Operations Subregional strategic shift from two to three areas: More RC System Operators monitoring the system Wide area coordination with specific and targeted increase in focus Dedicated on-shift engineer in each control room to ensure accuracy of data and provide study support Improved and more specific regional procedures, operating guides, and corresponding displays Key Takeaway – Additional FTEs are required to monitor to the level of granularity expected by regulators. Additional staff will help avoid failure to identify and address reliability issues that require RC System Operator action. Creating an additional subregion will reduce each RC System Operator’s area of responsibility and allow for greater focus when monitoring system conditions and addressing system events. Risk – Maintaining the status quo presents a greater challenge in monitoring to the level of granularity expected by the regulators and may result in a failure to identify and address reliability issues that require operator action.

Operations Support - Applications
16 11 10 7 6 6 5 5

34 28 24 15 Risk: Inability to support staff in timely manner, which can affect the RC’s ability to monitor the Western Interconnection 6 6 5 5

Operations Support - RCWorkbook – Monitoring RTCA
CONFIDENTIAL Post-contingent branch, interface, and bus voltage limit exceedances presented. Island contingencies as well as unsolved contingencies (potential IROLs) are presented using this WECC RC developed tool. Key Takeaway – The RCWorkbook is one of the custom applications the WECC RC IT Applications developed and now needs to maintain.

External -Training Hours
Key Takeaway – WECC RC required to provide training to BAs, TOPs, and GOPs Supports BA/TOP/GOP compliance Risk: Non-compliance

Training - External Savings
Training savings to entity participants: calculate each 8-hr day of training at $250 Per Diem $75/day + $125/day hotel Grand Total savings per year for external entities (add col C+D) Year Cost Savings Per Diem +Hotel Grand Total Savings hrs/8 = 236 days x $200 = $47, $106,200 2. 236x$250 = $59,000 hrs/8 = 236 days x $200 = $97, $219,600 2. 488x$250 = $122,000 hrs/8 = 805 days x $200 = $161, $362,250 2. 805x$250 = $201,250 hrs/8 = 1043 days x $200 = $208, $469,350 x$250 = $260,750 Airfare and car rental are not included in the estimated savings

Training - Internal Cost Savings
Airfare and car rental are not included in the estimated savings

Real-time Operations Are these numbers by Quarter?
Key Takeaway – the Vancouver office has a significantly greater workload, in support of both the Northwest and California, than the Loveland office. Are these numbers by Quarter? 2009 2010 2011 2012

Real Time Operations Key Takeaway - Additional FTEs will help identify and address reliability issues that require RC System Operator action. Risk –Identifying and addressing such issues reduces vulnerability to major system disturbances and potential non-compliance.

Existing WECC RC Functions and Current Staffing
Operations Planning Current - day/real-time Support Next-day Study Data Validation and Mapping Seasonal SOL Coordination and Validation Emergency Plan Coordination 9 5 FTE Totals Real-time Operations Monitor and analyze real-time system conditions Coordinate system restoration activities Proactively respond to and mitigate potential adverse impacts to the BES Communicate with and direct corrective actions of registered entities during actual SOL exceedances Identify and mitigate potential/actual IROLs in real- time operations horizon 33 Compliance Document and Evidence Management Audit Preparation Event Analysis Standards Development Compliance Reporting Disturbance Reporting Operations Support Application Architects Application Support DTS Support EMS Modeling RAS Engineer Architecture Support Communications IT Infrastructure Systems Administrator 20 3 WISP Broaden deployment of phasor measurement units throughout the Western Interconnection Implement telecommunication network necessary to support a large- scale production synchrophasor system Install synchrophasor applications for Western Interconnection Increase ability to integrate variable generation Enhance power system analysis capability/model validation 2 Training Develop/Deliver in-house training to new RC trainees and current RC staff to ensure compliance with mandatory standards and retention of NERC RC System Operator certification Conduct just-in- time training for new tools, applications, procedures, and processes Update job task analysis Develop/deliver NERC certified restoration training to BAs, TOPs, and GOs 6 Other Direction and Oversight Administrative Support WECC and NERC Committee Work WECCRC.org Development Membership Request Management 4 Existing WECC RC Functions and Current Staffing Current FTE – Filled Current FTE Approved – Unfilled 81 15 Totals

Improving Models - Adding Measurements
Adding measurements is necessary but translates to more alarms for the RC System Operators. Many measurements are still needed. Key Takeaway – Previous RCs (pre-2009) had far fewer measurements (40 percent less). There is a continued need to increase situational awareness throughout the system, including lower-voltage facilities. Risk – Missing measurements, even low-voltage facility measurements, leads to localized inaccurate state estimator results and a reduced awareness by the RC System Operators.

Reliability Coordination Task Force

Similar presentations

Presentation on theme: "Reliability Coordination Task Force"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Reliability Coordination Task Force

Similar presentations

Presentation on theme: "Reliability Coordination Task Force"— Presentation transcript:

Similar presentations

About project

Feedback