Presentation on theme: "Compiled by Chuck Newton Newton-Evans Research Company September 2012."— Presentation transcript:
Compiled by Chuck Newton Newton-Evans Research Company September 2012
2012 Usage Patterns and Trends in Control Center SOP, Visualization and Cyber Security Welcome to this briefing session:
A Review of Findings from Three Studies Conducted in 2012 by Newton-Evans Research (1) Standard Operating Procedures For Control Room Operations. Larger utilities and ISO/RTOs Small Sample (26 IOUs, RTO/ISOs and Large Public Utilities) Study commissioned by American Engineering University (2) NERC CIP Compliance Topical Study Small-Midsize utilities – Up to 200,000 customers More than 100 U.S. and Canadian Utilities Participated Study commissioned by Cyber Security Specialist Firm (3) Newton-Evans Study of Cyber Security for Protection and Control Larger Utilities (North America and International) More than 60 utilities from 30+ countries Participating Commissioned by CIGRE JWG B5 D2.46
Control Systems Operations Managers and Senior Staffers
% driven internally %Regulatory and reliability organizations Public Power44%56% Cooperative64%36% Investor-Owned51%49% Vendors48%53% ISO/RTO40%60% Summary48%52% 1. How much of the details of real-time operating procedures are driven internally from the organization, how much driven from reliability (Reliability Coordinator, Transmission Operator, Balancing Authority) or regulatory organizations (NERC, FERC, state)?
2. Please rank the following types of information based on their importance in making decisions for various real-time procedures in each scenario. Use a scale of 1-5, with 1=”most important” and 5=”least important” (Using each number only once.) SCADA data Contingency Analysis data State Estimation data Verbal communications Other information sources <-- most important least important --> Normal operationEmergency operation Restorative operationPost contingency
Normal operation SCADA data Contingency Analysis data State Estimation data Verbal communicationsOther information sources Public Power Cooperative Investor- Owned Europe Vendor ISO/RTO Summary Emergency operation SCADA data Contingency Analysis data State Estimation data Verbal communicationsOther information sources Public Power Cooperative Investor- Owned Europe Vendor ISO/RTO Summary Summary
Restorative operation SCADA data Contingency Analysis data State Estimation data Verbal communications Other information sources Public Power Cooperative Investor- Owned Europe Vendor ISO/RTO Summary Post contingency SCADA data Contingency Analysis data State Estimation data Verbal communications Other information sources Public Power Cooperative Investor- Owned Europe Vendor ISO/RTO Summary
4. Who in the operational hierarchy executes the EMS applications, the higher or lower reliability authority? (Check all that apply)
5. Do any of the entities checked above in question #4 run EMS applications and compare results? Yes, all involved 71% No 17% Other 12%
6a. How are actions coordinated for events near the boundaries of Balancing Authority Areas or Reliability Coordinating Areas? (Check all that apply) For the survey group as a whole, verbal communications is the dominant method for coordinating events near the boundaries of BAA’s or RCA’s. ICCP, however, is also frequently used among all of the domestic utilities and RTO’s.
6b. How is corrective action decided and carried out? By a impressive margin (88%), corrective action is decided and carried out through cooperative decisions and actions by both the Balancing Authority and Reliability Coordinating Areas. 88% 8% 4% 0%20%40%60%80%100% Cooperative decisions and actions by both the Balancing Authority and Reliability Coordinating areas Only the Balancing Authority decides and takes action Only the Reliability Coordinating area decides and takes action
6c. If just one entity decides and takes corrective action, what is the MAIN driver of this decision? (Pick one) Equipment responsibility or ownership (44%) is the main driver for the eighteen respondents to this question. However, this value increases to seventy-one percent (71%) if only the responses from the seven investor owned utilities are considered. 0% 44% 17% 22% 17% 0%20%40%60% Proximity of event to boundary (i.e. further away) Equipment responsibility or ownership Severity of event Time-criticalness of response Other
8. What type of control center wall board do you use? tile/magnet ic2D video3D videootherTotal Public Power33028 Cooperative03014 Investor-Owned43017 Europe01001 Vendor22002 ISO/RTO02103 Summary
9. What visualizations are most relevant during NORMAL SECURE (NORMAL) OPERATION of the grid? Ninety-two percent (24 out of 26) of the survey respondents rated Topological Visuals as Extremely Important during Normal Secure (Normal) Operation of the grid. Dynamically Colored Visuals were also viewed as Extremely Important to 58% of respondents.
10. What visualizations are most relevant during NORMAL INSECURE (ALERT/CONTINGENCY) OPERATION of the grid? 27% 92% 15% 54% 31% 69% 8% 58% 8% 31% 35% 15% 12% 31% 15% 0% 54% 12% 54% 19% 62% 0%20%40%60%80%100% Geographical Visuals Topological Visuals Contour Visuals Tabular Visuals Animated Visuals Dynamically Colored Dynamically Sized Extremely Important Somewhat Important Not Important to Our Operations
11. What visualizations are most relevant during EMERGENCY OPERATION of the grid?
12. What visualizations are most relevant during RESTORATION OPERATION of the grid?
13. Which type of display (large control center board or desk top screen) is an operator more likely to use in the following situations? Desk top computer screens are more likely to be used by a control room operator during all operation phases. However, during Normal and Post Contingency Operations the respondents tend to use them slightly more (69% and 73% respectively) than during Emergency and Restorative conditions where the use of a control center board significantly Increases (from 19% to 31%).
Responses from the survey group find that visualizations on the operator’s desktop computer screen offer significantly more benefits than the control center board. The control center board is cited as being better for wide area viewing. 14. What is the difference between the visualizations on the control center board and the operator’s desk top computer screen?
15. Are certain visualizations more appropriate or efficient on the control center board or operator’s desk top computer screen? Overall, responses to this question provided a little more balance when comparing visualizations on the control center board to operator desk computer screens. However, there are some significant differences in three areas: Tabular Visuals efficiency on operator’s desk computer screen received an overwhelming preference (84%) to those of control center boards, Geographical Visuals were found to be more appropriate for control center board use by a 3 to 1 margin, while Topological Visuals were better suited to operator desk top computers by a 2 to 1 margin.
16. How are critical events visualized (e.g. operating limit violations, line tripping, generator tripping, etc)? (Check all that apply) The two visualization methods predominantly used by the twenty-six respondents for critical events are Blinking Values (81%) and Highlighted and Blinking Values (73%). IOU’s indicated the highest use of Character Tag and Exception List. 81% 38% 73% 12% 42% 8% 23% 0%20%40%60%80%100% Blinking values Character Tag Highlighted and Blinking Values Zooming Exception List Panning Other
20. Please rank the relevancy of the following data in the visualization of the grid from 1-6 where 1=”most relevant” and 6=”least relevant” (Using each number only once). Device states (2.21), Voltage (2.54) and Power (2.79) were ranked the “most relevant” data in the visualization of the grid by survey respondents. For the most part, this was pretty consistent among all of the survey groups with the exception of the one European utility Voltage Current Power Frequency Device states Other <-- most relevant least relevant--> -->
Study of Small-to-Mid-Size Utilities Regarding NERC CIP Topics Study undertaken Jan-Apr 2012 More than 100 Utilities Participated Having from 20,000 to 200,000 customers
1. Does your utility have Critical Cyber Assets under NERC CIP? In spite of the seeming change in definitions of what is a critical cyber asset, two-thirds of the respondents indicated that they had NO cyber assets that are considered “critical” under current NERC CIP definitions.
2. How much did your utility spend on cyber security Operations and Maintenance in 2011? Responses here were reported across all dollar ranges. More than one-third spent less than $25,000 per year on cyber security O&M in Just over one third spent from $25,000 to $200,000. Thirteen percent spent more than $200,000. Four respondents indicated that cyber security O&M was not a budgeted item.
4. What were your utility’s capital expenditures for cyber security in 2011? One half of the respondents to this question reported spending less than $25,000 in capital expenditures for cyber security during Nearly one quarter stated that expenditures ranged from $25,000 to $200,000. Thirteen percent replied that they had invested more than $200,000 for cyber security items. Again four respondents replied that cyber security was not a separately budgeted CAPEX line item.
6. Have utility work practices and procedures changed as a result of NERC CIP requirements? Seventy percent of all respondents indicated that utility work practices and procedures have changed as a result of NERC CIP requirements. Importantly, 40% of all respondents stated that work practices and procedures have changed significantly due to NER CIP requirements. Most of the 30% reporting ‘no change’ in work practices and procedures hold the view that they do not have critical cyber assets as currently defined by NERC.
If NERC CIP requirements have caused changes, please explain : Respondent #1 We have added workflows to the process to demonstrate/document compliance. Respondent #4 Station access procedures Respondent #5 Limiting and logging access to dispatch & other source areas; lots of documentation & audit preparation; lots of effort to ensure compliance but not necessarily improve security. Respondent #6 Installation of new SCADA system required improvements in physical access requirements Respondent #7 Maintenance of CIP rules is a massive and continuous undertaking. It took 14 FTE's to get through the most recent audit Respondent #8 Device installation, testing, access management, patch management Respondent #9 NERC CIP does not apply Respondent #10 Level of reporting and documentation requirements required have increased significantly. Respondent #12 All actions and occurrences have to be verified under CIP regulations Respondent #13 While our utility does not currently fall under version 4 of the CIP standards we are actively preparing for full compliance because we anticipate version 5 will affect us significantly. At the very least a proactive cyber security program is a good practice and enhances the reliability of both the BES and non-BES power systems.
Respondent #19 Add a tremendous burden for security and logging of activities Respondent #20 Process to develop a security program has begin. We are taking small incremental steps. Respondent #24 Sign in sheets required into certain areas during certain time frames; escorts required into certain areas Respondent #25 Since NERC CIP requirements are not yet applicable, our efforts are in anticipation of future changes. However, such future changes shown in pending drafts of the NERC standards will have a VERY significant impact to work practices and procedures. Respondent #26 At this time, our utility will not consider substation LANs for IEDs & RTUs due to pending NERC requirements. Respondent #27 We are distribution but have made changes in anticipation of NERC/CIP Respondent #28 We evaluate each new technology initiative for its ability to put us into CIP requirements. Using communicating faulted circuit indicators as an example, we chose a hosted service rather than bringing the data in-house to avoid any potential CIP changes.
7. Does your utility offer in-house training for cyber security? Nearly one half (47%) of the survey respondents reported that their utility offers some form of in-house training for cyber security. Another 17% plan to offer such in-house training by More than one third (37%) of the survey sample do not offer cyber security training on an in-house basis.
9. Do you currently outsource any cyber security tasks to a third party? More than one half (53%) of the survey participants indicated that they DO outsource at least some cyber security tasks to outside services to third parties. Another 10% plan to do so by year-end 2014.
12. What are the certification requirements that your employees must have to work with projects involving cyber security Perhaps surprisingly, more than three quarters of the responding utility officials reported that there are currently NO certification requirements for employees in order to work with projects involving cyber security topics. Of the handful of utilities that indicated one or more certifications as requirements, CISSP (15%), CISM (11%), CISA (7%) and Comp TIA (4%) were specifically cited. A few reported other requirements such as CISCO Systems in the listing below the chart.
16. Have NERC CIP requirements caused your utility to increase the number of full time employees dedicated to cyber security activity? Yes, 23% No, 77% Have NERC CIP requirements caused your utility to increase the number of full time employees dedicated to cyber security activity?
19. Which of the following cyber security technologies/methods do you currently use?
Larger Utilities (North America and International) Participation from More than 60 utilities in 30+ countries Conducted with P&C Managers; Operations Managers; Some IT Management Involvement
1a. Are you offering your P&C System personnel (engineers and field technicians) any cybersecurity training for their job? ?responsibilities?
3. Have your P&C system personnel signed acceptable use policies?
1a. Are you offering your P&C System personnel (engineers and field technicians) any cybersecurity training for their job? responsibilities?
1b. Do you tailor cybersecurity training to address the issues related to job responsibility?
2a. How would you consider the quality and completeness of cybersecurity training in your organization?
2b. In your opinion, could your cybersecurity training be improved?
3. Have your P&C system personnel signed acceptable use policies?
4. Do you have a cybersecurity incident response plan for your P&C system?
5. Do you monitor P&C system personnel access to and use of P&C system components?
6. Do you test P&C system patches to correct cybersecurity defects prior to deployment?
7a. Do you have adequate controls in place to monitor P&C system behavior in order to indicate that a security incident has taken place?
7b. Do you benchmark or maintain a scorecard of P&C system cybersecurity incidents?
8. What is your short list of cybersecurity solutions needed to protect your P&C systems? Need to understand vulnerabilitiesNeed to plan to fix them Segmentation of networksUpdated password maintenance Actually, we've implemented cybersecurity measures by following NERC-CIP guideline since However, only control system has been secured but not for protection system. Do not connect relays to the network. Currently as little as possible connection to the outside Web. All personnel have dedicated pc's to connect to P&C systems. Anti-virus systemPhysical separated networkRestricted firewall Password Policy - # of characters and 90 day expiration was implemented Access control (authorization, etc.), closed network configuration (net separation)Audit Log, Backup Antivirus & Firewall software etc. Security of the network access to P&C Security of the computer devices used as tools Security practices of the personnel Remote secure access (through SCADA or IP solution)Secure mobile local access to devices Restricted physical accessRestricted electronic accessTraining and awareness Enfording NERC-CIP standards usage Implementing firewalls in the substation Implementing anti-malware software in DCS P&C devices have no connectivity to any system. None at this time outside the substation. Remote accessProtocolIPS Security gatewaysFirewallsHardwired telephone switch Perimeter access control, both physical and electronic Intrusion detection and prevention software Centralized software patches and password management Security enforcement points Centralized configuration management system Extend our existing remote access system
11a. Do you allow employees to use their personal devices (i.e. personal flash drive, smart phone, tablet, etc.) for P&C maintenance or configuring P&C components?
11b. If “NEITHER” to the above, are you planning to allow employees to use their personal devices (i.e. personal flash drive, smart phone, tablet, etc.) for P&C maintenance or configuring P&C components? “BYOD” could affect views
11c. Do you support programs loaded on employee personal devices?
11d. What is your estimate of the percent of employees using personal devices for P&C maintenance?
11e. What is your estimate of the percent of employees using personal devices for configuring P&C components?
11f. Do you enforce security policies and encryption for employee personal devices?
12a. Do you allow third party support technicians to use their personal devices (i.e. personal flash drive, smart phone, tablet, etc.) for P&C maintenance or configuring P&C components?
12b. If “NEITHER” to the above, are you planning to allow third party support technicians to use their personal devices (i.e. personal flash drive, smart phone, tablet, etc.) for P&C maintenance or configuring P&C components?
12c. Do you support programs loaded on third party support technicians' personal devices?
12d. What is your estimate of the percent of third party support technicians using personal devices for P&C maintenance?
12e. What is your estimate of the percent of third party support technicians using personal devices for configuring P&C components?
12f. Do you enforce security policies and encryption for third party support technicians' personal device?
13a. Are your P&C cybersecurity policies and procedures derived from regulatory requirements?
13c. If no, from where are your cybersecurity policies derived? 49% 40% 52% 54% 60% 52% 22% 0% 29% 22% 40% 16% 76% 80% 74% 7% 0% 10% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% SummaryNorth AmInternational Utility guidelines Generally accepted industry guidelines Professional association recommendations P&C staff recommendations IT department guidelines Other
14. From the list below, rank the inhibitors for implementing strong security policies for P&C system operations in order from 1-4, with 1=”strongest inhibitor” and 4=”weakest inhibitor.” SummaryNorth AmInternational Cost to maintain and operate a strong security system Perimeter security provided by and supported by IT is adequate Lack of interoperability between P&C system components P&C system components do not incorporate strong security mechanisms
16a. Does your utility have the technology or business processes needed to manage role-based access control (RBAC) for P&C systems?
Upcoming EMS/SCADA/DMS Study 4 th Quarter 2012 Study of Control Systems usage patterns and plans among the world’s electric power delivery utilities. We need your help for this study to serve as the bridge between what YOU need and want in control systems and what systems providers-vendors need to know in order to develop solutions to meet your needs.
Other 2012 Research Topics U.S. Manufacturing Readiness for Smart Grid Cloud Computing Outlook for Small-Midsize Utilities and Usage of Specific IT/OT Applications Packages Substation Processing Platform Options Fault Current Limiting Devices U.S. Market for Bus Duct Assessment of American Manufacturing Industry Readiness for Smart Grid Roll Out
Prepared by Chuck Newton Newton-Evans Research Company September 2012 Thanks for sitting in on this briefing!
Appendix Slides – 2012 Findings on Cyber Security and Usage and Plans
Looking at Smart Grid Opportunities for Growth at Mid- Year 2012 ….What stands in the Way? by Chuck on June 25, 2012 Why we believe the near-term investment priority for utilities of all types must be cyber security-related! (Security is not always considered part of smart grid spending)! This year, Newton-Evans Research has already undertaken a number of national and international studies of cybersecurity issues, and the findings lead us to believe that the single most critical issue facing utilities of all types is the near-term requirement to shore up cyber defenses, policies and procedures. Unfortunately, these cyber security investments will likely continue to usurp funding from other smart grid activities, but this investment must be a priority, in my opinion.
Findings from Jan-Mar 2012 Survey of Protection Engineers Relay Protocol Use North AmericaInternational
Extent of Use of IEC in Substations North AmericaInternational
Findings from Jan-Mar 2012 Survey of Protection Engineers Features of IEC Being Used/Planned North AmericaInternational