Monitoring EMS Infrastructure Ann Moore San Diego Gas & Electric September 13, 2004 EMS Users Group Meeting-St. Louis

2 Agenda SDGE – Who we are and What we do NERC Near-Term Actions NERC Urgent Action Standard 1200 EMS Infrastructure Critical Cyber Assets IT Monitor Sample Displays What’s Next

3 Sempra Energy Sempra Energy is a Fortune 500 energy services holding company with over 12,000 employees  Sempra Energy Utilities San Diego Gas & Electric (SDG&E) Southern California Gas Company (SoCalGas)  Sempra Energy Global Enterprises Sempra Energy International Sempra Energy LNG Corp. Sempra Energy Solutions Sempra Energy Resources Sempra Energy Trading Sempra Fiber Links

4 SDGE & Electric T&D 1.3 million customers 3 million population Service territory includes San Diego County and Southern Orange County 4,150 MW area peak load (9/10/04) 130 Transmission RTUs (69kV, 138kV, 230kV, and 500kV) – GE XA21 EMS 900 Distribution RTUs (12kV) - ACS Prism DMS

5 NERC Near-Term Actions To Assure Reliable Operations Failures of System Monitoring and Control Functions: Review and as necessary, establish a formal means to immediately notify control room personnel when SCADA or EMS functions, that are critical to reliability, have failed and when they are restored.  Establish an automated method to alert power system operators and technical support personnel when power system status indications are not current, or that alarms are not being received or annunciated.

6 In-House Implementation Generating a pseudo Alarm/Event every 5 minutes 08/16/04 13:00:01 PDT DCTSta A/E CHECK SUCCESS... AE 5 Min. Success 08/16/04 13:05:01 PDT DCTSta A/E CHECK SUCCESS... AE 5 Min. Success 08/16/04 13:10:01 PDT DCTSta A/E CHECK SUCCESS... AE 5 Min. Success 08/16/04 13:15:01 PDT DCTSta A/E CHECK SUCCESS... AE 5 Min. Success “Check” process to check A/E logs Sending automatic notifications How about…  Other critical processes?  Other system characteristics?

7 NERC Cyber Security Standard NERC Urgent Action Standard 1200 presents standards to “monitor” and protect critical cyber assets At Sempra, we take compliance seriously –SDGE Self-Certification – 3/1/04 –“Substantial Compliance” – 3/1/04 –“Full Compliance” – 3/1/05

8 EMS Infrastructure EMS DMZ EMS DMZ Office Network Primary Control Center Backup Control Center WAN CORPCORP CAISO DMZ CAISO DMZ EMS SYSTEM EMS SYSTEM WAN FIREWALL EMS SYSTEM EMS SYSTEM CAISO DMZ CAISO DMZ FIREWALL EMS DMZ EMS DMZ Office Network CAISOCAISO

9 Critical Cyber Assets EMS nodes : 40+ UNIX boxes –Application Hosts: IBM AIX –Oracles: IBM AIX –Front End Processors: IBM AIX –Dispatcher Workstations: SUN Solaris Windows Servers : 10+ servers –PI Servers –PI OPC/Interfaces –SQL Servers –Web Servers Network Devices : switches, routers, and firewalls

10 Monitoring All A proactive and preventive way to monitor EMS infrastructure health to ensure the system performance and reliability Monitoring all EMS infrastructure for disk, file, paging, cpu, swap and memory, …etc. Monitoring EMS processes and applications Establishing performance baseline standards Avoid finger pointing Root cause analysis and problem solving Automatic notification via and cell phone

11 IT Monitor OSIsoft PI OSIsoft PI EMS DMS Non-Scada DNP Metering Cyber Assets Ping NetFlow TCP Response PerfMon SNMP

12 Main Display Index

13 EMS Node Status Overview

14 Dispatcher Workstation CPU

15 Application Host Index

16 Application Host 1

17 Monitoring Oracle-Oracle Database MIB’s Monitoring ipcs Implementing SNMP Traps SNMP agent in facility equipments Secured SNMP traffic (encrypting SNMP?) Utilizing Syslog interface Integrating with Cisco Works, IDS and HP Openview, …etc. THANKS!! What’s next…