1. Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE

2. Introduction - Experience Dragan Dokic | President, Summit Energy Tech Focus on utility sector – Infrastructure systems management – Custom business systems software development 16 years of experience in IT industry 10 years in utility sector – Managed network operations for PNGC Power [Portland, OR] from September 2002 to October 2011 – Presentation focuses on lessons learned in field network reliability, security and manageability from this experience

3. Introduction PNGC’s 2001 – 2011 field network – 92 office, substation and repeater sites at 11 distribution utilities in Oregon, Idaho System mission – Gather real-time load data 24/7 for power scheduling operation in Portland – Support local utility SCADA/AMI/Site Security operations

4. PNGC Power WAN – July 2011

5. Toledo, OR

6. Boardman, Oregon

7. Junction City, Oregon

8. Lewiston, ID

9. Malta, ID

10. The Moon

11. Areas of Focus

12. Reliability – Network Design Keys to success – Diversity in media Combine land lines, fixed wireless [private/public], mobile wireless and satellite – Diversity in providers Local and national – Dynamic Routing [OSPF] Routers exchange knowledge of local network with neighboring routers Enterprise grade routers / switches a requirement Perfect world configuration – Private wired/wireless ‘island’ with two Internet gateways using distinct media and distinct providers

13. Connectivity overview Primary router Backup router

14. Link cost overview Primary Backup

15. Link cost calculation Sub A -> Main Office via Satellite tunnel: 3 + 1 = 4

16. Link cost calculation Sub A -> Main Office via 900Mhz+DSL tunnel: 1 + 1 + 1 = 3

17. Open Shortest Path Link cost via Satellite tunnel [4] higher than via DSL tunnel[3]; therefore, packets will traverse 900Mhz/DSL tunnel in normal operation

18. Normal Operation Open Shortest Path From substation A to Main Office

19. Normal Operation Open Shortest Path From substation B to Main Office

20. Link down operation If DSL tunnel is down, packets will traverse satellite tunnel; Sub A  Main Office X

21. Link down operation If DSL tunnel is down, packets will traverse satellite tunnel; Sub B  Main Office X

22. Questions?

23. Security – Overview Wireless link encryption Function specific VLANs No default routes!

24. Wireless Link Encryption Media device level [e.g. Radio, Modem] – WEP, WPA, WPA2 Routing device level [e.g. Cisco 891 router] – IPSEC End device level [e.g. DIGI TS4 port server] – SSL

25. At what level to secure data?

26. Security - Wireless Link Encryption [continued] Most secure option? – Use all three if management overhead is not an issue Most efficient but secure enough option? – Use routing device site-to-site VPN capabilities – Advantages: Support for best commercially available security technologies [e.g., AES-256] Comprehensive change logging capabilities Standardized configuration throughout the system [less management overhead]

27. Security – Function Specific VLANs Define VLAN’s per business function – SCADA, AMI, Security System, Wireless, VOIP, Network Mgmt. Firewall traffic between VLANs on need-to-access basis – E.g., Prevent personnel attached to substation wireless VLAN to access documentation stored on a server at the main office from accessing recloser controls in the SCADA VLAN Reliability advantages – Non-critical VLANs [e.g. AMI, security] can be shut down automatically/remotely if link quality is too poor to carry all traffic, but good enough to carry SCADA

28. One VLAN per business function

29. High-speed link outage scenario

30. Security – No Default Route! Do not use default routes through service provider- supplied gateways Define a single host route back to the main office, then establish default route through VPN tunnel This is the most effective method to prevent attacks sourced from the Internet Always use in conjunction to regular firewall configuration lists [not a substitute!]

31. Less secure Provider gateway

32. More secure Provider gateway

33. Questions?

34. Manageability - Overview Tools – network management systems Addressing – developing a scheme Watchdog system – preventing lockout

35. Manageability – Tools Network Management Systems [NMS] Protocols used SNMP, Syslog, ICMP, HTTP Applications PRTG Solarwinds Syslog

36. Manageability – Tools [continued] How to collect data? Push vs. Pull – Pull: Poll devices using SNMP/HTTP/ICMP at regular intervals [e.g., every – Push: Devices send data per defined event triggers – SNMP traps – Syslog messages What data to collect? – Availability [ping] – Network utilization – Input voltages – RSSI [radio link quality]

37. Manageability – Tools [continued] Pull example: – 5 minute SNMP poll of UPS for input voltage – If voltage drops below threshold of 108VAC for a duration of time longer than 5 minutes, an alert will be triggered by NMS [e-mail, text message, event log] – But what if voltage drops for 2 minutes only in between polls? You may not know it even happened. Push comes to rescue: – UPS sends SNMP trap to NMS as soon as voltage drops below 108VAC – Alert is triggered by NMS when trap is received

38. Paessler PRTG – Screen shot

39. Solarwinds Kiwi Syslog – Screen shot

40. Manageability – Addressing Develop consistent scheme to use system wide Recommended private range: 10.0.0.0/8 – First octet: same for entire system – Second octet: site ID [e.g. 8=Springfield Sub] – Third octet: business function ID [e.g., 4=AMI] – Fourth octet: device itself [e.g., Collector #1] 1 st octet ‘fixed’ 2 nd octet = site ID 3 rd octet = vlan/business function 4 th octet = device Subnet Mask [255.255.255.0]

41. Manageability – Addressing [continued] Large network? – Group sites by region using second octet – Allows for address summarization if needed. Example: – Eastern division region: 10.64-127.0.0 Summary address: 10.64.0.0/10 – Western division region: 10.128-191.0.0 Summary address: 10.128.0.0/10

42. Manageability – Watchdog System General concept – Reboot key remote communications devices if connectivity to central site is interrupted Benefit – Prevent unnecessary site visits due to Operator error Device lock-up [e.g., buggy firmware, heat issues]

43. Manageability – Watchdog System [continued] Hardware requirements: – SNMP-capable switched PDU with task scheduling and delayed power cycling command capabilities – Example: APC AP7900 8-port 15A PDU Software capability requirements: – Centralized command override mechanism using NMS – Send SNMP ‘Set’ to cancel pending power cycling command

44. Manageability – Watchdog System Example ‘Delayed’ power cycle schedule is defined on PDU: – Outlets to power cycle:1,2 [e.g., radio, router] – Frequency: 60 minutes – Command execute delay:30 minutes Network management system running at main office sends an SNMP delayed power-cycle command cancel message – Frequency:every 5 minutes Process – If delayed power cycle cancel command cannot reach the PDU at least one time during the 30 minute reboot delay period, outlets 1 and 2 will be power cycled and communication will (hopefully!) be restored

45. Questions?

46. Thank you!