1. Teleprotection with MPLS Ethernet Communications - Development and Testing of Practical Installations
Tariq Rahman and James Moralez, San Diego Gas & Electric Company
Solveig Ward and Eric A. Udren, Quanta Technology, LLC
Michael Bryson and Kamal Garg, Schweitzer Engineering Laboratories, Inc.
Presented at
72nd Georgia Tech Protective Relaying Conference
Atlanta, GA
May 2-4, 2018

2. Background

3. San Diego Gas & Electric System
Provides natural gas and electricity to San Diego County and southern Orange County in southwestern California to 3.6 million consumers
1.4 million electric meters and 873,000 natural gas meters in a service area that spans 4,100 square miles
Currently, SDG&E uses TDM network for teleprotection and SCADA
The TDM network consists of a mix of direct fiber, T1 multiplexers on TDM SONET, microwave radio, leased-line, and channel bank equipment
San Francisco
Los Angeles
SDG&E
San Diego
SoCal Gas

4. Introduction
Technology evolution is driving towards Ethernet communications - converged utility communications network
Typically, packet based IP routing in an Ethernet WAN had been fundamentally less predictable than the deterministic point to point TDM or serial data communications circuits
Teleprotection is migrating from SONET to MPLS Ethernet
To validate the design and in preparation for substation field installations within the SDG&E system, laboratory testing was performed using a Real Time Digital Simulator or RTDS® system model
Test MPLS routers & network configurations were applied to protective relays at the SDG&E Integrated Test Facility (ITF)

5. Utility Communications Services
Utility IT departments were historically responsible for enterprise functions represented by the gray “System Administration and Support“ box on the upper right - functions supported with generations Ethernet networking for decades.  The cost of providing communications for these functions must be low, and time frames of response interact with human processes of hours to months or longer.  These enterprise functions make minimal demands on communications performance or speed; and most are tolerant of delays and outages of information flow.
Notice that the time axis has logarithmic calibrations. 
The blue “System Priority’ functions are important to maintaining reliable service to customers, or restoring it when lost; as well as operating transmission or distribution systems in cost-efficient ways.  These functions have been migrating to Ethernet IT networks over the last decade.  Time frames of response are still human, ranging from minutes to hours.  Network speeds still have little role; but network reliability and availability in the short term becomes more important, and we have more economic justification for moving to private networks to insure good availability.
The red box on the lower right shows “System Critical” or mission-critical functions which actually monitor, protect and control transmission or distribution grids in real time.  The performance of private dedicated circuits is cost-justified, and required to avoid failures or blackout risks.  These communications-based functions and their data networks are today mostly based on reliable serial and TDM data communications, including SONET ring networks.
The most demanding function on the far left is teleprotection – transmission line relays require highly available and often redundant data communications with only milliseconds of latency allowed, and other restrictions, to meet performance requirements.  Ethernet IT networks have not been able to meet these requirements in the past, but the latest generation of MPLS Ethernet networking solutions looked promising.  There is strong business and economic pressure from the IT business and world to move the System Critical functions to this platform, even for teleprotection.  Our mission as described in this paper is to determine if and how MPLS Ethernet networking can support reliable and secure protective relaying for our most critical transmission lines and demanding relaying methods.
Note that the figure is 3D. In addition to the time axis (x) and criticality axis (y), the size of the rectangles also illustrate the volume of data for the different applications. Admin data makes up the majority of data services IT needs to provide while teleprotection and other system critical data has a small volume. This is one of the reasons it’s been difficult to motivate IT/telecom to make the required effort to satisfy the requirements of the “smallest” service.

6. Evolution of Telecommunications

7. SONET Characteristics
Point-to-point connection
Deterministic and low latency (1 – 3 ms)
Equal transmit and receive delay (no asymmetry)
Ring redundancy
Substation multiplexer fail-over as low as ms

8. Ethernet IP Characteristics
Ethernet is based on IEEE standard with various versions supporting higher data rates and lower latency
Widely adopted packet-based technology
Non-deterministic latency
Basis for IEC P&C

9. MPLS Characteristics
MultiProtocol Label Switching – packet label field routes Ethernet packets among MPLS routers
Dynamic and static routing available
Predictable latency
Pseudowire services to support TDM/Serial communications
Low latency enabled by using a static pre-defined path, and the use of small jitter (data) buffers for teleprotection traffic
High priority provisioning through the use of Quality of Service (QoS) configuration
MPLS ensures minimal asymmetry by routing transmit and receive packets over static paths via the same network nodes
Path fail-over times ms
Mitigated by using redundant teleprotection channels in the relay with 0 – 2 ms fail-over time

10. SDG&E MPLS Project Drivers
MPLS is the current communications transport standard being widely adopted in other Industrial Control Systems (ICS) environments such as water, public safety networks, land mobile radio backhaul, etc.
As MPLS is adopted into substation communications -replacing instead of upgrading older technology - it is expected to deliver significant benefits to overall utility communications, with higher service availability
Provides a reduction in maintenance costs (O&M) as utility operates a single communications system
Provides comprehensive network monitoring and network diagnostics

11. Project Development

12. SDG&E Methodology
Development of business requirements based on internal and external drivers
Development of in-depth technical requirements, and requirements traceability matrix
Creation of an MPLS network lab testing environment
RTDS lab testing of teleprotection over MPLS
Eight 500 kV, 230 kV, 138 kV and 69 kV circuits; 2 and 3 terminal line protection applications
Implementation and testing of channel monitoring functions

13. SDG&E Methodology (cont.)
While the RTDS testing is utilizing MPLS communications channels, it is not possible to emulate field production MPLS network conditions within a lab environment.
To ensure that any issues that may arise during the field test period are related to the communications network, and are not due to a protection scheme deficiency, the schemes and relay settings are thoroughly tested in the RTDS lab on a model that accurately simulates the circuit each test relay will be installed on.
In doing this, root cause analysis of any protection mis-operations during field testing will not need to involve extensive protection scheme evaluation and testing, since schemes and settings have already been fully verified.

14. SDG&E Methodology (cont.)
Installation of transmission line field test relays and monitoring for a period of 12 months
MPLS network data and relay channel monitoring data will be continuously collected and analyzed, with the use of analytic tools so that relay reports and MPLS network reports can be correlated for any communications errors. This has been a shortcoming with SONET communications where the relays will record channel interruptions that are not observed by network monitoring tools.
Assuming a successful field trial testing period, the migration of teleprotection will commence as MPLS network service is migrated to substations

15. RTDS Model
Exhaustive application for 8 transmission lines. SDG&E has an RTDS system model for their 500 kV and 230 kV system. Models for 138 kV line testing and 69 kV line testing were developed. Lines tested included 500 kV, 230 kV, 138, and 69 kV; with 2 and 3 terminals.

16. Test Requirements and Test Setup
2 relay manufacturers, 3 relay types tested. 87L is particularly sensitive to latency and asymmetry performance. Also tested POTT in same relays.
The failover scheme we developed is completely different from that for SONET. 50 ms path failover of network is too slow. So each relay connects permanently to two network paths; the relay fails over if its primary path fails, in 3 ms or less. This is possible because of the data capacity of the MPLS connections compared to typical SONET channels.
With this dual connection, bumpless failover is possible in relay design.
Once the Ethernet network path outage is detected by the routers, they perform an Ethernet network failover reroute, and the failed path recovers, restoring full dual redundancy.
Note that two redundant network paths serve three relay pairs in this connection, with no single point of failure.
Note that cross-connections are flipped for Relay 1 versus Relay 2 so that both network paths are being used for redundant protection in normal situations.
The network paths should have fixed or static configuration to assure low latency and asymmetry. For Ethernet network failover, the backup path or paths should also be statically configured (preconfigured in the routers, not based on a network mesh path search by the router in real time).
Latency < 5 ms
Asymmetry < 2 ms
Failover < 3 ms
Availability > 99.95%

17. Asymmetry < 2 ms
87L with channel based synchronization uses the loop delay divided by 2 for alignment
Channel
Delay
Local current
90 deg.
error
Correct compensation
Incorrect compensation
Differential current
Local current memorized for comparison
Current received from remote end

18. Asymmetry <2 ms - Test Setup
2 ms asymmetry introduced 87
L Channel Asymmetry
Test Setup
Relay 1
L CH X 2
L CH 3
MPLS Router
Asymmetry Delay
(Linux Desktop)
87L Forward Path
L Return Path
We need to insure protection security - this is a test of the asymmetry impact on the relay’s protection security and dependability, not a network test. The network in the lab has latency and asymmetry far below 1 ms. The field network should perform nearly as well if primary and backup Ethernet paths are statically configured in the routers. The IT NOC has this 2 ms specification and should monitor it in real time. Relays should also monitor this, to assure protection engineers.

19. Fail-over <3 ms – Test Setup
The connection break device is driven by the RTDS that applies faults. So the test programming picks inopportune, worst-case timing to fail the network. – to insure robustness of protection.

20. Latency <5 ms
Relays measure latency from (a)-(a) or (b)-(b) depending on the relay type
5 ms specification is for (b)-(b)
Achieving this latency target requires adjustment of MPLS router jitter buffer and other settings. Both the routers and the relays must be programmed to monitor this timing in service, with real time connectivity of monitoring data. Don’t just trust the IT department, which may not deeply understand the role of relays, to handle the channel monitoring.

21. Latency <5ms - Test Results
Thousands of faults are simulated on the RTDS transmission network model. In these trip time results, the left plot is the baseline of fastest possible operation with back-to-back direct optical fiber connection between the relays. The right plot shows acceptable timing with MPLS Ethernet routers as the communications path. While not shown, timing with a SONET multiplexers is slower than the baseline too, and is only at best fractionally faster than MPLS in these tests.
Note: Operating time peak of Relay 2 at 50% location was due to that relay's protection logic sensitivity to detecting certain fault types, hence variability in operate times.  The trip time peak did not have to do with communication path variability or jitter.
One way latency ≈ jitter buffer setting
<5 ms requirement met with jitter buffer = 3 or 5
Larger buffer -> better reliability
Smaller buffer -> higher bandwidth (3 -> 896 kbps)

22. Communication Requirement
Summary of Test Results
Communication Requirement
Specification
Results
Latency
< 5 ms
Pass1
Asymmetry
< 2 ms
Pass2
Failover
< 3 ms
Pass3
Availability
> 99.95%
N/A
1 Latency < 5 ms achieved with specific Jitter Buffer and Payload MPLS router settings.
2Asymmetry < 2 ms achievable with specific network design. Laboratory tests show protection operates correctly at 2 ms asymmetry specification limit.
3Failover < 3 ms achievable with 2 of 3 relays meeting specification. Protection system with designed failover paths and protective relay failover meets failover specification. MPLS routers do not meet failover specification by failing over to backup Ethernet router path.
The not-available yet results for conformance to availability specification depends on field service experience from IT NOC and from relay monitoring data from programmed and connected relay monitoring functions to be deployed. We recommend both, to cross-check performance and to insure unavailability of paths is diagnosed and corrected down to root cause. In working with IT, trust but verify.

23. Conclusions
Laboratory testing has shown that MPLS networks are a viable communications medium for protective relay telecommunication traffic if designed to account for latency, asymmetry, failover and availability.
RTDS tests validated settings for routers and switches of the field MPLS network, as well as for the relays.
RTDS testing has allowed SDG&E to specify and set the channel/communications monitoring parameters in the relays to support MPLS Ethernet performance monitoring – had not been implemented or needed with TDM.

24. Acknowledgements
A special “Thank You” to those who have contributed to the creation and completion of this paper and presentation:
Mike Mahoney – Burns & McDonnell
Clint Struth and Cory Struth – SCI Networks
Terry Wright – GDC Consulting