Selecting a Network Topology for Reliable Machine Control Presenter: Dan Strachan–Kendall Electric

Agenda – Building a Machine Network JF Network Requirements Topologies Switches Plant Convergence at the Machine Level Tools and Where to learn more Reference Architectures Solutions 2 2

EtherNet/IP - Review EtherNet/IP provides a single network technology for a variety of network and application requirements including Motion, Safety, Discrete, Drives and Process applications. EtherNet/IP is the name of the Ethernet network that uses the Common Industrial Protocol. The “IP” in EtherNet/IP means Industrial Protocol as in Ethernet/Industrial Protocol

Ethernet/IP – OSI Model Open Systems Interconnection Layer Name Layer No. Function Examples Application Layer 7 Network Services to User App CIP Presentation Layer 6 Encryption/Other processing Session Layer 5 Manage Multiple Applications Transport Layer 4 Reliable End-to-End Delivery Error Correction IETF TCP/UDP Routers Network Layer 3 Packet Delivery, Routing IETF IP Switches Data Link Layer 2 Framing of Data, Error Checking IEEE 802.3/802.1 Physical Signal type to transmit bits, pin-outs, cable type Cabling Layer 1 TIA - 1005

EtherNet/IP and ODVA Although originally developed by Rockwell Automation, EtherNet/IP is an open network. Many other companies, including competitors, sell EtherNet/IP compatible products. This is one of the reasons it is so attractive to our customers. ODVA, the Open DeviceNet Vendors Association, manages the development of the open network technologies based on the Common Industrial Protocol (CIP™), and assists manufacturers and users of CIP Networks through tools, training and marketing activities.

Reference Architectures Solutions Agenda JF Network Requirements What are the machine network requirements What requirements will be added at machine commissioning time Managed vs. unmanaged switches Things to consider: Security requirements, non-Control traffic requirements, Protection from interference, resiliency requirements Topology and Physical layer requirements Embedded Switches, cabling and grounding selection Network Performance requirements Reference Architectures Solutions 6 6

Machine level Network Considerations JF Control Requirements and Physical Layout I/O, Safety and motion control how much how fast, distances, noise Integration to upstream or downstream equipment Line Controller Safety interlocking Integration of data (Plant Connection) SQL or other servers for data collection and monitoring Supply chain integration Remote Access Troubleshooting, monitoring, program changes Lets start by thinking about the functions that a machine or skid level network must play. The most critical role of this network will be control of the local machine, it must transmit information from I/O modules and drives to and from the controller at deterministic rates. This network is often used for communications with upstream and downstream equipment for example on a bottling line the filling machine may need some interaction with the bottle molding and hopper upstream and maybe a labeling machine and case packer downstream. This could be done through a line controller or direct communication There is more and more demand for data from machines or skids and the network will be used to harvest this data for upper level systems. Remote access may also be desired, for startup and ongoing support needs. Protection against unauthorized remote access must also be considered. 7

Agenda Topologies Reference Architectures Solutions JF Topologies Machine Network Segmentation methods Advantages and disadvantages NAT Layer 3 Dual NIC solution Remote Access Methods Inside/Out approach Outside/In approach Understanding remote access requirements Other Considerations for Information Integration Network Addressing Requirements Application requirements Reference Architectures Solutions Copyright © 2009 Rockwell Automation, Inc. All rights reserved. 8 8

Rockwell \ Cisco Model

Topology Flexibility with EtherNet/IP RING – Maximum availability LINEAR - Simplify cable management HYBRID – Obtain maximum flexibility Ethernet is topology neutral, choose the best option for the application needs Linear- Advantage: minimal wiring, distance and simplicity, Disadvantage: Tolerance for outages is low, power down a device anywhere on the chain and connectivity is lost to all devices past the break, less efficient use of bandwidth. Star- Advantage: Most efficient with system bandwidth, minimal impact if a device is removed or lost, Disadvantage: more wiring, cost of external switches Ring- Advantage: resilient connectivity with minimal wiring, Disadvantage: some configuration required, only tolerance for a single failure Hybrid- Mix and match for application needs These examples are all smaller systems. However, larger systems with numerous external switches have considerations for topology as well. These considerations may differ from those at the device level. For example with switch topologies resiliency tends to be more critical because loss of connectivity will mean loss of communications to many more end devices. Bandwidth needs are also typically greater at this level because communication has been aggregated for several end devices. STAR– Connect broad range of devices EtherNet/IP is topology neutral for maximum flexibility

Embedded Switch Technology The Embedded Switch Technology embeds Ethernet switch features into your hardware to support high performance applications. Our dual-port products with the embedded switch feature let you connect devices directly to your network, without a separate Ethernet switch, helping to reduce overall cost and simplifying system configuration. These products support linear and device-level ring (DLR) topologies for EtherNet/IP™ applications.

Embedded Switch Technology Device-level Topologies 802.3 operation Autonegotiation, with 10/100Mbps, Full/Half duplex Forced setting of speed/duplex 802.3 full duplex flow control Cut-through operation, with store & forward on contention QoS Multiple queues Prioritization via DSCP and 802.1Q/p Broadcast storm protection for host CPU Filtering of incoming unicast and multicast to host CPU Statistics and counters for the external ports IEEE 1588, precision time protocol (PTP) End to End (E2E) Transparent Clock (TC)

Embedded Switch Products Point I/O Adapter (1734-AENTR) ArmorPoint I/O Adapter (1738-AENTR) ControlLogix EtherNet/IP module* (1756-EN2TR/ EN3TR) EtherNet/IP Tap* (1783-ETAP) ETAP enables single-port Ethernet devices to join linear or ring topology EtherNet/IP Tap* (fiber offerings) ArmorBlock I/O options (1732E) Kinetix 6500 Kinetix 5500 1715 I/O Products PowerFlex Drives (75x Comm card for the family) Flex I/O Adapter CompactLogix L3x, L2x and L1x controllers* ControlLogix XT module* *Ring supervisor products – need at least one to establish device-level ring

Device Level Ring (DLR) 50 Nodes Max and Up to 100 Meters between Devices

Device Level Ring (DLR) Device-level Ring Topology ODVA - open standard enabling suppliers to develop compatible products Support for ring and linear topologies, fiber and copper implementations Network traffic is managed to ensure timely delivery of critical data (Quality of Service, IEEE-1588 Precision Time Protocol, Multicast Management) Ring is a single fault tolerant network Designed for 1-3 ms convergence for simple EtherNet/IP device networks

Device Level Ring (DLR) Primary Advantages Simple Installation Resilience to a single point of failure on the network Fast recovery time when a single fault occurs on the network Disadvantage The primary disadvantage of a DLR topology is the additional effort required to set up and use the network as compared to a linear or star network

Device Level Ring (DLR) Protocol Device-level Ring Topology CompactLogix Controller Forwarding Active Ring Beacon Supervisor Beacon Stratix 5700 IE Switch Blocking Announce Announce ETAP ArmorPoint I/O PowerFlex POINT I/O POINT I/O Some IACS applications, such as safety and motion require network convergence times faster than what switch oriented resiliency protocols can provide IACS Device resiliency protocols, such as the Device Level Ring, DLR, provide network convergence in the 1-3 ms range for simple automation device networks Device Level Ring is a Layer 2 protocol that provides link-level, physical redundancy This is 2-port embedded switch technology managed by the ODVA, which Cisco and Rockwell Automation are principle members of Supervisor blocks traffic on one port Sends Beacon frames on both ports to detect break in the ring Sends Announce frames on unblocked port

Device Level Ring (DLR) Protocol Device-level Ring Topology CompactLogix Controller Active Ring Supervisor Stratix 5700 IE Switch Link Failure ETAP Link Status Link Status ArmorPoint I/O PowerFlex POINT I/O POINT I/O All faults that are detectable at physical layer Physical layer failure detected by protocol-aware node Status message sent by ring node and received by ring supervisor

Device Level Ring (DLR) Protocol Device-level Ring Topology CompactLogix Controller Forwarding Forwarding Active Ring Supervisor Stratix 5700 IE Switch Link Failure ETAP ArmorPoint I/O PowerFlex POINT I/O POINT I/O After failure detection, ring supervisor unblocks blocked port Network configuration is now a linear topology Fault location is readily available via diagnostics

Device Level Ring (DLR) Protocol Device-level Ring Topology CompactLogix Controller Forwarding Active Ring Beacon Supervisor Stratix 5700 IE Switch Beacon Blocking Announce Announce ETAP ArmorPoint I/O PowerFlex Some IACS applications, such as safety and motion require network convergence times faster than what switch oriented resiliency protocols can provide IACS Device resiliency protocols, such as the Device Level Ring, DLR, provide network convergence in the 1-3 ms range for simple automation device networks Device Level Ring is a Layer 2 protocol that provides link-level, physical redundancy This is 2-port embedded switch technology managed by the ODVA, which Cisco and Rockwell Automation are principle members of POINT I/O POINT I/O Once ring is restored, supervisor hears beacon on both ports, and transitions to normal ring mode, blocking one port

Linear Network 50 Nodes Max and Up to 100 Meters between Devices

Linear Network Primary Advantages The network simplifies installation and reduces wiring and installation costs. The Network requires no special software configuration. Embedded switch products offer improved CIP Sync application performance on linear networks Disadvantage The primary disadvantage of a linear network is any break of the cable disconnects all devices downstream from the break from the rest of the network.

Why use DLR and Linear with Motion? Switch Topology QoS PTP Embedded Linear Direct Yes Plant Network PowerFlex Drives PanelView Plus CMX L36ERM Stratix 5700 CIP Encoders Point I/O Adapters Kinetix5500 ArmorBlock I/O

QoS What is QoS? Quality of Service (QoS): Provides prioritization of some packets at the expense of others The key to QoS is the policy that is being enforced When describing QoS, I really like the airport boarding gate analogy. (Click) Think of the chaos and delays that would occur if the plane was boarded by all passengers at once. Instead, the airlines have adopted a priority policy where First Class passengers board first and then coach is boarded by zones. This policy is designed to eliminate congestion on the jetway and in the isles of the plane. Source: See Knowledgebase article #66325 for more on this subject.

PTP What is PTP? CIP Sync = Precision Time Protocol (PTP) = PTP v2 = IEEE 1588 PTP: Device time synchronization Clock drift Time stamp compensation Grandmaster (GM) You hear it described a lot of ways, but PTP or Precision Time Protocol is the mechanism used to synchronize device clocks over a network. The PTP used today is based on the updated IEEE 1588 standard from 2008. It is also know as PTP v2 or CIP Sync, but we will refer to it as simply “PTP” throughout this presentation. Since time synchronization between devices is crucial to automation tasks and because the individual device clocks can drift over time, the PTP mechanism is used to constantly re-synchronize these clocks. Every second, the grandmaster clock sends an update out on the network. This is designed to keep all of the clocks within 100ns of each other. But what happens if these updates get delayed or have to go through several switches to get to the end devices? Slaves

Connects embedded and non-embedded switch devices Star Topology Star topology enables the connection of a broad range of devices – embedded switch and non-embedded switch devices Add/remove devices without impacting the operation of other devices Mix with linear connected devices to optimize switch port usage and reduce system cost (one port per line of devices) Star Device Star Switch (Star) Star topology connect a broad range of devices – embedded switch and non-embedded switch devices add/remove drives on-line without impacting the operation of other drives mix with linear connected devices to optimize switch port usage and reduce system cost (one port per line of devices) (Linear & Star) Ethernet controllers with embedded switch support allow integration of linear and star topologies (Ring) Ethernet controllers with embedded switch support allow device level ring (DLR) for single fault tolerate operation and failure location diagnostics maximize machine uptime reduce scrap product improve MTTR Connects embedded and non-embedded switch devices

Hybrid Topology Combine topologies to meet your application requirements Linear, Ring, Star Switch, Device with embedded switch Hybrid – Device

Reference Architectures Solutions Agenda JF Switches Reference Architectures Solutions 28 28

Switch Considerations GW Advantages Disadvantages Managed Switches (ie. Statix 5700) Unmanaged Switches (ie. Stratix 2000) Embedded Switches (ie. CompactLogix controller) Segmentation services (VLANs) Diagnostic information Security services Prioritization services (QoS) Multicast management services Network resiliency Loop prevention More expensive Requires some level of support and configuration to start up No management capabilities No security No diagnostic information Difficult to troubleshoot No resiliency support No loop prevention Inexpensive Simple to set up There are 3 classes of switch options for a machine, some very large machines may get into layer 3 switches as well. Managed switches have the most options and functionality for maximum flexibility, unmanaged switches are very simple to use but have the least flexibility. Embedded switches are included with the end device, creating different topology options. They do have managed features, and because they are embedded in the end device will require no configuration (if you want to use them in a ring, at least one device must have the supervisor box checked in software, ETAPs also have configuration capability for port mirroring for diagnostic purposes). They will already have information about the nature of the application they will be used in because they are embedded in the end device. For example, an embedded switch contained inside of an I/O module will be ready out of the box to run that kind of application. Diagnostic information Prioritization services (QoS) Time Sync Services (1588 Transparent Clock) Network resiliency Loop prevention Limited management capabilities May require minimal configuration for Ring Topology

Switches 101 Managed Switches: Advantages: Disadvantages: Ability to segment network (minimize network latency and jitter) Network diagnostics and security features Some provide security protocols Some provide loop prevention protocols Some provide Quality of Service (QoS) Some support precision time protocol (PTP) Disadvantages: Higher initial cost than an unmanaged switch Some technical knowledge needed for setup A managed switch offers a lot of features and options, but the key is knowing what they are and how to utilize them. Buying a switch with security or diagnostics like port utilization and a fault log is smart, but it’s only helpful if your technical staff knows how to use them. Some managed switches offer Quality of Service or QoS which prioritizes certain network traffic at the expense of others. Some switches support the concept of Precision Time Protocol or PTP which helps with time synchronization on the wire. It helps to understand how we use these features for our industrial automation needs when selecting your switch.

Switches 101 Unmanaged Switches: Advantages: Disadvantages: Direct traffic to appropriate destinations, without flooding to unnecessary ports Ideal for small, isolated networks Lower initial investment than a managed switch Disadvantages: No diagnostics No security No loop prevention No QoS or prioritization of some traffic at the expense of other traffic No PTP or adjustment to time stamp after passing packet Clearly, the backbone of the success we’ve seen in the Midrange products is the reduced system cost. CompactLogix controllers, Kinetix 350 servos, PF4-series drives and POINT I/O all work to reduce the initial investment for our machine builders while still providing the Logix advantage. We’ve been able to win business with this platform that we simply couldn’t have before. Adding a managed switch with Quality of Service and PTP to a 1 or 2 axis system could (in some cases) double its cost. Having a compact, industrially rated unmanaged switch like the Stratix 2000 has probably enabled you to win these orders, but you should always consider the trade-off. This switch provides no diagnostics, security or loop prevention, does not prioritize application traffic over other traffic and does not adjust the time stamp of packets as they pass through the switch. Finally, it should be noted that some vendors provide an unmanaged switch by name which actually includes Quality of Service, but we’ll talk about this feature shortly. Some of you have experienced issues with these types of switches, and this presentation (slide 11) should help you to understand why and how to avoid future issues.

Plant Convergence at the Machine Level Agenda JF Plant Convergence at the Machine Level Summary of Machine network best practices Talk through 2 example machine networks - Process Skid example - Packaging with Motion and Safety example 32 32

Segmentation Islands of Automation with Isolated Networks VFD Drive HMI Controller I/O Servo Drive I/O I/O I/O HMI HMI VFD Drive Controller Servo Drive VFD Drive I/O Controller Instrumentation

Plant Network Connectivity Options JF CompactLogix L36ERM chassis Plant Network EtherNet/IP EtherNet/IP ArmorBlock I/O Kinetix 5500 With a single interface on the controller, what are my options to integrate this controller into my end users network infrastructure. It is a good idea to segment the machine from the plant network. A second Controller interface is only one way to do that. POINT I/O PV+ EOI 34

Convergence Options ???? 1. Physical Controls Network with 1 Convergence Point 2. Vlan’s using existing plant back bone 3. Network Address Translation (NAT) 4. CIP Bridges (Confidential – For Internal Use Only) Copyright © 2008 Rockwell Automation, Inc. All rights reserved.

Industrial Automation Network Catalyst 3750 StackWise Switch Stack Cell/Area Zones Levels 0–2 Rockwell Automation Stratix 8000 Layer 2 Access Switch HMI Controller HMI Drive Controller Drive Drive HMI I/O I/O I/O Controller I/O Cell/Area Zone #1 Redundant Star Topology Flex Links Resiliency Cell/Area Zone #2 Ring Topology Resilient Ethernet Protocol (REP) Cell/Area Zone #3 Bus/Star Topology

VLAN Segmentation Enterprise-wide Plant-wide Site-wide Business Systems Levels 4 & 5 – Data Center Enterprise Zone Level 3.5 - IDMZ Level 3 - Site Operations Industrial Zone Plant-wide Site-wide Operation Systems Physical or Virtualized Servers FactoryTalk Application Servers & Services Platform Network Services – e.g. DNS, AD, DHCP, AAA Remote Access Server (RAS) Call Manager Storage Array Plant LAN – VLAN17 - Layer 2 Domain Plant IP - Subnet 10.17.10.0/24 Levels 0-2 Cell/Area Zones LAN – separate from Plant IP Schema – separate from Plant, all nodes within the Cell/Area Zone must have a unique IP address, IP addresses can be reused across Cell/Area Zones In this example we use the Stratix 5700 to provide Network Address Translation between each Cell/Area Zone and the plant-wide / site-wide network If Cell/Area Zones need to communicate between each other, it must go through a Layer 3 switch that can provide inter-VLAN routing Cell/Area Zone #1 VLAN10 Subnet 192.168.1.0/24 Cell/Area Zone #2 VLAN20 Subnet 192.168.1.0/24 Cell/Area Zone #3 VLAN30 Subnet 192.168.1.0/24 37

NAT Concept Identical Machines are Less Expensive to Build, Install and Maintain Plant Wide Network Machine A Machine B Machine C 9300-ENA with NAT Application PanelView™ 192.168.1.2 Controller 192.168.1.1 Controller 192.168.1.1 Controller 192.168.1.1 Identically configured machines can be connected on a plant wide network . The Ethernet devices that need to communicate on the plant wide network can be translated to different IP addresses, allowing them to coexist. Machines that are identically configured are less expensive to build, install and maintain Just as important, the network traffic from devices that are not translated is blocked from reaching the plant wide network. By translating only the devices needed, the number of IP addresses required on the plant wide network can be significantly reduced. The 9300-ENA allows machines on plant wide networks using the addresses that the IT dept. has specified, without changing machine configuration. Network Switch Network Switch Network Switch Kinetix 192.151.186.1.3 Kinetix 192.151.186.1.3 Kinetix 192.151.186.1.3

Setup The 9300-ENA’s 1:1 NAT concept is simple. IP addresses of Ethernet Devices on the local network can be translated to different IP addresses on the network connected to “uplink” port. It is a very important to realize that network traffic to/from devices that are not in the translation table (or in the table but not active) is blocked. No Ethernet traffic generated on these devices can go through the module. No Ethernet traffic from the “public” network can reach these devices.

CompactLogix™ 5370 Controller Advantages to NAT Advantages to this approach Allows OEM to keep their machine isolated from the End User’s network to limit impact on machine performance Allows OEM to set IP address of their machine without consideration for requirements End User may have Allows OEM to set all IP addresses the same for multiple machines 9300-ENA blocks machine IP address from End User view & is easy to configure User can see CIP and non-CIP devices through 9300-ENA CompactLogix™ 5370 Controller PanelView Plus Compact Switch 9300-ENA Kinetix 350 9300-ENA can be configured to allow access to the controller on the plant-wide network, but blocks traffic and prevents access to other devices on the control network 40

NAT Capable Devices Stratix 5700™ Stratix 5900TM 9300-ENA 1783-NAT Catalog Number Integrated - 1783-BMS10 GGN or 1783-BMS20 GGN 1783-SRKIT 1783-NATR Port count 10 (8 + 2 Gb) or 20 (18 + 2 Gb) port versions 1 Gb, 4 FE 2, plus configuration port 2 Configuration Web Interface Integrated Architecture® Studio 5000® Interface Command Line Interface Stratix™ Configurator SW Performance Best - HW Wire-speed Translations Better – SW implementation Better - SW implementation Nested NAT 2 levels Ports (Uplinks) Connecting to Plant Up to 2 -1 Gb Support for Ring (REP) and Redundant Star 1 - 1 Gb 1 - 100 Mb port 1-100Mb port Translations supported 128 with subnets No fixed limit 128 32

CIP Bridges Information Network Isolated networks - two NICs for physical network segmentation Benefits Clear network ownership demarcation line Challenges Limited visibility to control network devices for asset management Limited future-ready capability Control Network

Exercise - OEM Machine Builder Specification: Compact Logix 1769-L33ERM PanelView Plus6 (Located in Main Panel) 2 Point IO Adapters (Located in the Field) 2 K350 Drives (Located in Main Panel) 2 PF 525 Drives (Located in Main Panel)

Option 1

Option 2

Exercise - OEM Machine Builder Specification: Compact Logix 1769-L33ERM PanelView Plus6 (Located in Main Panel) 2 Point IO Adapters (Safety) 2 K350 Drives (Located in Main Panel) 2 PF 525 Drives (Located in Main Panel)

Option 1

Option 2

Exercise - OEM Machine Builder Specification: Compact Logix 1769-L33ERM PanelView Plus6 (Located in Main Panel) 2 Point IO Adapters (Safety) 2 K350 Drives (Located in Main Panel) 2 PF 525 Drives (Located in Main Panel) Need Plant Connection (No IT Support)

NAT

Agenda Tools and Where to learn more Reference Architectures Solutions JF Tools and Where to learn more Reference Architectures Solutions 51 51

Tools - IAB

Tools - IAB

Tools - IAB

Tools - IAB

Tools – Ethernet Capacity Tool

Additional Material ODVA Website: http://www.odva.org/ Media Planning and Installation Manual http://www.odva.org/Portals/0/Library/Publications_Numbered/PUB00148R0_EtherNetIP_Media_Planning_and_Installation_Manual.pdf Network Infrastructure for EtherNet/IP: Introduction and Considerations http://www.odva.org/Portals/0/Library/Publications_Numbered/PUB00035R0_Infrastructure_Guide.pdf Device Level Ring http://www.odva.org/Portals/0/Library/CIPConf_AGM2009/2009_CIP_Networks_Conference_Technical_Track_Intro_to_DLR_PPT.pdf The CIP Advantage http://www.odva.org/default.aspx?tabid=54

Additional Material Rockwell Automation Networks Website: http://www.ab.com/networks/ EtherNet/IP Website: http://www.ab.com/networks/ethernet/ Media Website: http://www.ab.com/networks/media/ethernet/ Embedded Switch Technology Website: http://www.ab.com/networks/switches/embedded.html Publications: ENET-AP005-EN-P Embedded Switch Technology Manual ENET-UM001G-EN-P EtherNet/IP Modules in Logix5000 Control Systems …. provides connection and packet rate specs for modules 1783-UM003 Stratix 8000 and Stratix 8300 Ethernet Managed Switches User Manual ENET-WP0022 Top 10 Recommendations for plant-wide EtherNet/IP Deployments ENET-RM002A-EN-P Ethernet Design Considerations Reference Manual ENET-AT004A-EN-E Segmentation Methods within the Cell/Area Zone ENET-RM003A-EN-P Embedded Switch Technology Reference Architectures Network and Security Services Website: http://www.rockwellautomation.com/services/networks/

Additional Material Cisco and Rockwell Automation Alliance Websites http://www.ab.com/networks/architectures.html Design Guides Converged plant-wide Ethernet (CPwE) Application Guides Fiber Optic Infrastructure Application Guide Education Series Whitepapers Top 10 Recommendations for plant-wide EtherNet/IP Deployments Securing Manufacturing Computer and Controller Assets Production Software within Manufacturing Reference Architectures Achieving Secure Remote Access to Plant-Floor Applications and Data

Thank You!!!