Practical use of Ethernet OAM Joerg Ammon Systems Engineer Service Provider May 2011 © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 1

Overview A variety of Operations, Administration, and Management (OAM) protocols and tools were developed in recent years for MPLS, IP, and Ethernet networks. These tools provide unparalleled power for an operator to proactively manage networks and customer Service Level Agreements (SLAs). This session reviews the various OAM tools available in MPLS/IP/ Ethernet networks at various layers of the stack and recommends/reviews best practices for choosing the right OAM protocol to use in a network. May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information 2

OAM Tools Scope of this presentation May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information 3 Management Plane (NMS,EMS) Network Plane (Network Elements) Scope of this presentation: OAM tools across network elements OAM&P Scope of this presentation is within network plane only (not management plane)

OAM Layering OAM is layered… Service Layer OAM Network Layer OAM Transport Layer OAM... and hierarchical For example, service layer for Operator A is transport layer for the service provider Each layer supports its own OAM mechanisms Operator A has an MPLS network and uses MPLS OAM tools Operator B has an Ethernet network and uses Ethernet OAM tools May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information 4 Service Provider Operator A Network Customer Network Customer Location 1 Customer Location 2 Operator B Network Service OAM MPLS OAM (Operator A) Ethernet OAM (Operator B) MPLS Link OAM Ethernet

Layer 2 Trace Port Loop Detection UDLD Single-link LACP Keep-alive 802.1ag CFM/ Y.1731 PM OAM Tools Business Problem Fault detection, verification, and isolation at every level Proactive detection of service degradation Performance Monitoring (PM) and SLA verification Business Problem Fault detection, verification, and isolation at every level Proactive detection of service degradation Performance Monitoring (PM) and SLA verification Each layer has its own best-suited OAM tools Brocade Solution Standards-based, end-to-end OAM Comprehensive/scalable MPLS, IP, and Ethernet OAM tools Brocade Solution Standards-based, end-to-end OAM Comprehensive/scalable MPLS, IP, and Ethernet OAM tools May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information ah EFM OAM LSP Ping and TracerouteBFD for RSVP-TE LSPs Ping and TracerouteBFD for OSPF and IS-IS VRF Ping and Traceroute (Layer 3 VPN) 802.1ag CFM for VPLS/VLL Y.1731 PM for VPLS/VLL (Layer 2 VPN) VPN IP Layer 2 MPLS

Layer 2 OAM + Layer 2 VPN CFM/PM: 802.1ag CFM, Y.1731 PM May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information 6

Layer 2 OAM + Layer 2 VPN CFM/PM: 802.1ag CFM, Y.1731 PM May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information 7 Layer 2 Trace Port Loop Detection UDLD Single-link LACP Keep-alive 802.1ag CFM/ Y.1731 PM 802.3ah EFM OAM LSP Ping and TracerouteBFD for RSVP-TE LSPs Ping and TracerouteBFD for OSPF and IS-IS VRF Ping and Traceroute (Layer 3 VPN) 802.1ag CFM for VPLS/VLL Y.1731 PM for VPLS/VLL (Layer 2 VPN) VPN IP Layer 2 MPLS

IEEE 802.1ag CFM Facilitates Path discovery Fault detection Fault verification and isolation Fault notification Fault recovery Supports Continuity Check Messages (CCMs) LinkTrace Loopback messages Connectivity Fault Management (CFM) Brocade Implementation Support for minimum CCM timers (3.3 ms) using hardware offload 3.3 ms, 10 ms, 100 ms, 1 s, 1 min, 10 min Support for MIPs and up/down MEPs Support for all eight MD levels (0-7) Support for the following types of endpoints/services VLANs and VPLS/VLL endpoints Brocade Implementation Support for minimum CCM timers (3.3 ms) using hardware offload 3.3 ms, 10 ms, 100 ms, 1 s, 1 min, 10 min Support for MIPs and up/down MEPs Support for all eight MD levels (0-7) Support for the following types of endpoints/services VLANs and VPLS/VLL endpoints 8 Service Provider Operator A Network Customer Network Customer location 1 Customer location 2 Operator B Network MEP MIP Customer CFM Service Provider CFM Operator A CFMOperator B CFM May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

IEEE 802.1ag CFM MD (Maintenance Domain) The part of a network for which faults in Layer 2 connectivity can be managed MEP (Maintenance End Point) A Maintenance Point (MP) at the edge of a domain that actively sources CFM messages Two types: up (inward*) MEP or down (outward) MEP MIP (Maintenance Intermediate Point) A maintenance point internal to a domain that only responds when triggered by certain CFM messages MA (Maintenance Association) A set of MEPs established to verify the integrity of a single service instance (a VLAN or a VPLS) MD (Maintenance Domain) The part of a network for which faults in Layer 2 connectivity can be managed MEP (Maintenance End Point) A Maintenance Point (MP) at the edge of a domain that actively sources CFM messages Two types: up (inward*) MEP or down (outward) MEP MIP (Maintenance Intermediate Point) A maintenance point internal to a domain that only responds when triggered by certain CFM messages MA (Maintenance Association) A set of MEPs established to verify the integrity of a single service instance (a VLAN or a VPLS) Terminology ME (Maintenance Entity) A point-to-point relationship between two MEPs within a single MA MD Level An integer from 0 to 7 in a field in a CFM PDU that is used, along with the VLAN ID, to identify which MIPs/MEPs would be interested in the contents of a CFM PDU ME (Maintenance Entity) A point-to-point relationship between two MEPs within a single MA MD Level An integer from 0 to 7 in a field in a CFM PDU that is used, along with the VLAN ID, to identify which MIPs/MEPs would be interested in the contents of a CFM PDU 9 Service Provider Operator A Network Customer Network Customer location 1 Customer location 2 Operator B Network MEP MIP Customer MA Service Provider MA Operator A MAOperator B MA MD level 5 (7, 6, or 5) MD level 3 (4 or 3) MD level 1 (2, 1, or 0) ME Down MEP UP MEP (*): “inward” in respect to the device May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

IEEE 802.1ag CFM Continuity Check Message (CCM) A periodic hello message multicast by an MEP within the maintenance domain LinkTrace Message (LTM) A multicast message used by a source MEP to trace the path to other MEPs and MIPs in the same domain All reachable MIPs and MEPs respond back with a Link Trace Unicast Reply (LTR) The originating MEP can then determine the MAC addresses of all MIPs and MEPs belonging to the same Maintenance Domain Loopback Message (LBM) Used to verify the connectivity between a MEP and a peer MEP or MIP A loopback message is initiated by a MEP with a destination MAC address set to the desired destination MEP or MIP (Unicast) The receiving MIP or MEP responds to the Loopback message with a Loopback Reply (LBR) (Unicast) A loopback message helps a MEP identify the precise location of a fault along a given path Connectivity Check, LinkTrace, and Loopback Messages 10 Periodic CCM (multicast) Periodic CCM MEP LTM (multicast) LTR (Unicast) MEP MIP LBM (Unicast) LBR MEP LTR (Unicast) May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

Hierarchical Fault Detection Customer detects fault using Continuity Check and locates fault using Link Trace Provider A detects fault using Continuity Check and locates fault using Link Trace Provider B detects fault using Continuity Check, but isolates fault using MPLS OAM (see MPLS OAM section) A service provider (not shown) would detect this fault in a similar way using Continuity Check and Link Trace from CPEs (Customer Premise Equipment) Example: fault in Operator B network (an MPLS Network) P MEP MIP 3: Provider A’s Continuity Check detects end-to-end fault 4: Provider A Link Traces isolate fault inside Provider B’s network 1: Customer Continuity Check detects end-to-end fault 2: Customer Link Traces isolate fault past customer MIPs 5: Provider B’s Continuity Check detects service fault Operator BOperator A (Location A1) Customer Network (Site 1) Operator A (Location A2) Customer Network (Site 2) Fault Localized PE MPLS (VPLS/VLL) MIPs and MEPs at VPLS/VLL endpoints Fault 11 May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

IEEE 802.1ag Configuration Example Configure a down MEP on CE1 CE1(config)#cfm-enable CE1(config-cfm)#domain-name CUST_1 level 7 CE1(config-cfm-md-CUST_1)#ma-name ma_5 vlan-id 30 priority 3 CE1(config-cfm-md-CUST_1-ma-ma_5)#ccm- interval 10-second CE1(config-cfm-md-CUST_1-ma-ma_5)#mep 1 down vlan 30 port ethe 1/1 CE1(config-cfm-md-CUST_1-ma- ma_5)#remote-mep 2 to 2 Configure a down MEP on CE1 CE1(config)#cfm-enable CE1(config-cfm)#domain-name CUST_1 level 7 CE1(config-cfm-md-CUST_1)#ma-name ma_5 vlan-id 30 priority 3 CE1(config-cfm-md-CUST_1-ma-ma_5)#ccm- interval 10-second CE1(config-cfm-md-CUST_1-ma-ma_5)#mep 1 down vlan 30 port ethe 1/1 CE1(config-cfm-md-CUST_1-ma- ma_5)#remote-mep 2 to 2 To verify end-to-end connectivity between CE1 and CE2 12 CE1CE /1 2/1 PE1PE2 MPLS VLL Configure a down MEP on CE2 CE2(config)#cfm-enable CE2(config-cfm)#domain-name CUST_1 level 7 CE2(config-cfm-md-CUST_1)#ma-name ma_5 vlan-id 30 priority 3 CE2(config-cfm-md-CUST_1-ma-ma_5)#ccm- interval 10-second CE1(config-cfm-md-CUST_1-ma-ma_5)#mep 2 down vlan 30 port ethe 2/1 CE1(config-cfm-md-CUST_1-ma- ma_5)#remote-mep 1 to 1 Configure a down MEP on CE2 CE2(config)#cfm-enable CE2(config-cfm)#domain-name CUST_1 level 7 CE2(config-cfm-md-CUST_1)#ma-name ma_5 vlan-id 30 priority 3 CE2(config-cfm-md-CUST_1-ma-ma_5)#ccm- interval 10-second CE1(config-cfm-md-CUST_1-ma-ma_5)#mep 2 down vlan 30 port ethe 2/1 CE1(config-cfm-md-CUST_1-ma- ma_5)#remote-mep 1 to 1 LSP ping and LSP traceroute tools would be used inside the MPLS network to detect and diagnose LSP failures Create a VLL instance (PE1) PE1(config)#router mpls PE1(config-mpls)vll pe1-to-pe2 30 PE1(config-mpls-vll)vll-peer PE1(config-mpls-vll)untagged ethe 1/1 PE1(config-mpls-vll)vlan 30 PE1(config-mpls-vll-vlan)tagged ethe 1/1 Configure CFM on PE1 PE1(config)#cfm-enable PE1(config-cfm)#domain-name CUST_1 level 7 PE1(config-cfm-md-CUST_1)#ma-name ma_5 vll-id 30 priority 3 PE1(config-cfm-md-CUST_1-ma-ma_5)#ccm- interval 10-second In the above configuration, a MIP is created by default on the VLL port. Create a VLL instance (PE1) PE1(config)#router mpls PE1(config-mpls)vll pe1-to-pe2 30 PE1(config-mpls-vll)vll-peer PE1(config-mpls-vll)untagged ethe 1/1 PE1(config-mpls-vll)vlan 30 PE1(config-mpls-vll-vlan)tagged ethe 1/1 Configure CFM on PE1 PE1(config)#cfm-enable PE1(config-cfm)#domain-name CUST_1 level 7 PE1(config-cfm-md-CUST_1)#ma-name ma_5 vll-id 30 priority 3 PE1(config-cfm-md-CUST_1-ma-ma_5)#ccm- interval 10-second In the above configuration, a MIP is created by default on the VLL port. Create a VLL instance (PE2) PE2(config)#router mpls PE2(config-mpls)vll pe2-to-pe1 30 PE2(config-mpls-vll)vpls-peer PE2(config-mpls-vll)untagged ethe 2/1 PE2(config-mpls-vll)vlan 30 PE2(config-mpls-vll-vlan)tagged ethe 2/1 Configure CFM on PE2 PE2(config)#cfm-enable PE2(config-cfm)#domain-name CUST_1 level 7 PE2(config-cfm-md-CUST_1)#ma-name ma_5 vll-id 30 priority 3 PE2(config-cfm-md-CUST_1-ma-ma_5)#ccm- interval 10-second In the above configuration, a MIP is created by default on the VLL-endpoint. Create a VLL instance (PE2) PE2(config)#router mpls PE2(config-mpls)vll pe2-to-pe1 30 PE2(config-mpls-vll)vpls-peer PE2(config-mpls-vll)untagged ethe 2/1 PE2(config-mpls-vll)vlan 30 PE2(config-mpls-vll-vlan)tagged ethe 2/1 Configure CFM on PE2 PE2(config)#cfm-enable PE2(config-cfm)#domain-name CUST_1 level 7 PE2(config-cfm-md-CUST_1)#ma-name ma_5 vll-id 30 priority 3 PE2(config-cfm-md-CUST_1-ma-ma_5)#ccm- interval 10-second In the above configuration, a MIP is created by default on the VLL-endpoint. May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

ITU-T Y.1731 Performance Management Standards-based performance management for Ethernet networks Interoperates in a multivendor environment Supports high-precision, on-demand measurement of round-trip SLA parameters Frame Delay (FD) Frame Delay Variation (FDV) Measurements done between MEPs 13 MEP Frame Delay Frame Delay Variation ETH-DM Brocade MLX MEP: Management Enforcement Point ETH-DM: Ethernet Delay Measurement Benefits SLA monitoring and verification Applicability Aggregation, metro, and core networks Delay-sensitive applications, such as voice Differentiated services with SLA guarantees Brocade differentiation Hardware-based time-stamping mechanism Measurements with microsecond granularity Y.1731 PM for VPLS/VLL MEP May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

ITU-T Y.1731 Performance Management Example 14 NetIron# cfm delay_measurement domain md2 ma ma2 src-mep 3 target-mep 2 Y1731: Sending 10 delay_measurement to 0012.f2f7.3931, timeout 1000 msec Type Control-c to abort Reply from 0012.f2f7.3931: time= us Reply from 0012.f2f7.3931: time= us Reply from 0012.f2f7.3931: time= us Reply from 0012.f2f7.3931: time= us Reply from 0012.f2f7.3931: time= us Reply from 0012.f2f7.3931: time= us Reply from 0012.f2f7.3931: time= us Reply from 0012.f2f7.3931: time= us Reply from 0012.f2f7.3931: time= us Reply from 0012.f2f7.3931: time= us sent = 10 number = 10 A total of 10 delay measurement replies received. Success rate is 100 percent (10/10) ==================================================================== Round Trip Frame Delay Time : min = us avg = us max = us Round Trip Frame Delay Variation : min = 45 ns avg = 839 ns max = us ==================================================================== MEP 3 ETH-DM Brocade MLX MEP 2 May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

Link OAM Supports point-to-point (single) link OAM Monitors and supports troubleshooting individual links Standards-based for Ethernet networks Interoperates in a multivendor environment Supports Fault detection and notification (alarms) Discovery Remote failure indication Loopback testing IEEE 802.3ah Ethernet First Mile (EFM) OAM 802.3ah OAM 802.3ah OAM NetIron#show link-oam info detail ethernet 1/1 OAM information for Ethernet port: 1/1 link-oam mode: active link status: up oam status: up Local information multiplexer action: forward parse action: forward stable: satisfied state: up loopback state: disabled dying-gasp: false critical-event: false link-fault: false Remote information multiplexer action: forward parse action: forward stable: satisfied loopback support: disabled dying-gasp: false critical-event: false link-fault: false NetIron#show link-oam info detail ethernet 1/1 OAM information for Ethernet port: 1/1 link-oam mode: active link status: up oam status: up Local information multiplexer action: forward parse action: forward stable: satisfied state: up loopback state: disabled dying-gasp: false critical-event: false link-fault: false Remote information multiplexer action: forward parse action: forward stable: satisfied loopback support: disabled dying-gasp: false critical-event: false link-fault: false 15 May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

Layer 2 OAM Summary 16 Layer 2 Trace Port Loop Detection UDLD Single-Link Keep-Alive 802.1ag CFM Y.1731 PM 802.3ah EFM OAM Intended Application Layer 2 network troubleshooting, detection of mis-configuration Link keep-alive Single-link keep-alive Service verification Performance (SLA) verification Customer access verification Supports Layer 2 topology discovery, Layer 2 loop detection Layer 2 loop detection Link keep-alive Single-link keep-alive Layer 2 Connectivity Check, Link Trace, Loopback One-way delay and delay variation Single-link OAM: Fault Detection, Discovery, Loop-back, and so on GenerationManualAutomatic CC: auto LT, LB: manual Manual Auto, Manual (LB) StandardNo Yes Remember: OAM is layered and hierarchical (service OAM for an operator is transport OAM for a service provider) May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

MPLS OAM 17 May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

MPLS OAM 18 Layer 2 Trace Port Loop Detection UDLD Single-link LACP Keep-alive 802.1ag CFM/ Y.1731 PM 802.3ah EFM OAM LSP Ping and TracerouteBFD for RSVP-TE LSPs Ping and TracerouteBFD for OSPF and IS-IS VRF Ping and Traceroute (Layer 3 VPN) 802.1ag CFM for VPLS/VLL Y.1731 PM for VPLS/VLL (Layer 2 VPN) VPN IP Layer 2 MPLS May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

LSP Ping and LSP Traceroute LSP Ping and LSP Traceroute provide OAM functionality for MPLS networks based on RFC LSP Ping and LSP Traceroute tools provide a mechanism to detect MPLS data plane failure. MPLS echo requests follow the same data path that normal MPLS packets would traverse. LSP Ping is used to detect data plane failure and to check the consistency between the data plane and the control plane. LSP Traceroute is used to isolate the data plane failure to a particular router and to provide LSP path tracing. MPLS OAM tools 19 May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

LSP Ping The basic idea is to verify that packets that belong to a particular Forwarding Equivalence Class (FEC) actually end their MPLS path on a Label Switching Router (LSR) that is an egress for that FEC. LDP LSP Ping and RSVP LSP Ping are supported. 20 NetIron# ping mpls ldp Send 5 80-byte MPLS Echo Requests for LDP FEC /32, timeout 5000 msec Type Control-c to abort !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max=0/1/1 ms. Syntax: ping mpls ldp... options NetIron# ping mpls ldp Send 5 80-byte MPLS Echo Requests for LDP FEC /32, timeout 5000 msec Type Control-c to abort !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max=0/1/1 ms. Syntax: ping mpls ldp... options LDP LSP Ping NetIron# ping mpls rsvp lsp toxmr2frr-18 Send 5 92-byte MPLS Echo Requests over RSVP LSP toxmr2frr-18, timeout 5000 msec Type Control-c to abort !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max=0/1/5 ms. Syntax: ping mpls rsvp lsp | session... options NetIron# ping mpls rsvp lsp toxmr2frr-18 Send 5 92-byte MPLS Echo Requests over RSVP LSP toxmr2frr-18, timeout 5000 msec Type Control-c to abort !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max=0/1/5 ms. Syntax: ping mpls rsvp lsp | session... options RSVP LSP Ping MPLS Network PE P (LER) (LSR) PE (LER) LSP LSP Ping Echo Request Echo Reply May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

LSP Traceroute With LSP traceroute, an echo request packet is sent to the control plane of each transit LSR, which confirms that it is a transit LSR for this path. Transit LSRs return echo replies. LDP LSP Ping and RSVP LSP Ping are supported. 21 NetIron# traceroute mpls ldp Trace LDP LSP to /32, timeout 5000 msec, TTL 1 to 30 Type Control-c to abort 1 10ms return code 3(Egress) Syntax: traceroute mpls ldp... options NetIron# traceroute mpls ldp Trace LDP LSP to /32, timeout 5000 msec, TTL 1 to 30 Type Control-c to abort 1 10ms return code 3(Egress) Syntax: traceroute mpls ldp... options LDP LSP Traceroute NetIron # traceroute mpls rsvp lsp toxmr2frr-18 Trace RSVP LSP toxmr2frr-18, timeout 5000 msec, TTL 1 to 30 Type Control-c to abort 1 1ms return code 3(Egress) Syntax: traceroute mpls rsvp lsp | session... options NetIron # traceroute mpls rsvp lsp toxmr2frr-18 Trace RSVP LSP toxmr2frr-18, timeout 5000 msec, TTL 1 to 30 Type Control-c to abort 1 1ms return code 3(Egress) Syntax: traceroute mpls rsvp lsp | session... options RSVP LSP Traceroute MPLS Network PE P (LER) (LSR) PE (LER) LSP LSP Traceroute Echo Request Echo Replies May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

MPLS OAM Summary LSP PingLSP TracerouteBFD for RSVP-TE LSPs Intended Application To detect data plane failure and to check the consistency between the data plane and the control plane To isolate the data plane failure to a particular router and to provide LSP path tracing Fast data plane failure detection for RSVP LSPs SupportsConnectivity verification Connectivity troubleshooting, fault localization Fast data plane failure detection (link may be up, but data path is down) GenerationManual Automatic StandardYes 22 May 2011© 2011 Brocade Communications Systems, Inc. Company Proprietary Information

Observation September 2010© 2010 Brocade Communications Systems, Inc. Company Proprietary Information 23 ICMPPingCFM Operates atLayer 3Layer 2 SpecificationRFC792 RFC1208 (RFC 1983) 802.1ag PublishedSept 1981 March 1991 (Aug 1996) July 1983 Dec years of work for going down one layer of OAM

Thank You 24 © 2011 Brocade Communications Systems, Inc. Company Proprietary Information