1. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 1 ECSE-6660 Availability, Survivability, Protection/Restoration, Fast Re- Route http://www.pde.rpi.edu/ Or http://www.ecse.rpi.edu/Homepages/shivkuma/ Shivkumar Kalyanaraman Rensselaer Polytechnic Institute shivkuma@ecse.rpi.edu Based in part on slides of James Manchester (formerly Tellium, now RPI), and some NANOG presentations

2. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 2 q Availability: the driver… q Survivability: protection and restoration architectures q Fast-Reroute Overview

3. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 3 Availability: Impact of Outages Service Outage Impact 50msec0200msec2sec10sec 5min 30min "Hit" TriggerChange- over of CCS Links FCCReportable Packet(X.25)Disconnect Call- Dropping Dropping Private Line Disconnect May Drop VoicebandCalls APS 1stRange 2ndRange 3rdRange 4thRange 5thRange 6thRange Social/BusinessImpacts Disruptions cost a lot of money!

4. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 4 Market Drivers for Survivability n Customer Relations n Competitive Advantage n Revenue q Negative - Tariff Rebates q Positive - Premium Services q Business Customers q Medical Institutions q Government Agencies n Impact on Operations n Minimize Liability

5. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 5 Network Survivability: drivers q Availability: 99.999% (5 nines) => less than 5 min downtime per year q Since a network is made up of several components, the ONLY way to reach 5-nines is to add survivability in the face of failures… q Survivability = continued services in the presence of failures q Protection switching or restoration: mechanisms used to ensure survivability q Add redundant capacity, detect faults and automatically re-route traffic around the failure q Restoration: related term, but slower time-scale q Protection: fast time-scale: 10s-100s of ms… q implemented in a distributed manner to ensure fast restoration

6. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 6 Failure Types & Other Motivations q Types of failure: q Components: links, nodes, channels in WDM, active components, software… q Human error: backhoe fiber cut q Fiber inside oil/gas pipelines less likely to be cut q Systems: Entire COs can fail due to catastrophic events q Protection allows easy maintenance and upgrades : q Eg: switchover traffic when servicing a link… q Single failure vs multiple concurrent failures… q Goal: mean repair time << mean time between failures… q Protection also depends upon kind of application: q SONET/SDH: 60 ms (legacy drop calls threshold) q Do data apps really need this level of protection? q Survivability may hence be provided at several layers

7. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 7 Network Survivability Architectures Mesh Restoration Architectures Linear Protection Architectures Ring Protection Architectures

8. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 8 Network Availability & Survivability Availability is the probability that an item will be able to perform its designed functions at the stated performance level, within the stated conditions and in the stated environment when called upon to do so. Reliability Reliability + Recovery Availability =

9. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 9 Quantification of Availability Percent Availability N-NinesDowntime Time Minutes/Year 99% 2-Nines5,000 Min/Yr 99.9% 3-Nines500 Min/Yr 99.99% 4-Nines50 Min/Yr 99.999% 5-Nines5 Min/Yr 99.9999% 6-Nines.5 Min/Yr

10. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 10 PSTN : The Yardstick ? q Individual elements have an availability of 99.99% q One cut off call in 8000 calls (3 min for average call). Five ineffective calls in every 10,000 calls. Facility Entrance AN 0.01 % 0.005 % 0.02 % 0.005 % LELE NINI LELE NINI LDLD AN 0.01 % PSTN End-2-End Availability 99.94% NI : Network Interface LE : Local Exchange LD : Long Distance AN : Access Network Source : http://www.packetcable.com/downloads/specs/pkt-tr-voipar-v01-001128.pdf

11. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 11 Services Determine the Requirements on Network Availability Source : www.t1.org

12. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 12 IP Network Expectations ServiceDelayJitterLossAvailability Real Time Interactive (VOIP, Cell Relay..) LLLH Layer 2 & Layer 3 VPN’s (FR/Ethernet/AAL5) M Internet ServiceHHML Video ServicesLMMH H L L L : Low M : Medium H : High

13. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 13 Measuring Availability: The Port Method q Based on Port count in Network q Does not take into account the Bandwidth of ports e.g. OC-192 and 64k are both ports q Good for dedicated Access service because ports are tied to customers. (Total # of Ports X Sample Period) - (number of impacted port x outage duration) (Total number of Ports x sample period) x 100

14. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 14 The Port Method Example q 10,000 active access ports Network q An Access Router with 100 access ports fails for 30 minutes. q Total Available Port-Hours = 10,000*24 = 240,000 q Total Down Port-Hours = 100*.5 = 50 q Availability for a Single Day = (240000-50/240,000)*100 = 99.979166 %

15. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 15 The Bandwidth Method q Based on Amount of Bandwidth available in Network q Takes into account the Bandwidth of ports q Good for Core Routers (Total amount of BW X Sample Period) - (Amount of BE impacted x outage duration) (Total amount of BW in network x sample period) x 100

16. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 16 The Bandwidth Method Example q Total capacity of network 100 Gigabits/sec q An Access Router with 1 Gigabits/sec BW fails for 30 minutes. q Total BW available in network for a day = 100*24 = 2400 Gigabits/sec q Total BW lost in outage = 1*.5 = 0.5 q Availability for a Single Day = ((2400-0.5)/2,400)*100 = 99.979166 %

17. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 17 Defects Per Million q Used in PSTN networks, defined as number of blocked calls per one million calls averaged over one year. DPM = [ (number of impacted customers x outage duration) (total number of customers x sample period) ] x 10 -6

18. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 18 Defects Per Million Example q 10,000 active access ports Network q An Access Router with 100 access ports fails for 30 minutes. q Total Available Port-Hours = 10,000*24 = 240,000 q Total Down Port-Hours = 100*.5 = 50 q Daily DPM = (50/240,000)*1,000,000 = 208

19. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 19 Basic Ideas: Working and Protect Fibers

20. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 20 Protection Topologies - Linear q Two nodes connected to each other with two or more sets of links Working Protect Working Protect (1+1)(1:n)

21. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 21 q Two or more nodes connected to each other with a ring of links q Line vs. Drop interfaces q East vs. West interfaces Protection Topologies - Ring E W W E W EW E D L L WorkingProtect

22. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 22 Protection Topologies - Mesh q Three or more nodes connected to each other q Can be sparse or complete meshes q Spans may be individually protected with linear protection q Overall edge-to-edge connectivity is protected through multiple paths Working Protect

23. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 23 Topologies: Rings, # Fibers, Directionality DCC ADM ADM ADM DCC ADM ADM ADM 2 Fiber Ring Each Line Is Full Duplex DCC ADM ADM ADM 4 Fiber Ring Each Line Is Full Duplex DCC ADM ADM ADM Uni- vs. Bi- Directional All Traffic Runs Clockwise, vs Either Way

24. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 24 SONET: Automatic Protection Switching (APS) ADM Line Protection Switching Uses TOH Trunk Application Backup Capacity Is Idle Supports 1:n, where n=1-14 Automatic Protection Switching Line Or Path Based Line Or Path Based Revertive vs. Non-Revertive Revertive vs. Non-Revertive Restoration Times ~ 50 ms Restoration Times ~ 50 ms K1, K2 Bytes Signal Change K1, K2 Bytes Signal Change ADMADMADM Path Protection Switching Uses POH Access Line Applications Duplicate Traffic Sent On Protect 1+1 ADMADM

25. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 25 Protection Switching Terminology q 1+1 architectures - permanent bridge at the source - select at sink q m:n architectures - m entities provide protection for n working entities where m is less than or equal to n q allows unprotected extra traffic q most common - SONET linear 1:1 and 1:n q Coordination Protocol - provides coordination between controllers in source and sink q Required for all m:n architectures q Not required for 1+1 architectures unless they employ bi-directional protection switching

26. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 26 1+1 vs 1:n Working Protect Working Protect (1+1)(1:n)

27. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 27 SONET Linear 1+1 APS BRSW TX RX SW RX BR TX Working Protection Working Protection TX = Transmitter RX = Receiver BR = Bridge SW = Switch

28. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 28 SONET 1:1 Linear APS BRSW TXRX SW RX BR TX RX APS Channel TX = Transmitter RX = Receiver BR = Bridge SW = Switch Protection Working Protection

29. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 29 SONET Linear APS APS Controller Local SF/SD Detection Management Commands K1/K2 Bytes Linear APS States

30. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 30 Protection Switching: Terminology q Dedicated vs Shared: working connection assigned dedicated or shared protection bandwidth q 1+1 is dedicated, 1:n is shared q Revertive vs Non-revertive: after failure is fixed, traffic is automatically or manually switched back q Shared protection schemes are usually revertive q Uni-directional or bi-directional protection: q Uni: each direction of traffic is handled independent of the other. q Fiber cut => only one direction switched over to protection. Usually done with dedicated protection; no signaling required. q Bi-directional transmission on fiber (full duplex) => requires bi-directional switching & signaling required

31. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 31 Current Architectures: Ring Protection Today: multiple “stacked” rings over DWDM (different s)

32. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 32 Unidirectional Path Switched Ring (UPSR) A-B A-B B-A B-A Path Selection W P fiber 1 fiber 2 A B C D Failure-free State Bridge Bridge * One fiber is “working” and the other is “protect” at all nodes… * Traffic sent SIMULTANEOUSLY on working and protect paths… * Protection done at path layer (like 1+1)…

33. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 33 Unidirectional Path Switched Ring (UPSR) A-B B-A Path Selection D fiber 1 Path Selection A B fiber 2 C W P Failure State Bridge Bridge

34. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 34 UPSR: discussion q Easily handles failures of links, transmitters, receivers or nodes q Simple to implement: no signaling protocol or communication needed between nodes q Drawback: does not spatially re-use the fiber capacity because it is similar to 1+1 linear protection model q I.e. no sharing of protection (like m:n model) q BLSRs can support aggregate traffic capacities higher than transmission rate q UPSRs popular in lower-speed local exchange and access networks (traffic is hubbed into the core) q No specified limit on number of nodes or ring length of UPSR, only limited by difference in delays of paths

35. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 35 Deployment of UPSR and BLSR Regional Ring (BLSR) Intra-Regional Ring (BLSR) Access Rings (UPSR)

36. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 36 Bidirectional Line Switched Ring (BLSR/2) AÔCAÔCAÔCAÔC C ÔA AÔCAÔCAÔCAÔC Working Protection 2-Fiber BLSR B A D C

37. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 37 Bi-directional Line Switched Ring (BLSR/2) AÔCAÔCAÔCAÔC C ÔA Ring Switch A B C D AÔCAÔCAÔCAÔC C ÔA Ring Switch Working Protection 2-Fiber BLSR

38. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 38 Bi-directional Line Switched Ring (BLSR/2) AÔCAÔCAÔCAÔC C ÔA Ring Switch A D AÔCAÔCAÔCAÔC C ÔA Working Protection Ring Switch Node Failure B C 2-Fiber BLSR

39. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 39 Node Failures => “Squelching” AÔCAÔCAÔCAÔC C ÔA Ring Switch A D AÔCAÔCAÔCAÔC C ÔA Ring Switch Node Failure B C 2-Fiber BLSR Customer 1 Customer 2

40. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 40 Bi-directional Line Switched Ring (BLSR/4) AÔCAÔCAÔCAÔC C ÔA AÔCAÔCAÔCAÔC 4-Fiber BLSR C A D Working Protection B

41. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 41 Bidirectional Line Switched Ring AÔCAÔCAÔCAÔC C ÔA AÔCAÔCAÔCAÔC 4-Fiber BLSR C A D Working B Span Switch Protection

42. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 42 Bidirectional Line Switched Ring AÔCAÔCAÔCAÔC C ÔA AÔCAÔCAÔCAÔC 4-Fiber BLSR C A D Working B Protection Ring Switch Node Failure Also Need to Squelch any Misconnected Traffic

43. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 43 BLSR: Discussion q BLSR/2 can be thought of as BLSR/4 with protection fibers embedded in the same fiber q I.e. ½ the capacity is used for protection purposes in each fiber q Span switching and ring switching is possible only in BLSR, not in UPSR q 1:n and m:n capabilities possible in BLSR q More efficient in protecting distributed traffic patterns due to the sharing idea q Ring management more complex in BLSR/4 q K1/K2 bytes of SONET overhead is used to accomplish this

44. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 44 Mesh Restoration DCS Central Controller DCS DC DC = Distributed Controller Reconfigurable (or Rerouting) Restoration Architecture Self Healing Restoration Architecture

45. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 45 Mesh Restoration DCS Line or Link Restoration Working Path Path Restoration Control: Centralized or Distributed Route Calculation: Preplanned or Dynamic Type of Alternate Routing: Line or Path

46. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 46 Mesh Restoration vs Ring/Linear Protection Extracted from: T-H. Wu, Emerging Technologies for Fiber Network Survivability, See References

47. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 47 Fast Reroute q Do the “restoration” at the MPLS (I.e. Layer 2) … q Also possible to do fast-reroute at layer 3 (IP) with BANANAS framework. q Issues: q Can MPLS re-route as fast as SONET (50ms)? q Can traditional IP re-route as fast as MPLS?

48. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 48 Fast Reroute (2) q First question: how fast is fast? q Do you really need 50 ms failover? q Second question: can you reroute really quickly while maintaining network stability? q Third question: what are the scalability issues with fast reroute?

49. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 49 Fast Reroute: MPLS vs. IP A B C 100010 IP routing to B pkt to B MPLS detour to B

50. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 50 Fast Reroute vs IP Routing IP q All nodes must be told of failure q Fast propagation, fast SPF trigger: how stable? q One step to full re- convergence MPLS (RSVP-TE) q Only the two ends of the link need be told (no signaling) q Local operation: explicit routing; more stable q Two step process: detour + converge