1. >> BellSouth.net > Engineering & Architecture FloridaMIX Next Generation Exchange February 2001

2. >> BellSouth.net > Engineering & Architecture February 20012 Agenda Introduction/Background Operations “Classic” Options Optical/TDM Transport Core ATM Transport Evolution Facilities/Survivability Participants

3. >> BellSouth.net > Engineering & Architecture February 20013 Design Team BellSouth.net Susan Campbell Mike Duckett, Sr. Director, Eng. & Architecture Robbie Harrell Christian Kuhtz Troy Meuninck Deron Ringen Tim Seaver (+ various clueful vendor people & peering geeks)

4. >> BellSouth.net > Engineering & Architecture February 20014 Company Background $27.6 Billion Total Revenue; $3.54 Billion Data Services Revenues (2000) 956,000 Internet Customers 3.4 Million Fiber Route Miles Deployed Customers: –34.7 Million US –6.9 Million Latin America

5. >> BellSouth.net > Engineering & Architecture February 20015 What is it? FloridaMIX is an exchange facility providing infrastructure in the greater Miami area to connect organizations with each other, regardless of their geography. Minimum DS-3 port speed, up to OC-48 (at launch; OC-192 mid 2001). Ethernet (100base, GE) introduced 3Q 2001. SONET & SDH, conversion late 2001. built from the ground up to be resilient, highly available, reliable, diverse,.. Hosted by BellSouth.

6. >> BellSouth.net > Engineering & Architecture February 20016 Why? Infrastructure to support multimedia content distribution will be critical to economic development of the tri-county area, Florida and the Southeast Improved access to International Markets Enable economic private peering infrastructure for large ISPs Provide smaller ISPs and private businesses local access to the Internet backbones

7. >> BellSouth.net > Engineering & Architecture February 20017 Design Criteria Guaranteed Peering Quality Flexible, rapidly provisionable, any to any connectivity. Independence –Location –Carrier High degree of survivability

8. >> BellSouth.net > Engineering & Architecture February 20018 R&D facilities BellSouth.net IP Engineering & Architecture Lab @ Perimeter (Atlanta, GA) –Dedicated, substantial replica of components (optical transport core, TDM/IP/MPLS fabrics, provisioning engines) for the FloridaMIX. To be used for Design, Testing, Regression Testing, Troubleshooting, Design Evolution. –Immediate access to 40+ IP networking & systems engineering people. Additional testing efforts at various vendor labs around the nation.

9. >> BellSouth.net > Engineering & Architecture February 20019 Connection Models Two general models are available: –Colocate at FloridaMIX Data Center –Attach remotely to one or more FlordiaMIX POPs.

10. >> BellSouth.net > Engineering & Architecture February 200110 Core Layout Co-location Data Center POP OC-48 DWDM fabric

11. >> BellSouth.net > Engineering & Architecture February 200111 Coverage

12. >> BellSouth.net > Engineering & Architecture February 200112 Example Applications Traditional Public & Private Peering Content Distribution Injection Point Backup Data Centers for South America. Generic Broadband, Rapid Provisioning Infrastructure (Bandwidth on Demand) Bandwidth Brokerage …

13. >> BellSouth.net > Engineering & Architecture February 200113 Operations BellSouth will operate and manage the FlordiaMIX backbone and equipment through dedicated organization. FloridaMIX NOC run by optical & IP engineers. –3 engineering seats (24x7) in Atlanta –3 engineering seats (24x7) in Miami Merit will manage the public peering RS (MOU in progress). Communication is a top priority.

14. >> BellSouth.net > Engineering & Architecture February 200114 Pipes Core and access fiber infrastructure for FlordiaMIX provided by –FPL Fibernet –BellSouth Telecommunications Any optical loop will accepted, as long as conduit space is available. Electrical interfaces on a case-by-case basis. Intent is to be connected to every cable landing site. Negotiations in progress.

15. >> BellSouth.net > Engineering & Architecture February 200115 Access Conduit access must be negotiated with landlord (not affiliated with BellSouth). As long as landlord provides conduit space, we will connect. Contractual guarantees for “any loop” will be available – intent: “no carrier jail”.

16. >> BellSouth.net > Engineering & Architecture February 200116 Transport Highlights Robust optical transport service offerings ranging from an unprotected service to full route redundancy with SONET/SDH like recovery and restoration (< 50ms per link including detection and restoration) Service connections support SONET and SDH framing Service connection capacities range from STS-1 to STS-48 in STS-1 increments Support for optical VPNs

17. >> BellSouth.net > Engineering & Architecture February 200117 Fiber Layout FPL FiberNet BST POP 1 dark fiber pair each

18. >> BellSouth.net > Engineering & Architecture February 200118 Fiber Layout 6 λ per inter-POP pair @ turn-up installed capacity for 24 λ per POP (equipment capable of 80 λ) no concern about fiber grade w/ OC-48, OC-192’s depend on actual fiber condition. Can mix and match OC-48 & OC-192 (no interruption once OC-192 is ready).

19. >> BellSouth.net > Engineering & Architecture February 200119 Optical POP layout Sycamore SN16000 Sycamore SN8000 Edge Core DWDM

20. >> BellSouth.net > Engineering & Architecture February 200120 Equipment Choices Why Sycamore? “If you believe software is the critical gate, you have to go with a software centric solution. Neccessary for on-demand enhancements, and SONET/SDH is crucial in an international environment.”

21. >> BellSouth.net > Engineering & Architecture February 200121 TDM Transport

22. >> BellSouth.net > Engineering & Architecture February 200122 ATM Transport

23. >> BellSouth.net > Engineering & Architecture February 200123 ATM Transport The ATM transport provides a fully redundant OC12 backbone. Public peering over ATM at DS3,OC3 and OC12 port speeds utilizing ATM UBR class of service. Private peering and transport over ATM at DS3, OC3 and OC12 port speeds utilizing ATM CBR, VBR and UBR class of service.

24. >> BellSouth.net > Engineering & Architecture February 200124 ATM Transport Minimum guaranteed bandwidth over the public ATM infrastructure. Rapid provisioning to meet increasing customer requirements. No overbooking or over-subscription allowed in order to maximize throughput of IP based traffic. CMN capabilities are in the works.

25. >> BellSouth.net > Engineering & Architecture February 200125 Architectural Requirements Any-to-Any connectivity Closed User Group Guaranteed Bandwidth Multiple Transmission Interfaces Types Intelligent Congestion Management Transparent transport of tagged frames Link Aggregation Jumbo Frame Support Loop Detection Multicast Traffic Manageability of Service SLA Support Unified Control Plane

26. >> BellSouth.net > Engineering & Architecture February 200126 Phase II

27. >> BellSouth.net > Engineering & Architecture February 200127 Phase II

28. >> BellSouth.net > Engineering & Architecture February 200128 Facilities 4 POPs –10,000 sq.ft each Colo Data Center in West Miami –70,000 sq.ft. raised floor, 100,000 sq.ft. total –Additionally Hosting Centers are obvious alternates (3 rd party)

29. >> BellSouth.net > Engineering & Architecture February 200129 Survivability no site is in category 1 evac zone (subject to storm surge) 3 POPs category 5, 1 POP category 3 Own generator, battery plant per site. 6 hrs minimum battery coverage. (Colo has UPS) “Miami has at times severe weather conditions, you have to have the resources and committment to fly in resouces (people, equipment, fuel) to guarantee high availability and resiliency.”

30. >> BellSouth.net > Engineering & Architecture February 200130 Build-Out Plan Three Phases: Phase 1: Phase 2: Phase 3: Private Peering via Optical, TDM, ATM Public Peering via ATM Private & Public Peering via Ethernet. Private & Public Peering via MPLS. Fully Automated Provisioning

31. >> BellSouth.net > Engineering & Architecture February 200131 Current Timeline 1 st Quarter 2001 –Deploy phase one (Target: very late CQ1) Optical Transport ATM private and public services 2 nd Quarter 2001 –Assess and define Ethernet and transport agnostic transport mechanisms.(MPLS, GMPLS, MPLS “Circuit Cross Connect”, RFC2547(bis) layer 3 VPN’s, etc)

32. >> BellSouth.net > Engineering & Architecture February 200132 Participants The following have signed LOI’s w/ publicity release statements: Qwest, Diveo, Intermedia, FPL Fibernet, UUNET (a WorldCom Company), Exodus, PSINet, Nova Southeastern University, Computer Office Solutions, Hyperbyte, Cyberlinyx, Advanced Data Center, Inc., Worldwide Internet Services, Synergyx, Worldwide Telnet and Florida Atlantic University

33. >> BellSouth.net > Engineering & Architecture February 200133 Thank You!

34. >> BellSouth.net > Engineering & Architecture February 200134

35. >> BellSouth.net > Engineering & Architecture February 200135 ATM Transport –DS3-OC12 support –Scalable non-blocking switch fabric –Support for automatic protection switching (APS) for redundancy and protection –Support of both SONET and SDH framing –Intelligent cell discard (visibility into IP frames) –Per VC queuing –Congestion control –Flow control –Ease of provisioning

36. >> BellSouth.net > Engineering & Architecture February 200136 ATM NAP Peer 1 Peer 2 Peer 3 IX ATM OC-3 Pipe DS-3 Pipe Transport Pipe ATM UBR VC

37. >> BellSouth.net > Engineering & Architecture February 200137 ATM NAP Peer 1 Peer 2 Peer 3 IX ATM 25 Mbps offered bandwidth Congestion Point

38. >> BellSouth.net > Engineering & Architecture February 200138 ATM NAP IP Packet Cell IP Chunk Cell IP Chunk Cell IP Chunk Cell IP Chunk … packet shredding occurs @ Peer 1 & Peer 2

39. >> BellSouth.net > Engineering & Architecture February 200139 ATM NAP IP Packet Cell IP Chunk Cell IP Chunk Cell IP Chunk Cell IP Chunk … congestion cell discard occurs @ ATM IX switch

40. >> BellSouth.net > Engineering & Architecture February 200140 ATM NAP IP Packet Cell IP Chunk Cell IP Chunk Cell IP Chunk … re-assembly fails @ Peer 3

41. >> BellSouth.net > Engineering & Architecture February 200141 ATM NAP Conclusion (in a nutshell): –Must have packet aware discard mechanism to guarantee decent goodput. (otherwise, for example, <5% cell loss can translate into 70+% AAL5’ed IP frame loss/corrupted frames). –Testing shows ABR w/ VSVD in the core allows for very nice IP traffic guarantees over ATM. –Long term migrate to a packet based infrastructure. => Strong focus on using Ethernet & MPLS w/ strong R&D effort.

42. >> BellSouth.net > Engineering & Architecture February 200142 Congestion Management Frame based congestion management –Primarily at the provider edge of the FlordiaMIX. –Pushed out to “customer” edge in the case of hosting centers/carrier hotels for multiple providers sharing a transport pipe. Exact configurations are presently under study

43. >> BellSouth.net > Engineering & Architecture February 200143 Ethernet CUG Ethernet switches at the FloridaMIX can join 802.1q VLANs to create closed user groups (CUG) for peering purposes. –Multicast –experimental peering services such as IPv6