Global R&E Network Interconnects: AMPATH Update Ernesto Rubi, Sr. Network Engineer AMPATH International Exchange Florida International University Miami, FL Quilt Peering Workshop St. Louis, Missouri October , 2006

Quilt Peering Workshop, October 2006 Outline Network OverviewNetwork Overview Layer 1/2 Peering - UpdateLayer 1/2 Peering - Update GLIF/GOLE Activities/Infrastructure:GLIF/GOLE Activities/Infrastructure: The WHREN-LILA projectThe WHREN-LILA project AtlanticWaveAtlanticWave Open Exchange Points in the Western HemisphereOpen Exchange Points in the Western Hemisphere 2

Quilt Peering Workshop, October 2006 AMPATH International Exchange Point Characteristics l Current Technologies:  Various 10 GigE circuits currently in place to other networks include:  10 GigE to FLR ( Florida’s RON )  10 GigE to UltraLight’s  10 GigE to AtlanticWave ( Week ending Oct. 13th )  10 GigE to SC 06 SCInet - Delivering A-Wave to SC.  NRENs peer directly with AMPATH via international circuits.  NRENs peer with Abilene via multi-hop eBGP; however this will change with A-Wave implementation.  Direct peering to other NRENs available via AtlanticWave, WHREN-LILA circuit, FLR or UltraLight. l Located at Miami’s Tier-1 NAP: NAP of the Americas

Quilt Peering Workshop, October 2006 AMPATH International Exchange Point AMPATH 7609 Lucent CBX 500 Cisco ONS STM-16 Port Currently configured as VC4-8c with 1.22 Gbps effective bandwidth. MIAMI Foundry FastIron RNP ANSP CLARA Abilene A-Wave NLR UltraLight WHREN-LILA Cisco ONS SONET Gigabit Ethernet 10 Gigabit Ethernet EtherChannel All peer VLANs A -Wave To SC 2006 SCINet/Show Floor

WHREN-LILA IRNC Award l 5-year NSF Cooperative Agreement  Florida International University (IRNC awardee)  Corporation for Education Network Initiatives in California (CENIC)  Project support from the Academic Network of Sao Paulo ( ANSP )  CLARA, Latin America  CUDI, Mexico  RNP, Brazil  REUNA, Chile l Links Interconnecting Latin America (LILA) aims to Improve connectivity in the Americas through the establishment of new inter-regional links l Western-Hemisphere Research and Education Networks (WHREN) serves as a coordinating body whose aim is to leverage participants’ network resources to foster collaborative research and advance education throughout the Western Hemisphere 5

Links Interconnecting Latin America l Miami - Sao Paulo link: 1.2Gbps by year end, evolving to 2.5Gbps l Connects State of Sao Paulo academic network (ANSP) and Exchange Point, regional network (CLARA), Brazilian NREN (RNP), other international networks l San Diego - Tijuana link: operating at 2 x 1 Gbps, providing dedicated GigE links to regional network (CLARA) and Mexican NREN (CUDI) l East and west coast connectivity to I2 Abilene and other US and global R&E networks

Current RedCLARA topology l backbone ring: 155 Mbps (Global Crossing + Cisco) l Access links of 10 to 45 Mbps l Connection to Europe (GÉANT) at 622 Mbps from Brazil

WHREN-LILA Miami-Sao Paulo Distributed Exchange Point Sao Paulo Cisco RedCLARA router RNP switch ANSP switch RNP switch RNP router WHREN-LILA Red CLARA Miami Cisco Redundant dark fiber USP HEPGrid UERJ SPRACE UNESP BARUERICOTIA SDH/SONET N X GbE Sao Paulo has become a distributed exchange point, interconnecting ANSP, CLARA, RNP, supporting a distributed CMS Tier-2 facility, with inter-regional connections to U.S. R&E networks and E.U. GEANT

Quilt Peering Workshop, October 2006 Sao Paulo Exchange Point Characteristics l Built using Cisco ONS SDH Chassis  STM circuit between Miami and Sao Paulo is configured currently as VC4-8c ( ~ 1.22 Gbps ) linear circuit  Drawback: Static Provisioning, tear-down required if peers require additional bandwidth  VCAT and sw-LCAS can be used to accommodate network dynamics ( SC ‘06 )  STS-v ( VCAT ) circuits along with sw-LCAS allows network reconfiguration without service interruption and provides greater circuit granularity l Ethernet L2 End to End Service currently provided using ML cards installed on ONS  Currently 2 ML cards provide a total of 4 GigE ports ( can add more if needed )  GigE ports can be configured with various 802.1Q VLANs as well as transport user-defined VLANs using QinQ mapping  Bridge groups provide the necessary mappings via internal ONS cross-connect fabric to previously provisioned SDH circuits  CoS/QoS and other policing methods can be applied to ML ports to conform to requirements as well as QinQ VLAN mappings

Quilt Peering Workshop, October 2006 RNP Juniper M320 RedCLARA RNP COTIA USP RNP Catalyst 3550 ANSP 1 x GigaEth SDH/SONET 2 x GigaEth Sao Paulo / Miami - Via 2 x STM-4 / VC4-8c ( Current as of 8/14/06 ) ML Slot 5 / Port 0 AMPATH 7609 SDH Circuit: VC4-8C 1.2 Gbps WHREN-LILA ONS CLARA GSR RNP 3550 ML Slot 5 / Port 0 VLAN 2008 VLAN 2007 VLANs VLAN 2006 ANSP Foundry

Quilt Peering Workshop, October 2006 Future Implementation Goals 11 l Flexibility for end user requirements  ONS ML GigE ports assigned to individual end users to accommodate end2end flows ( LHC experiments, network research )  To accommodate 2 x 1 Gbps flows and an additional 300 Mbps flow:  Circuit provisioned using 2 x STS-12c-2v (1 Gbps) VCAT groups, an additional STS-3c-2v ( 300 Mbps ) VCAT group and/or multiple STS-1c-2v circuits over the STM-16 circuit.  sw-LCAS can be used to accommodate network dynamics. This means bandwidth can be re-provisioned while not affecting data flows.  Achieves: Granularity to 51 Mbps ( I.e: efficient use of bandwidth ) l Traffic Engineering  Peers are can be guaranteed bandwidth at port speeds or given CoS via policing which ensures better than best effort availability for sensitive data flows  VLAN mappings can be preserved throughout; a combination of Ethernet Wire Services or Ethernet Relay Service can be implemented using ML card to ensure flow separation and data security/integrity

Quilt Peering Workshop, October 2006 Sao Paulo / Miami - Via VC4-8c or Full STM-16 ( Near Future ) RNP Juniper M320 RedCLARA RNP COTIA USP RNP Catalyst 3550 ANSP 1 x GigaEth SDH/SONET 2 x GigaEth Slot 5, Port 0 Slot 5, Port 1 Slot 6, Port 0 To AMPATH 7609 Using EtherChannel? VLAN 2008 VLAN 2007 VLAN 2006 Policing/CoS Applied at Input of ML Card in SP STS-v-x VCAT Circuit using sw- LCAS

Quilt Peering Workshop, October 2006 Thank You! Questions/CommentsQuestions/Comments Ernesto Rubi AMPATH Engineering Group Visit us at: