1. National LambdaRail Layer 2 and 3 Networks 3 May 2005 Internet2 Spring Members Meeting Layers 2 & 3 Users BOF

2. Joint Techs - Feb. 13, 20052 NLR L2 Services Summary Goals Provide circuit-like options for users who can’t use, or don’t need, a 10G Layer1 wave. Experiment on future national L2 network capabilities. –Create distributed nationwide broadcast domain. –Create tools and procedures for automated and user-controlled provisioning of L2 resources. –Create network management tools for time-of-day based resource reservation.

3. Joint Techs - Feb. 13, 20053 NLR Layer 2 locations (Sites in blue are already connected) Jacksonville: Level3, 814 Phillips Hwy Atlanta: Level3, 345 Courtyard, Suite 9 Raleigh: Level3, 5301 Departure Drive WashDC: Level3, 1755/1757 Old Meadow Road Suite: 111, McLean VA. NYC: MANLAN, 32 Avenue of the Americas, 24th Floor Pittsburgh: Level3, 143 South 25th Cleveland: Level3, 4000 Chester Avenue. Chicago: Level3, 111 N. Canal, Suite 200 Kansas City: Level3, 1100 Walnut Street, MO. Denver: Level3, 1850 Pearl St, Suite 4 Seattle: PacWave, 1000 Denny Way (Westin) Sunnyvale: Level3, 1360 Kifer Road Suite: 251 Los Angeles: Equinix, 818 W. 7th Street, 6th Floor El Paso: Wiltel, 501 W. Overland Houston: Wiltel, 1124 Hardy St. Tulsa: Wiltel, 18 W. Archer Baton Rouge: Wiltel, No Address Yet Albuquerque: Level3, 104 Gold St.

4. Joint Techs - Feb. 13, 20054 HOU TUL ELP KAN Layer 2 Initial Logical Topology BAT LAX ALB PIT WDC CLE ATL RAL CHI NYC DEN SVL SEA JAC 10GE wave Cisco 6509 switch

5. Joint Techs - Feb. 13, 20055 NLR L2 Hardware Cisco Catalyst 6509-NEBs-A Chassis Height: 36.65 in. 93.09cm Chassis Width: 17.20 in. 43.68cm Chassis Depth: 20.32 in. 51.61cm Weight: 270 lb fully loaded DC Power: 2 -2500 WDC power supplies We will need 1 60A feed for each

6. Joint Techs - Feb. 13, 20056 Generic NLR L1, L2 and L3 PoP Layout CRS-1 Colo EastWest NLR demarc 15808 6509 15454 DWDM 1G wave, link or port 10G wave, link or port

7. Joint Techs - Feb. 13, 20057 NLR L2 Services Summary Initial Services Dedicated Point to Point Ethernet – VLAN between 2 members with dedicated bandwidth from sub 1G to multiple 1G. Best Effort Point to Multipoint – Multipoint VLAN with no dedicated bandwidth. National Peering Fabric – Create a national distributed exchange point, with a single broadcast domain for all members. Initial user ports will be 1GE.

8. Joint Techs - Feb. 13, 20058 NLR L2 Services Summary Expected Near Term Services Dedicated Point to Multipoint : Dedicated bandwidth for multipoint connections Scavenger: Support less-than-best-effort forwarding. This would be useable for all connections. Connections: Support 10GE user ports

9. Joint Techs - Feb. 13, 20059 NLR L2 Services Summary Possible Long-range Services User-controlled Web-based provisioning and configuration Allow users to automatically create new services, or reconfigure existing services on the network using a web-based tool. Time-sensitive provisioning – Allow users to have L2 connections with bandwidth dedicated only at certain times of day, or certain days.

10. Joint Techs - Feb. 13, 200510 Layer 3

11. Joint Techs - Feb. 13, 200511 NLR L3 Services Summary Base Features, Day 1 BGP to each member, with their 2 connections configured any way they’d like. Options include but aren’t limited to: primary/backup, route-based load balanced, or 1 for traffic/1 for measurement. ISIS core IGP IPv4 Multicast: PIM, MSDP, MBGP by default. IPv6: BGP peering by default Security: Each member will be able to black-hole traffic to one of their hosts using special BGP community

12. Joint Techs - Feb. 13, 200512 NLR L3 Services Summary Features Document Day One Connection Each member gets a 10GE connection and a VLAN backhauled over the L2 network to a second node. General operations of the network, including base features (configuration with no experiment running), connections, and communication of experiments. Future Possibilities Peering with other R&E networks. Commodity Internet Connections or peering.

13. Joint Techs - Feb. 13, 200513 NLR Layer 3 locatons Atlanta: Level3, 345 Courtyard, Suite 9 WashDC: Level3, 1755/7 Old Meadow Rd Suite: 111,McLean VA. NYC: MANLAN, 32 Avenue of the Americas, 24th floor Chicago: Level3, 111 N. Canal, Suite 200 Denver: Level3, 1850 Pearl St, Suite 4 Seattle: PacWave, 1000 Denny Way Los Angeles: Equinix, 818 W. 7th Street, 6th Floor Houston: Wiltel, 1124 Hardy St.

14. Joint Techs - Feb. 13, 200514 UNM LA Duke/NC FLR OK PSC HOU Layer 3 Initial Logical Topology showing planned member connections LAX WDC ATL CHI NYC DEN SEA Cisco CRS-1 router 10GE wave CICITN PNWGP UCAR/FRGP LEARN Cornell CENIC GATech MAT P NLR L2 NLR L2t NLR L2

15. Joint Techs - Feb. 13, 200515 NLR L3 Hardware Cisco CRS-1 Half-rack Chassis Height: 38.5 in. 99.06cm, (with base cosmetics) Chassis Width: 17.5 in. 44.45 cm Chassis Depth: 36.6 in 93.0 cm; (40.5 in. 100.844 cm, including full cosmetics) Weight: 330.8 lb (138 kg) chassis with fans, PDU, and blanks (as shipped) 650 lb (294.8 kg) chassis as shipped, including power shelves, all line cards, and route processors DC Power: 7900 WDC Needs 3 65A feeds per power supply Clearance: 3 feet front and back required. Four-post mounting is strongly preferred.

16. Joint Techs - Feb. 13, 200516 CRS1 base configuration CRS-1 (half-rack) 2 route processors (RPs) 4 switch fabric cards 2 Power Entry Modules 2 control plane software bundle licenses (IOS-XR) with crypto 2 memory modules for each RP (required) – 2GB each 1 8x10GE line card 1 multi-service card (MSC) 1 8x10GE PLIM 1 line card software license 1 extra MSC 1 extra line card software license 8 XENPAK 10G-LR optics fan trays and air filters

17. Joint Techs - Feb. 13, 200517 CRS1 base configuration Sites that had at least 7 of their 8 10GE interfaces assigned at initial installation receive a second 8x10GE, including the MSC and software license. Chicago Denver Houston These location have a total of 12 XENPAK 10G-LR optics modules. We call the first configuration “A”, and the configuration with the additional 8x10GE type “B”. The NLR layer 3 network will be comprised initially of 5 type “A” routers and 3 type “B” routers.

18. Joint Techs - Feb. 13, 200518 Layer 3 Engineering sub- committee recommendations Add logical router capability to select sites Logical routers are a collection of line cards and Route Processors (RP) which together form a distinct router within the chassis. In order to provide additional logical routers for hardware- level separation of services, additional line cards and RP resources must be installed. Logical routers may allow multiple potentially destabilizing experiments to be run simultaneously on the NLR network with the least Add 1GE availability to select sites This provides a lower speed and less expensive class of interface to which connectors can attach allows for temporary connections, direct monitoring and measurement connections, and direct attachment of reasonably priced experimental tools such as traffic generators and protocol testers. Add additional 10GE to select sites There are three main reasons for recommending this addition. First, with the base configuration, all but one site is already oversubscribing the slot from day 1. Also, some sites are already completely full and have no room for expansion, even with oversubscription. Although oversubscription may be fine for many experimental scenarios, it would be better to have the option of oversubscription or not, depending on the project at hand. Because the costs are so heavily loaded into the line cards, it is probably unrealistic to expect the “next” connector to absorb the costs of moving to a second 8x10GE card, and it will be a while before 10GE is supported in SPA format. Third, similar to the 1GE line cards recommended above, it provides a more interesting architecture with greater fanout and provides another line card which can be used in the case of logical routers

19. Joint Techs - Feb. 13, 200519 NLR L3 Services Summary Private Test-lab Network Connections. Route Advertiser Connections: Get active commodity routing table for experiments, but no actual commodity bandwidth drainage. Pre-emptible Connections: Allow other types of connections to use unused ports on a temporary basis, such as for a conference or measurement project.

20. Joint Techs - Feb. 13, 200520 NLR L3 Services Summary Future Options MPLS (Traffic Engineering, L2/L3 VPNs): This can be done on an experimental basis Day 1. May be rolled into Base Feature list if there is general interest. IPv6 Multicast: Will be implemented as support and standards mature. User-Controllable Resource Allocation: Will be supported as experiments, and rolled into the base feature list if there is general use and interest.

21. Joint Techs - Feb. 13, 200521 NLR L3 Services Summary Service Expectations Document (Exhibit F, Attachment C) Set user expectations for service on L3 network Make clear the experiment support model Service Expectations, SLA isn’t good measure since the network may appear “down” because of experiments. Network may not have same uptime as production network, but will have same level of service and support as production network

22. Joint Techs - Feb. 13, 200522 NLR L3 Services Summary Experiment Support Each experiment will have a representative from the L2/L3 Support Center and a representative from the ESC. If necessary, the prospective experiment will be sent to the NNRC for review. L2/L3 Services staff will be responsible for scheduling network assets for experiments and will see the experiment through to completion. In general, experiments will be scheduled on a first- come-first-served basis.

23. Joint Techs - Feb. 13, 200523 NLR L3 Services Summary Network Conditions Way of communicating the current state of the network to users. Users may choose to have their interfaces automatically shutdown under any Network Conditions they desire. Users will receive notification of changes to Network Condition, with focused communication to those who will be turned on or off because of it. Tools will be available for users to monitor and track Network Conditions.

24. Joint Techs - Feb. 13, 200524 Network Conditions NetCon 7- No Experiment Currently Active NetCon 6- Experiment Active, No Instability Expected NetCon 5- Possible Feature Instability/No General Instability Expected NetCon 4- Possible Network Instability NetCon 3- Congestion Expected NetCon 2- Probable Network Instability/Possible Impact on Connecting Networks NetCon 1- Network Completely Dedicated

25. Joint Techs - Feb. 13, 200525 NLR Layer 2/3 Service Center projects Out of band monitoring/access network Database development General support applications Specific support applications Experimentation support Monitoring and measurement support

26. Joint Techs - Feb. 13, 200526

27. Joint Techs - Feb. 13, 200527

28. Joint Techs - Feb. 13, 200528

29. Joint Techs - Feb. 13, 200529

30. Joint Techs - Feb. 13, 200530 NLR Layer 2/3 - discussion What do users want/need? Tools? User groups? Monitoring and measurement ability? Full routes? Control over route propagation? Control over protocols and timers? Direct access to login and configure routers? A router “ghost” service? Instruction/workshops? Commodity access or ISP collaboration? Collaboration with projects like PlanetLab and WAIL?

31. Joint Techs - Feb. 13, 200531 NLR User Resources http://noc.nlr.net info@nlr.net research@nlr.net esc@nlr.net