1. NLR Layer2/Layer3 Users BOF NLR status update Philadelphia Internet2 Member Meeting 19 September 2005 Brent Sweeny, Indiana University

2. 9/19/2005NLR Layer2/Layer3 users BOF2 National LambdaRail design NLR WaveNet PoP NLR WaveNet & FrameNet PoP NLR WaveNet, FrameNet & PacketNet PoP NLR owned fiber

3. 9/19/2005NLR Layer2/Layer3 users BOF3 Generic NLR L1, L2 and L3 PoP Layout CRS-1 Colo EastWest NLR demarc 15808 6509 15454 DWDM 1G wave, link or port 10G wave, link or port

4. 9/19/2005NLR Layer2/Layer3 users BOF4 DAL SYR TUL PEN ELP PHO BAT ALB HOU WDC OGD CLE NYC SAA 4 4 JAC 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 44 4 4 SLC 4 4 Level3 fiber WilTel fiber 4 4 RAT 4 Cisco 15808 terminal Cisco 15808 OADM Cisco 15454 terminal Cisco 15454 OADM 8 4 4 8 STA CHI KAN PIT BOI CLE ATL POR RAL DEN SVL SEA 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 4 4 LAX 8 8 NLR Layer 1

5. 9/19/2005NLR Layer2/Layer3 users BOF5 Layer 1 Phase 2 Deployment DAL SYR TUL PEN ELP KAN PHO BAT ALB HOU WDC OGD CLE NYC SAA DEN 4 4 JAC 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 44 4 4 SLC 4 4 Level3 fiber WilTel fiber LAX 4 4 RAT 4 Cisco 15808 terminal Cisco 15808 OADM Cisco 15454 terminal Cisco 15454 OADM 8 4 4 8

6. 9/19/2005NLR Layer2/Layer3 users BOF6 Layer 1 baseline Opportunity to connect into lambda fabric Opportunity to connect into lambda fabric Point to point Point to point Other endpoint could be anywhere Other endpoint could be anywhere Early examples: Early examples: HOPI HOPI Ultralight Ultralight iGRID iGRID SC05 (Supercomputing) SC05 (Supercomputing) Experiments support center (for all layers) Experiments support center (for all layers)

7. 9/19/2005NLR Layer2/Layer3 users BOF7 Layer 2 Network Design HOU TUL ELP KAN BAT LAX ALB PIT WDC CLE ATL RAL CHI NYC DEN SVL SEA JAC 10GE wave 10GE managed wave Yellow sites are done Cisco 6509 switch SVL

8. 9/19/2005NLR Layer2/Layer3 users BOF8 Layer 2 installation status Switch installations completed: SunnyvaleDenver Kansas CityChicago ClevelandPittsburgRaleighWashington DC JacksonvilleAtlantaLos AngelesTulsa El PasoHouston New York City scheduled for this week (Sept 20-21). Baton Rouge waiting for Katrina side-effects to calm down. (Was scheduled for the week of August 29, now tentatively Oct 11-12.) Albuquerque scheduled for mid-Oct. Interconnections being completed now.

9. 9/19/2005NLR Layer2/Layer3 users BOF9 Layer 2 service baseline 1GE Connection into local 6500 1GE Connection into local 6500 Access to “national peering VLAN” Access to “national peering VLAN” Additional Options: Additional Options: Dedicated point to point Etherrnet, Nx1GE Dedicated point to point Etherrnet, Nx1GE Best-effort point to multipoint (no dedicated bw) Best-effort point to multipoint (no dedicated bw) Soon: Soon: 10GE ports 10GE ports Dedicated point-to-multipoint Dedicated point-to-multipoint

10. 9/19/2005NLR Layer2/Layer3 users BOF10 Goal Provide circuit-like options for users who can’t use, can’t afford, or don’t need, a 10G Layer1 wave via point-to-point layer2 VLANs. Provide circuit-like options for users who can’t use, can’t afford, or don’t need, a 10G Layer1 wave via point-to-point layer2 VLANs. Experiment with large-scale layer2 capabilities. Experiment with large-scale layer2 capabilities.

11. 9/19/2005NLR Layer2/Layer3 users BOF11 Dimensions of NLR Layer2 capabilities point-to-point or multipoint point-to-point or multipoint public or private public or private bandwidth guaranteed or best effort bandwidth guaranteed or best effort resilient (spanning tree) or nailed up resilient (spanning tree) or nailed up temporary or permanent temporary or permanent experimental or stable/production- oriented experimental or stable/production- oriented Some of the above dimensions are either/or, some are more spectrum-like. The following slides show some expected uses of NLR layer 2, and how these dimensions may relate to each type of project.

12. 9/19/2005NLR Layer2/Layer3 users BOF12 National Peering Fabric National Peering Fabric Multipoint, public, best-effort, resilient, permanent, stable Multipoint, public, best-effort, resilient, permanent, stable NLR allocated addresses, peer with any other member across the layer 2 fabric policy-free. NLR allocated addresses, peer with any other member across the layer 2 fabric policy-free. Possible to have more than one (say by MTU). Possible to have more than one (say by MTU). Ready to go today. Ready to go today. Backup connections to networks such as Abilene, commodity providers, etc. Backup connections to networks such as Abilene, commodity providers, etc. Point-to-point, private, permanent, stable Point-to-point, private, permanent, stable Could be best-effort or guaranteed. Could be best-effort or guaranteed. Could be nailed-up or resilient. Could be nailed-up or resilient. Load-balance or leave idle until needed. Load-balance or leave idle until needed. Ready to go today, though we only have pricing for the guaranteed nailed-up case. Ready to go today, though we only have pricing for the guaranteed nailed-up case.

13. 9/19/2005NLR Layer2/Layer3 users BOF13 To enable a flexible topology for NLR layer3 To enable a flexible topology for NLR layer3 Best-effort, private, temporary, experimental Best-effort, private, temporary, experimental We have 8 layer3 nodes, but the topology between them can be made much more interesting by creating various connections over the layer2 network. We have 8 layer3 nodes, but the topology between them can be made much more interesting by creating various connections over the layer2 network. Enables layer3 experimentation. Enables layer3 experimentation. To provide members with a second path into the NLR layer3 network To provide members with a second path into the NLR layer3 network Point-to-point, private, permanent, experimental Point-to-point, private, permanent, experimental Connect to a second node on the layer3 backbone. Connect to a second node on the layer3 backbone. Load-balance or leave idle until needed. Load-balance or leave idle until needed. Included in membership. Included in membership.

14. 9/19/2005NLR Layer2/Layer3 users BOF14 Temporary connections for special projects Temporary connections for special projects Guaranteed, private, temporary, stable Guaranteed, private, temporary, stable For remote instrumentation, where the member only has the remote resource reserved for a limited window. For remote instrumentation, where the member only has the remote resource reserved for a limited window. For conferences, demos, and other special events. For conferences, demos, and other special events. Provides a low latency/jitter path if needed. Provides a low latency/jitter path if needed. Nailed-up if latency is critical, probably resilient if not. Nailed-up if latency is critical, probably resilient if not. Could be point-to-point or multipoint. Could be point-to-point or multipoint. Technically, this is possible today, but we have no pricing model for it. Technically, this is possible today, but we have no pricing model for it.

15. 9/19/2005NLR Layer2/Layer3 users BOF15 Bootstrapping circuit-like research Bootstrapping circuit-like research Point-to-point, private, guaranteed, temporary, nailed-up, experimental Point-to-point, private, guaranteed, temporary, nailed-up, experimental To enable a researcher to get started while waiting for funding or provisioning of a layer1 circuit. To enable a researcher to get started while waiting for funding or provisioning of a layer1 circuit. Similar to a special event, but more experimental, a probably a stronger need for it to be nailed-up. Similar to a special event, but more experimental, a probably a stronger need for it to be nailed-up. Technically, this is possible today, but we have no pricing model for it. Technically, this is possible today, but we have no pricing model for it. Provide control plane network for optical experiments Provide control plane network for optical experiments Permanent, resilient, experimental Permanent, resilient, experimental A topology could be created for the oob management network needed for some dynamic optical networking experiments (GMPLS, etc.) A topology could be created for the oob management network needed for some dynamic optical networking experiments (GMPLS, etc.)

16. 9/19/2005NLR Layer2/Layer3 users BOF16 Cluster/Grid LAN Cluster/Grid LAN Multipoint, private, guaranteed, experimental Multipoint, private, guaranteed, experimental To enable remote clusters to appear on the same LAN. To enable remote clusters to appear on the same LAN. It is not known if spanning tree would be wanted. It is not known if spanning tree would be wanted. It could evolve into a more production-like requirement. It could evolve into a more production-like requirement. Technically, this is possible today, but we have no pricing model for it. Technically, this is possible today, but we have no pricing model for it. Experiment directly with Layer 2 Experiment directly with Layer 2 Could be of any type (experimental, obviously) Could be of any type (experimental, obviously) Web-based provisioning, direct user requests, etc. Web-based provisioning, direct user requests, etc. Concern about interaction with more production-like requirements. Concern about interaction with more production-like requirements.

17. 9/19/2005NLR Layer2/Layer3 users BOF17 ALB PHO BAT RAL JAC TUL PIT HOU LAX WDC ATL CHI NYC DEN SEA Cisco CRS-1 router 10GE wave Yellow sites are installed Layer 3 Network LAX

18. 9/19/2005NLR Layer2/Layer3 users BOF18 Layer 3 installation status With the work this week in NYC, all layer3 router installations are complete. Waves are being built between routers and for layer3 backhaul now.

19. 9/19/2005NLR Layer2/Layer3 users BOF19 Layer3 service baseline Each member gets two routed connections Each member gets two routed connections “local” 10GE “local” 10GE VLAN backhauled to 2 nd node VLAN backhauled to 2 nd node BGP peering with NLR L3 network BGP peering with NLR L3 network IPv4 unicast IPv4 unicast IPv4 multicast (MBGP/PIM/MSDP) IPv4 multicast (MBGP/PIM/MSDP) IPv6 unicast (multicast later) IPv6 unicast (multicast later) An ‘experimental’ (changeable, changing) L3 network An ‘experimental’ (changeable, changing) L3 network

20. 9/19/2005NLR Layer2/Layer3 users BOF20 Layer 3 coming services Likely eventual logical routers Likely eventual logical routers More 1GE options More 1GE options More 10G options More 10G options Pre-emptable connections Pre-emptable connections MPLS MPLS More user control—scheduling, testing, etc More user control—scheduling, testing, etc User access to measurement data User access to measurement data

21. 9/19/2005NLR Layer2/Layer3 users BOF21 NLR Engineering/Support Organization A very distributed, coordinated organization: Service desk at Indiana Service desk at Indiana Layer1 NOC and engineering at CENIC Layer1 NOC and engineering at CENIC Layer2/3 NOC and engineering at Indiana Layer2/3 NOC and engineering at Indiana Experiments support center at North Carolina Experiments support center at North Carolina

22. 9/19/2005NLR Layer2/Layer3 users BOF22 NLR User Resources http://noc.nlr.net http://noc.nlr.net http://noc.nlr.net info@nlr.net info@nlr.net info@nlr.net research@nlr.net research@nlr.net research@nlr.net esc@nlr.net esc@nlr.net esc@nlr.net

23. 9/19/2005NLR Layer2/Layer3 users BOF23 NLR Layer 2/3 - discussion What do users want/need? Tools? User groups? Instruction/workshops? Monitoring and measurement ability? Specify? Full routes? Control over route propagation? Control over protocols and timers? What would you do with logical routers? Direct access to login and configure routers? Commodity access or ISP collaboration? Collaboration with projects like PlanetLab and WAIL? Which others?

24. 9/19/2005NLR Layer2/Layer3 users BOF24 Summary from July ‘05 (Joint Techs BOF) Costs Costs More deeply-technical descriptions of the architecture, service offerings More deeply-technical descriptions of the architecture, service offerings L2 and L3 service-design docs L2 and L3 service-design docs Standardized interface into resources Standardized interface into resources And some thoughts about possibilities: "matchmaking" service, across all layers? "matchmaking" service, across all layers? Logical routers: separate forwarding/control planes, possibly enabling customer logical networks (RONs) Logical routers: separate forwarding/control planes, possibly enabling customer logical networks (RONs) Customer edge control Customer edge control Native non-IP, non-Ethernet protocols? Native non-IP, non-Ethernet protocols? Commodity internet peering? Commodity internet peering? What’s NLR trying to be? (“dancing-elephants or dancing- gerbils”, “facile/flexible”) What’s NLR trying to be? (“dancing-elephants or dancing- gerbils”, “facile/flexible”)