Joint Techs, Minneapolis, MN Tuesday February 13, 2007 Understanding Non-layer 3 Networks Dan Magorian, MAX Director of Engineering and Operations

Joint Techs, Minneapolis, MN Tuesday February 13, 2007 So, most of us have built non-Layer 3 nets, but do we “understand” them? In this community, people mean a lot of different things by “hybrid” and “non layer 3”. Good talks on this yesterday. extended ethernets (lambdas into ethernet switches using vlans) Dwdm-lit fiber without ethernet Traditional carrier sonet/sdh circuits Infiniband and other net stacks that may or may not use IP There’s a lot of various kinds of lashups these days, as folks need to provision non-routed connections between regional nets, to get to fiber-poor locations, for SC demos, etc. MAX provides production lambda services to participants for backhaul and ld transport. Eg, we connect Abilene and ESnet to the NGIX/E peer point, NASA and ESnet to the Mclean VA Level3 pop, National Capitol Region net, etc.

Looks easy from a high level view Sao Paulo (SPB) LON WDC ATL MIA AMS STK CER SEA CHI AUS LAX TOK A-Wave A globally integrated set of “light path” facilities: waves, exchange points, etc (Note: map not complete…) Sonet/SDH Ethernet NYC Ethernet, initially HKO

Joint Techs, Minneapolis, MN Tuesday February 13, 2007 As we all migrate to towards “hybrid” futures, understanding what we’re doing is part of the problem Usually everyone only contributes a piece of the solution, and the people for whom the service is being delivered often don’t know all the L1 details. One characteristic of these lashups is that they have a lot of transport boxes. Right now people usually go OEO (optical- electrical-optical) between each net operators’ dwdm systems. This leads to expensive rack-wasting boxes handing off between providers primarily for the purpose of maintaining service demarcs, and regeneration of course. All–optical handoffs can be cheaper and take less space, but take a lot of engineering and coordination, and are often subject to too many power budget and other variables to be reliable for non-experimental connections.

Joint Techs, Minneapolis, MN Tuesday February 13, 2007 Rising expectations, rising complexity So do these lashups work? Yes. Can they have O&M and performance issues? Definitely. Are we likely to keep doing more and more of them? No question. Why? Sometimes they’re the only way we can get something done. Sometimes they’re just the cheapest way to get something done. But not everything that’s cheap is good idea. As campuses gear up to hand off lambdas from researchers to RONs who hand them off to I2, NLR, and other backbones, complexity level of the lashups keeps rising. At the same time, as 10G becomes a commodity, application expectations for “clean pipes” keeps rising. Now have apps with minimal error checking running on pipes with almost no error correction, across inter-RON lashups w/ no error reporting

Joint Techs, Minneapolis, MN Tuesday February 13, 2007 Now add to that mix dynamic resource allocation Lots of good work in the community inc MAX’s DRAGON on dynamic resource allocation (lightpaths) and how we’re going to get there from here, eg GLIF forum. Some people think this will simplify the way we’re doing it now, others sceptical that it increases complexity needlessly. Eg, soon we’ll want to be able to figure out the end-to-end state of multiprovider/multivendor control plane signalling, not just the integrity of the ethernet data plane like we do now. But clearly, dynamic resource mechanisms will increase our capabilities and enable things not easily possible now, like timesharing resources between large short-duration flows.

Joint Techs, Minneapolis, MN Tuesday February 13, 2007 And new technology does help out Eg, right now vlans are a often scarce resource that require lost of time-consuming coordination since not all ethernet switches can do vlan translation. But once people start to use PBT, VPLS and 802.1ad, and other technologies that include vlan translation, that particular problem goes away without the need to create global or even project vlan registries (which we had to do for AWave for SC06: manual administrative resource allocation). Lots of examples of complexities dying out (eg atm)

Joint Techs, Minneapolis, MN Tuesday February 13, 2007 So is this a doom-and gloom picture? Let’s see: we’re making the network much more complex, we don’t really have any more talented network engineers to go around, we mostly don’t seem to be finding big pots of money lying around to pay people better or hire lots more expertise, and the tools to troubleshoot this brave new world mostly don’t exist yet. Yup, I’d say something is off there. Especially since we do seem as always to find the money to buy ever-fancier new toys, Matter or priorities, I guess. But, somehow we’ll stagger by, We often seem to do things backward by putting something together first and then figuring out how we should run it, And we never really do things the proper way that we should, But somehow we do manage to get lots of cool things done, not too badly.