1. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY DOE-UltraScience Net (& network infrastructure) Update JointTechs Meeting February 15, 2005 W. R. Wing

2. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Motivation - DOE Needs Extreme Networking  160-200 Gbs throughput by 2008  Probably only achievable by circuit bonding  0.1% packet re-order and jitter control  Probably only achievable at SONET layer  Line rate, provably-secure, connections  Only available at SONET or optical layer

3. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY The Well Known Bandwidth Problem  DOE needs 160 - 200 Gbs by 2008  Typical multi-stream throughput limited to 25+/- Gbs, and programming multi-steam is hard - hero efforts yield ~30 Gbs  Typical single-steam throughput to a single application ~1Gbs, even on a tuned network - hero effort yields 5-6+ Gbs  Networks will soon offer 40 Gbs channels  Firewalls throw it away anyway

4. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Even Without Firewalls - We Have a Technology Problem DOE needs 160-200 Gbs by 2008 This is on the “blue line” This isn’t a problem routers are going to fix - bonded IP channels ONLY work with VERY large numbers of streams (as Teragrid has discovered) Bonding at layer-2 (Ethernet) is just as bad Solution is circuit switching and Layer-1 bonding

5. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Jitter Control  Needed to deconstruct data sets, needed for remote vis, and for remote instrument control or steering  Routers and switch-routers are path deterministic, but not time-deterministic  Only at layer-1 (or SONET) do you get time-determinism

6. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Security  Security requirements will get more and more onerous  Solutions involving IPSec, VPN’s, and encryption have more and more trouble running at line speed  Alternate solution is to use optical/SONET circuits with provable, known end-points and inherent immunity to injected traffic  Again - only available at layer-1

7. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY In Summary  Only at layer-1 do you get:  Zero packet re-ordering  Zero jitter  Zero drops due to Congestion  Known (by definition) paths and end points  SONET (or below) can do this  Ethernet switches and switch/routers can’t  MPLS can’t  Layer-1 circuits can:  Bypass firewalls  Carry non-IP frames (e.g., Fiber-Channel over SONET)  Easily (transparently) support parallel, bonded circuits

8. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY MPLS Jitter - ORNL / Atlanta

9. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY How do we get there from here?  It requires a research network we control at least down to the SONET layer  It requires a research network with significant span  It requires a research network with at least two lambdas

10. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY UltraScience Net Research Testbed  Building an extended-regional lambda-switching testbed  Connect to NLR in Atlanta and Chicago  Use asset-trading to extend reach (Sunnyvale and East Coast)  Provide an evolving matrix of switching capabilities  Separately fund research projects (e.g., high- performance protocols, control, visualization) that will exercise the network and directly support applications at the host institutions

11. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY In More Detail -

12. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Connections to National Nets ORNL ORNL-TVA Internet2 NLR ORNL ESnet TeraGrid

13. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Research Projects  A progression of switching approaches  Study/compare SONET, MPLS(GMPLS), and all optical  New all-optical technologies coming (e.g., laser tuning)  Fast Local Storage  Storage Depots  Transport-optimized storage  Progression of experimental point-to-point transport technologies  Fiber channel  Infiniband  LAN-PHY Ethernet initial transport  Circuit-Switched backbone, Frame-Switched edge

14. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Digression: LAN-PHY vs. WAN-PHY  Conventional Wisdom: LAN-PHY will win  Cheaper, faster, etc. (based on POS experience)  False cost model (extra processing)  Ignore SONET advantages  New Transponders are transparent  Software selectable OC192 (WAN–PHY) or LAN– PHY  Costs now equal  Advantage now WAN-PHY  DCC and OAM non-trivial considerations

15. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY PoS vs. Ethernet DATA IP POSPADPOS DATA IP EEPAD DATA IP DATA IP POSPADPOSSONET DATA IP EEPAD DATA IP DATA IP EEPADSONET

16. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Status and Schedule  All contracts signed  Fiber, Co-Lo space and power, equipment and installation, smart-hands  Equipment shipping, built-out started  Chicago-Sunnyvale paced by NLR cross- connects in Chicago (Level(3) to Starlight)  NLR supplying 10Gig-E (LAN-PHY) initially  PNNL fiber schedule will pace their connection  First Chicago-Sunnyvale traffic this month  Details on following VuGraphs -

17. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Phase-1 (February)

18. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Phase-2 (February-March) N x 1Gig-E ORNL M160 SOX M320

19. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Phase-3 (March - April)

20. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Phase-4 (April - May)

21. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Thank You http://www.csm.ornl.gov/ultranet