1. KIT – University of the State of Baden-Württemberg and National Laboratory of the Helmholtz Association www.dfn.de / www.kit.edu DFN–Verein e.V DFN@100G – A Field Trial of 100G Technologies related to Real World Research Scenarios Bruno Hoeft (KIT/SCC) Robert Stoy (DFN) Co-authors: Andreas Hanemann (DFN)

2. 2 Agenda 18.05.2011 Testbed features and goals Testbed layout and commissioning 100GE test phase between endsystems Experiences Bruno Hoeft -- KIT Robert Stoy -- DFN

3. 3 DFN100G Testbed Features and Test Goals Features Use of typical fiber in the ground over distance of 447 km Not limited to optical transmission (contrast to most other trials) Interoperability: Fiber, Optical Transmission System, Routers, typical endsystems Prototype of new IEEE 802.3ba Standard 100GE interfaces in multivendor environment interconnnectiong routers and DWDM system Test of aggregated TCP and UDP streams behaviour through one IP Link at 100G Long duration tests Distributed environment with end users at KIT and FZJ, and monitoring at DFN Goals Bring the whole system into operation Fill the 100G Pipe with IP traffic Stress test Stability validation Get experience, learn from the unforeseen 18.05.2011 Bruno Hoeft -- KIT Robert Stoy -- DFN

4. 4 Agenda 18.05.2011 Testbed features and Goals Testbed layout and commissioning 100GE test phase between endsystems Experiences Bruno Hoeft -- KIT Robert Stoy -- DFN

5. 5 Testbed Topology Overview Huawei 100GE DWDM … Düsseldorf Köln Bonn Koblenz Frankfurt Wiesbaden Mainz Darmstadt Mannheim FZ Jülich KIT Karlsruhe Dernbach Gernsheim Porz Naurod 88 km 60 km 96 km 64 km 29 km 93 km 17 km J– 01 10.2.33.11 J– 09 10.2.41.11 K– 01 10.2.21.11 K– 09 10.2.29.11 End systems used for measurements: Linux Servers, 10GE Interfaces  DFN Measurement Systems,  Hades : OWD, OWDV Measurement  Iperf : Packet Loss and Packet Reordering  Additionally at KIT: UDP Traffic Load Generator  Huawei Transmission System (Two lamdas 100G + 40G) between locations KIT Karlsruhe, FZ Jülich CFPs 100GBase-LR4  Cisco Routers: CRS-3, Fabric Cards 140G/Slot MSC-40G + 10GE PLIMs + XENPAKs MSC-140G + 100G PLIM + CFPs 100GBase-LR4 ms-fzj1 hades- fzj1 ms-kit1 hades- kit1 ti-kit1 …  Gasline Fiber  normal ‘in the ground’ fiber  Proof of 100G readiness  447 km length with 7 segments and 6 amplifier nodes Bruno Hoeft -- KIT Robert Stoy -- DFN

6. 6 Methodology  Monitoring SNMP Monitoring of all router interfaces with high time granularity (1min) Data stored in RRD Files with extended round robin time (6 months) Router CLI Commands Output Logging (interval 10min) Store Console Logs Maintaining a log with events and manual activities.  Active Measurements Packet Loss and Packet Reordering Iperf UDP stream with 5 Gbit/s alternating in each direction at 5min intervals Availability: Permanent continuous pings between DFN end systems (1s interval) One Way Delay, One Way Delay Variation: Hades Boxes  Data Transmission and Link Load Background load generation using configured L3 Loop between Routers, and Software UDP Load Generator on Linux end system. Iperf TCP throughput measurements between multiple pairs of end systems. Bruno Hoeft -- KIT Robert Stoy -- DFN

7. 7 Test 1.x UDP Load on Routing Loop between routers test systems, administration: DFN, 10GE Interface: 10GBase LR 1.1 Single UDP/IP flow on routing loop, traffic in both directions 1.2. Multiple UDP/IP flows on routing loop, traffic in both directions trc-kit1 CRS-3 Te0/1/0/0 Te0/1/0/2 Te0/1/0/1 Te0/1/0/3 Te0/3/0/0 Te0/3/0/1 Te0/3/0/3 Te0/2/0/0 Te0/2/0/2 Te0/2/0/1 Te0/2/0/3 trc-fzj1 CRS-3 Te0/1/0/0 Te0/1/0/2 Te0/1/0/1 Te0/1/0/3 Te0/3/0/0 Te0/2/0/0 Te0/2/0/2 Te0/2/0/1 Te0/2/0/3 Hu0/0/0/0 KarlsruheJülich distance ca. 447km ms-kit1 Te0/3/0/2 hades- kit2 Te0/3/0/2 Te0/3/0/3 Te0/3/0/1 ms-fzj1 hades- fzj2 ti-kit1 x 127 18.05.2011 single mode fibre multi mode fibre 07/22/2010 time bar Bruno Hoeft -- KIT Robert Stoy -- DFN

8. 8 Test 1.2.2-1 Ten UDP/IP flows through L3 Routing Loop Page 8 07/22/201008/06/2010 time bar Bruno Hoeft -- KIT Robert Stoy -- DFN

9. 9 Experiences and Challanges during Comminssioning Phase  Fiber and optical transmission system was established in foreseen timeframe  Error free and stable 100GE transmission was verified during a 30h test using a 100GE Ethernet Tester.  Interconnecting Cisco with Huawei on the 100GBase-LR4 interface was a challenge, and solved  Prototype CFPs: Some of them have been manufactured more than one year before the IEEE Standard ratification. Small input error ratio seen at the 100GE router interface at FZJ (less than 10 -7 ) was caused by the oldest preproduction CFP at KIT New CFPs had no incompatibility issues, and run ‘plug and play’ 19.10.2010 time bar 18.05.2011 07/22/2010 Bruno Hoeft -- KIT Robert Stoy -- DFN

10. 10 Test 1. UDP Load on Routing Loop 1.2. Multiple UDP/IP flows on routing loop, traffic in both directions 1.2.2 Ten parallel UDP/IP flows on routing loop 19.10.2010 08/06/2010 18.05.2011 time bar Bruno Hoeft -- KIT Robert Stoy -- DFN

11. 11 Agenda 18.05.2011 Testbed features and Goals Testbed layout and commissioning 100GE test phase between endsystems Experiences Bruno Hoeft -- KIT Robert Stoy -- DFN

12. 12 Endhosts @ FZJ 19.10.201018.05.2011 FZJFZJ KITKIT J– 09 … J– 01 K– 01 … K– 09 Huawei 100GE DWDM Cisco CSR-3 9 hosts [K1 to K9]: Dual QuadCore (Intel® Xeon® CPU X5560 @2.80GHz 32 GB Memory Model: "Broadcom Corporation NetXtreme II BCM57711 10-Gigabit PCIe" Type 1 (three hosts [J1 to J3]): 2 * Dual Core (AMD Opteron (P 265)), 2GB RAM Model: "Myricom Myri-10G Dual-Protocol NIC (10G-PCIE-8A)" Type 2 (one host [J4]): 2 * Dual Core (Intel® Xeon™ CPU 3.00GHz, 2 GB RAM Model: "Myricom Myri-10G Dual-Protocol NIC (10G-PCIE-8A)" Type 3 (five hosts [J5 to J9]): Dual QuadCore (Intel® Core™ i7 CPU 860 @2.80GHz 3 GB RAM Model: "Myricom Myri-10G Dual-Protocol NIC (10G-PCIE-8A)" Bruno Hoeft -- KIT Robert Stoy -- DFN

13. 13 Overall Testbed Topology 18.05.2011 07/22/201008/06/2010 time bar Bruno Hoeft -- KIT Robert Stoy -- DFN

14. 14 Test 2.x TCP iperf Tests between connected end hosts 2.1. Single TCP/IP flow between connected iperf stations 2.2. Multiple TCP/IP flows 18.05.2011 08/09/2010 07/30/2010 time bar Bruno Hoeft -- KIT Robert Stoy -- DFN

15. 15 2. TCP iperf Tests 2.2 Multiple TCP/IP flows between KIT -/- FZJ 10GE hosts 2.2.2 simplex from KIT to FZJ, successive activation of TCP flows 19.10.2010 08/09/2010 18.05.2011 time bar Bruno Hoeft -- KIT Robert Stoy -- DFN

16. 16 Test 3.x TCP iperf Tests between end hosts and parallel UDP background traffic load on routing loop 3.1 Constant UDP load on Routing Loop and ramping up of TCP data between end hosts 3.2 Nine TCP streams between end systems and ramping up and down of UDP background load 18.05.2011 08/10/201008/11/2010 time bar x 127 Bruno Hoeft -- KIT Robert Stoy -- DFN

17. 17 9 TCP Streams, + UDP Background Load on Routing Loop, + UDP Loss Measurement (Test 3.2.1) 08/10/2010 18.05.2011 time bar Bruno Hoeft -- KIT Robert Stoy -- DFN

18. 18 9 TCP Streams, + UDP Background Load on Routing Loop, + UDP Loss Measurement (Test 3.2.1) 18.05.2011 08/10/2010 time bar Bruno Hoeft -- KIT Robert Stoy -- DFN

19. 19 Test 4.x max. TCP streams, end hosts KIT -> FZJ, incl. DFN hosts 4.2.1 TCP Streams KIT  FZJ 4.2.2 TCP Streams FZJ  KIT 18.05.2011 08/11/201008/16/2010 time bar tsc-fzj2 4900M Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 tsc-fzj1 4900M Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 trc-kit1 CRS-3 Te0/1/0/0 Te0/1/0/2 Te0/1/0/1 Te0/1/0/3 Te0/3/0/0 Te0/3/0/1 Te0/2/0/0 Te0/2/0/2 Te0/2/0/1 Te0/2/0/3 tsc-kit1 4900M Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 tsc-kit2 4900M Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 Te2/2 Te2/4 Te1/2 Te1/4 Te1/6 trc-fzj1 CRS-3 Te0/1/0/0 Te0/1/0/2 Te0/1/0/1 Te0/1/0/3 Te0/3/0/0 Te0/2/0/0 Te0/2/0/2 Te0/2/0/1 Te0/2/0/3 Hu0/0/0/0 KarlsruheJülich distance ca. 447km K1 K2 K3 K4 K5 K6 K7 K8 K9 J1 J2 J3 J4 J5 J6 J7 J8 J9 Te0/3/0/3 Te0/3/0/2 Te0/3/0/3 Te0/3/0/1 ms-fzj1 hades- fzj2 end systems, administration: KIT, FZJ, 10GE Interface: 10GBase SR end systems, administration: DFN, 10GE Interface: 10GBase LR single mode fibre multi mode fibre Te0/3/0/2 dfn-fzj1 dfn-fzj2 dfn-kit2 dfn-kit1 Bruno Hoeft -- KIT Robert Stoy -- DFN

20. 20 TCP Throughput FZJ -> KIT, 11 Streams (Test 4.2.2-0) 08/13/2010 18.05.2011 time bar Bruno Hoeft -- KIT Robert Stoy -- DFN

21. 21 Agenda 18.05.2011 Testbed features and Goals Testbed layout and commissioning 100GE test phase between endsystems Experiences Bruno Hoeft -- KIT Robert Stoy -- DFN

22. 22 Experience obtained from the tests 19.10.2010 GasLINE fiber infrastructure is ready for 100GE Total system, including router and DWDM systems, was very stable and highly available during the three weeks test period Early field tests During commissioning : some equipment needed manual adjustment by Cisco and Huawei engineers During pre production tests: Single vendor cfp combination was required, problem solved in the meantime by vendors CRC error ratio (below 10 -7 ) in one direction only (kit  fzj) Caused by the oldest pre production cfp in the testbed located at the router at KIT error ratio did not reduce the TCP throughput 100GE was tested successfully up to the applications point of view 100GE technology on all layers has been proven to be production ready 18.05.2011 Bruno Hoeft -- KIT Robert Stoy -- DFN

23. 23 Acknowledgements Thanks to the Project Partners: 19.10.201018.05.2011 Bruno Hoeft -- KIT Robert Stoy -- DFN

24. KIT – University of the State of Baden-Württemberg and National Laboratory of the Helmholtz Association www.dfn.de / www.kit.edu DFN–Verein e.V DFN@100G – A Field Trial of 100G Technologies related to Real World Research Scenarios Bruno Hoeft / Robert Stoy