1. TOPS “Technology for Optical Pixel Streaming” Paul Wielinga Division Manager High Performance Networking SARA Computing and Networking Services wielinga@sara.nl Courtesy EVL

2. iGrid2005 - GLVF OptIPuter panel Outline About SARA Rationale for adapting OptIPuter concept TOPS Demonstration US117b at iGrid2005 (Preliminary) Results Other demo’s SARA is involved in

3. iGrid2005 - GLVF OptIPuter panel SARA Computing and Networking Services SARA: National e-Science Support Center Home of  Dutch Supercomputing infrastructure  SURFnet NOC  SURFnet6 major core node  NetherLight (sister site of StarLight)  Amsterdam LightHouse OptIPuter affiliate international partner LHC Tier-1 site Proposed LOFAR storage site SARA is independent and not-for-profit

4. iGrid2005 - GLVF OptIPuter panel Teraflops, Terabytes, Terabits, Terapixels @SARA CAVE SGI Origin 3800 STK tape library Tiled-panel display IBM Cluster 1600 Linux Nationaal Rekencluster Netherlight SURFnet 6 SGI Altix 3700

5. iGrid2005 - GLVF OptIPuter panel Why we adopt OptIPuter Science pull:  Petabytes of complex data explored and analyzed by nationally and internationally dispersed scientists, in many teams  SARA’s mission as e-Science Support Center Life Sciences ~PBytes/sec LHC Tier-1 site (with NIKHEF) Brain Activity Analysis Univ Utrecht LOFAR

6. iGrid2005 - GLVF OptIPuter panel Why we adopt OptIPuter (2) Technology push:  SURFnet6 – Hybrid Optical and Packet Switching Network  5 Photonic RIngs  Light Path Provisioning for all universities and research institutions in NL

7. iGrid2005 - GLVF OptIPuter panel TOPS Technology for Optical Pixel Streaming Use OptIPuter concept to split visualization from display  Render images from very large data sets on central visualization cluster  Use high speed (lambda) networks to stream pixels to ultra- high resolution tiled panel displays Adapt the idea of video streaming: constant stream of pixels from renderer to display Use lossy protocols for long distance connectivity: High performance TCP hard to achieve, UDP performance trivial Light weight application Possibly incorporated in SAGE (EVL and UCSD)?

8. iGrid2005 - GLVF OptIPuter panel DEMO US117b – GLVF TOPS Very large 2D and 3D datasets located at SARA in Amsterdam Render cluster (29 nodes) at SARA 20 Gbps connectivity between cluster and NetherLight, via University of Amsterdam switch 2 * 10 Gbps lambda between Amsterdam and San Diego (SURFnet, CaveWave, WANPHY Amsterdam-SD) UCSD LambdaVision display at iGrid: 100 Megapixel Tiled Display 1 Frame = 100 Mpixel*24 bits = 2.4 Gbits Uncompressed and compressed mode Streaming pixels from Amsterdam to San Diego

9. iGrid2005 - GLVF OptIPuter panel Preliminary results First demoslot, 1 10 Gbps available  Sustained bandwidth used between rendering and display 9.2 Gbps Second demoslot, 2 * 10 Gbps  Sustained bandwidth 5,3 + 6,2 = 11,5 Gbps

10. iGrid2005 - GLVF OptIPuter panel Final results Third demoslot, 2 times 10 Gbps available  Sustained bandwidth used between rendering and display 18 Gbps  Peak bandwidth of 19.5 Gbps! New world record of transatlantic bandwidth usage by one single application visualizing scientific content

11. iGrid2005 - GLVF OptIPuter panel Preliminary results Limitation was MTU size – not whole path was jumbo- frame enabled Display cluster had 100% cpu load due to small MTU size Large UDP packet loss (upto 50%) Noticable difference in latency (visible) between CaveWave (with 4 switches) and direct WANPHY connection Later today:  new trial with jumbo frames enabled  Expect higher framerate and bandwidth usage of 15 Gbps or more

12. iGrid2005 - GLVF OptIPuter panel iGrid2005 demonstrations US117b: GLVF – Unreliable Stream of Images (TOPS) SARA participation in other demo’s:  US117a: SAGE (EVL) computing support in Amsterdam  NL102: Dead Cat (UvA) – Visualization support (Amsterdam)  US106: 10 Gb line-speed security (NORTEL) – Networking and computing support (Amsterdam)

13. iGrid2005 - GLVF OptIPuter panel Credits Bram Stolk (SARA) Jeroen Akershoek (SARA) Ronald van der Pol (SARA) Pieter de Boer (SARA) Hanno Pet (SARA) University of Amsterdam (Paola Grosso et al.) SURFnet (Dennis Paus et al.) Jason Leigh, Luc Renambot, Lance Long and colleagues at EVL and UCSD iGrid2005 networking and LambdaVision crews

14. Thank You