1. 1 Lambda Data Grid An Agile Optical Platform for Grid Computing and Data-intensive Applications Tal Lavian Qualifying Exam Proposal October 29th, 2003 Committee Members: Jean Walrand (chair), Randy Katz, Connie Chang, John Chuang, and John Strand

2. 2 Radical mismatch L1 – L3 Radical mismatch between the optical transmission world and the electrical forwarding/routing world. Currently, a single strand of optical fiber can transmit more bandwidth than the entire Internet core Current L3 architecture can’t effectively transmit PetaBytes or 100s TeraBytes Current L1-L0 limitations: Manual allocation, takes 6-12 months - Static. –Static means: not dynamic, no end-point connection, no service architecture, no glue layers, no applications underlay routing

3. 3 Current L3 Architecture Was not designed to support Data-Intensive applications L3 Xfer 1.5TB (10 12 ), 1.5KB packets –Make the same routing decision 1 Billion times (10 9 ) –Over 10+ hops Web, Telnet, email – small files Fundamental limitations with data-intensive applications –multi TeraBytes or PetaBytes of data –move 10KB and 10GB (or 10TB) are Different ( x 10 6, x 10 9 ) –1Mbs & 10Gbs are different ( x 10 6)

4. 4 Lambda Hourglass Data Intensive (DI) applications requirements –HEP, Astrophysics/Astronomy, Bioinformatics, Computational Chemistry Inexpensive disk – 1TB < $1,000 DWDM – Abundant optical bandwidth 280 λs, OC-192 Data-Intensive Applications Lambda Data Grid Abundant Optical Bandwidth CERN 1-PB 2.8 Tbs on single fiber strand

5. 5 Problem Lacking infrastructure to support DI applications Grid Computing research is lacking the networking aspects –focus: middleware, L7+ No L0-L1 –optimizes computation and storage –networking is not a first class citizen –gap between applications and resources VO is not possible without full network support as an integral part of Grid Computing “Without the network, nothing is possible!!” - Ian Foster: ANL “The heart of any Grid is the Network” - Fran Bermen: SDSC

6. 6 Hypothesis / Idea It is possible to: –adapt dynamic optical paths to meet DI application needs? –overcome packet switching limitations for DI applications? achieve high E2E bandwidth –to do so in a scalable way? –to build this abstraction in a layered architecture (reuse)? Use Lambda Data Grid (LDG) –OGSI-ification to optical control –Expand OGSA for the network –Extend GRAM/GARA give optical network equal status as storage & computation –Novel data-network scheduling scheme –Underlay network

7. 7 Challenges (== Research Agenda) Overcome Grid VO networking limitations –network as integral Grid resource –evaluate design tradeoffs Overcome static optical limitations –hide network complexity from the application –start with a single administrative domain Simplify the interaction between these two distinct and isolated research communities –get the architecture and interfaces right “Bandwidth is still a problem” –abundant optical bandwidth that data-intensive applications can’t use

8. 8 How can a Lambda Data Grid support HEP? Source: CERN ~100 Petabytes by 2007-8 An Exabyte ~2013 Tier2 Center Online System CERN Center PBs of Disk; Tape Robot FNAL Center IN2P3 Center INFN Center RAL Center Institute Workstations ~100-1500 MBytes/sec 2.5-10 Gbps 0.1 to 10 Gbps Physics data cache ~PByte/sec ~2.5-10 Gbps Tier2 Center ~2.5-10 Gbps Tier2 Center Experiment

9. 9 Overview of the plan Implement on-demand and scheduled data transfer and network resource services –As Grid services Investigate the interaction between OGSA/OGSI and the underlying networks Develop generic infrastructure for data intensive applications over dynamic optical networks. –Infrastructure that can support Grid services Non-Grid services Optical underlying networks Non-Optical underlying networks (extension) Evaluate by developing and deploying several real applications –Lambda Grid as a scalable infrastructure for application data transport –Effective management of network resources in heterogeneous environments –Useable by the Grid community

10. 10 Outline Initial Design –Lambda Data Grid –Overall architecture –Underlay network –Intelligent services (DTS & NRS) Research issues Design of the infrastructure for DI applications Encapsulating network resources into a Grid service, Middleware that can interface to the underlying network Resource scheduling and negotiation Evaluation Methodology –Actual performance measured –Two testbeds: Concept testbed in Santa Clara and OMMInet in Chicago Initial Results Research methodology Related work Timeline & Summary

11. 11 Outline Initial Design Research issues Evaluation Methodology Initial Results Research methodology Related work Timeline & Summary

12. 12 What is Lambda Data Grid? A service architecture –comply with OGSA –Lambda as an OGSI service –GRAM extension add parameter of network resource net resource co-allocation –on-demand and scheduled Lambda GT3 implementation –contribute code back to GT3 Data Grid Service Plane Network Service Plane Centralize Optical Network Control Lambda Service Grid Computing Applications Grid Middleware Our contribution

13. 13 Layered Architecture GRAM Net Ext Data Grid Service Network Service Centralize Optical Network Control Grid Middleware OGSA/OGSI GARA Net ExtCo-allocation Lambda OGSI-ification Intelligent Services NRS DTS Generic DI Infrastructure Co-reservation

14. 14 Lambda Data Grid details A framework providing optical paths for Data-Intensive applications Service architecture –interacts w/ optical control –dedicated on-demand optical path –not on the public Internet –novel data-networking scheduling scheme MIT Technology Review: Grid Computing - “One of the ten technologies that will change the world” OGSIOGSA GT3 Services GRAMGARACo-allocationCo-reservation DTS Intelligent Midleware services Ext NRS

15. 15 This framework will: Allow applications/services –to be deployed over the Lambda Data Grid Expand OGSA –for integration with optical network Extend OGSI –interface with optical control –infrastructure and mechanisms Extend GRAM and GARA –to provide framework for network resources optimization Provide generalized framework for multi-party data scheduling

16. 16 Data Transfer Schedule (DTS) Service Uses NRS Service Schedules data transfer requests May coordinate with other Grid Services Routing thru topology to create segment-by- segment path requests Rerouting/rescheduling to accommodate new requests Callback mechanism to clients for rescheduling Various resource costing mechanisms Developing a rich scheduling language DTS Service uses NRS -- schedules xfers, does routing in network

17. 17 Network Resource Scheduler (NRS) Service Overlays ODIN Imports topology from ODIN End-to-end path allocation/deallocation Segment-level path allocation/deallocation Contains knowledge of schedule, and ability to service schedule requests One NRS Service per administrative domain New concept: under-constrained schedule request NRS Service overlays this -- scheduling, export of topology and segment-level allocation interface. Centralized (one NRS, DTS per administrative domain)

18. 18 Outline Initial Design Research issues Evaluation Methodology Initial Results Research methodology Related work Timeline & Summary

19. 19 Encapsulation: Encapsulating network resources into Grid Service Common perception: –Network as bottleneck –Network as a plentiful shared resource that users do not have to reserve portions for their applications. Network should be thought of as a resource in the same way as processing and storage Issue: How to encapsulate network resources into a Grid service Difficulty: Network resources always entail entities that may not be co-located and often belong to different virtual organizations Network: Line not a point

20. 20 Adaptive Middleware: Middleware that can interface and adapt to the underling network New generation services must be capable of receiving information about the network and adaptively tune their transport requirements Issue: require middleware that is capable of interfacing and adapt to the underlying network technologies –Resource discovery service –Resource Information service –Resource reservation, allocation, and management –Network and data transfer scheduling services

21. 21 Generic Architecture: Exploring infrastructures for DI applications Issue : need infrastructure that –Supports movement of large data sets –Treats network resource as a first class citizen just like processing and storage resources. –Allows coordination of resources for applications

22. 22 Resource Scheduling and Negotiation Constrained vs Under-Constrained Requests Fully constrained problem: one (or no) solution –"Give me this route between nodes A and B at 3:00 today for 1/2 an hour" Under-constrained problem may allow many solutions –"Give me a route between nodes A & B any time after 4:00 today for 45 minutes" Under-constrained problems provide flexibility, especially when coordinating other resources & users Constrained vs under-constrained scheduling issues. Requests either succeed or fail. Success has info returned to caller (as does failure)

23. 23 Outline Initial Design Research issues Evaluation Methodology Initial Results Research methodology Related work Timeline & Summary

24. 24 metrics for success 1) concept accepted by the grid research community 2) contribution adopted by GGF (measured by drafts approval) 3) widespread adoption in the Data-intensive research community 1) Add GRAM/GARA extensions –part of OGSA/OGSI, WSDL, GT3 implementation –under-constrained, scheduler, on-demand, window –explicitly demonstrate co-allocation, and optimization 2) convince people to try Lambda as a services –CERN, ANL, SLAC, NASA, SDSC, LBNL, FLAB –Use OMNInet as a testbed, and my code as a platform –Incorporate feedback of real applications –Requests for collaborations and follow-ups Evaluation: Use real Grid applications

25. 25 Lambda Data Grid: Analysis What are we doing? –Enabling Data-intensive applications Why is it important? –Not really realizable with current packet switching technology –Network resource is an important dimension in OGSA but it has not been properly addressed –Provide infrastructure for next generation data services Why is it challenging? –Staking new ground for Grid services –Staking new ground for dynamic optical networks –Staking new ground for data-intensive services –Alternative to packet switching technology

26. 26 Optical Control Network Network Service Request Data Transmission Plane OmniNet Control Plane ODIN UNI-N ODIN UNI-N Connection Control L3 router L2 switch Data storage switch Data Path Control Data Path Control DATA GRID SERVICE PLANE 1 n Data Center 1 n 1 n Data Path Data Center Service Control Service Control NETWORK SERVICE PLANE GRID Service Request NTONC DWDM-RAM Service Architecture

27. 27 Santa Clara Testbed Site A (GT, Compute nodes) Site B (GT, Compute nodes) Dynamic VLAN Connection Lake Shore Omni Net Control Plane Sheridan S. Federal W Taylor GT1 GT2 GT3GT4 CP1-CP4 – Desktops running Debian Linux UNI-N SW GT1-GT4 – Linux workstations with Globus TK DWDM RAM Service Plane Globus or Demo App Plane UNI –C Test WS DTS NRS WS1 – CO2 Grid /DWDM RAM Control Server WS2 – ODIN/THOR Server Starlight Source Dest ODIN/THOR

28. 28 GGF -9 Demo Challenge: –Emerging data intensive applications require: Extremely high performance, long term data flows Scalability for data volume and global reach Adjustability to unpredictable traffic behavior Integration with multiple Grid resources Response: –LDG – A service architecture for data intensive Grids enabled by next generation dynamic optical networks, incorporating new methods for lightpath provisioning

29. 29 Data Management Service Uses standard ftp Uses OGSI calls to request network resources Uses NRS to allocate lambdas Designed for future scheduling λ Data ReceiverData Source FTP clientFTP server DTSNRS Client App OGSI

30. 30 Outline Initial Design Research issues Evaluation Methodology Initial Results Research methodology Related work Timeline & Summary

31. 31 Actual Performance Measured 0.5s2.6s0.5s464s0.3s11s ODIN Server Processing File transferdone, pathreleasedFile transferrequestarrives Path Deallocation request Data Transfer 10 GB Path ID returned ODIN Server Processing Path Allocation request 45s Network reconfiguration 0.14s FTP setup time

32. 32 10GB File Transfer

33. 33

34. 34 Path Allocation Overhead as a % of the Total Transfer Time Knee point shows the file size for which overhead is insignificant 1GB5GB 500GB

35. 35 Outline Initial Design Research issues Evaluation Methodology Initial Results Research methodology Related work Timeline & Summary

36. 36 Research Methodology Tradeoffs –Lambda allocation, co-allocation (storage computation, visualization) –resource analysis and measurements –algorithms optimization Proof of concept –understand the Grid needs –propose solution –GRAM extension Implementation –build a prototype –use the OMNInet Scalability, adoptability –what is missing for easy use? –apps interface –resource utilization Evaluation Analysis Design

37. 37 Research Methodology: Metrics Realization: –Overhead Breakdown of End-to-end transfer time Overall throughput –Scalability Support extremely large file transfer Significance: –Adoption by the Grid community Integration to the OGSA/OGSI –Adoption by DI community –Generality For building applications that can adapt to the underlying network technologies

38. 38 Research Methodology: Approach Concept testbed, Real implementation testbed, Collaboration Concept testbed –For exploring scheduling services and control infrastructure Real implementation –For realizing both control and data transport studies –Real services over real dynamic optical networks Collaboration –Collaborating with ICAIR, USC, UTS –(OptIPuter, BIRN, CERN, SLAC, ANL, CANARIE)

39. 39 Outline Initial Design Research issues Evaluation Methodology Initial Results & Testbed Research methodology Related work Timeline & Summary

40. 40 Related Work Grid: Blueprint, Anatomy, Physiology – –I. Foster, C. Kesselman OGSA/OGSI – I. Foster, S. Tuecke –(network dimension is needed in the architecture) GARA, GRAM –(inadequate in managing network resources) OptiPuter – D. Grosman, T. DeFanti CANARIE Grid testbed (&OBGP) – B. Arnuad CANARIE Lambda Disk - B. Arnuad Condor-G : J. Frey, T. Tannenbaum, M. Livny, I. Foster, S. Tuecke.

41. 41 Outline Initial Design Research issues Evaluation Methodology Initial Results & Testbed Research methodology Related work Timeline & Summary

42. 42 Pro Forma Timeline Phase 1 (0-6 months) –Continue to develop the Lambda Data Grid prototype OMNInet testbed in Chicago, GGF-9, SC-2003, GW-2004 –Develop new extension to GRAM and GARA incorporate Lambda as a service, extend co-reservation & scheduling measure/analyze the performance and scaling behavior Phase 2 (6-12 months) –Develop an OGSI wrapper to Lambda service integrate as part of OGSA, open to scientific community co-allocation, co-reservation, integration –Develop intelligent scheduling services –Analyze the overall performance, incorporate the enhancements Phase 3 (12-18 months) –Complete implementation, multi-data location, –Extract generalized framework for intelligent services and applications –Incorporate experience from the Grid research community –Measure, optimize, measure, optimize, …, write, graduate

43. 43 Summary Proof of concept –Agile optical network provides essential communication infrastructure for DI applications. –Network resources is a fundamental service within OGSA Develop an architecture for Data-Intensive services enabled by dynamic optical networks Extend the Grid Resource Allocation and Management service Extend the Open Grid Service Infrastructure for enabling intelligent services. Develop intelligent scheduling services for data coordination/transfer and network resources.

44. 44 futures work (5 years out) All items in "Evolutionary Plans" Lambda File System, “Best” replica Negotiations between multiple Administrative Domains Use predictive models of network and other resource use to inform scheduling & routing (statistical, data mining) Multiplexing multiple lambdas Ways of "filling the pipe" –working with virtual file systems and distributed RAIDS to write N lambdas into the optical pipe and to read N lambdas from the pipe to the destination file systems.

45. 45 Backup Slides

46. 46 New optical advances DWDM – a lot of bandwidth & growing Very low cost per bit –Optical vs. L3 - orders of magnitude cheaper per bit Optical is static vs. L3 is Dynamic –New direction dynamic optical lightpath –GMPLS, ASON, ASTN, UNI, NNI, –MEMS –MEF, OIF, RPR

47. 47 NAC AFM Edge Device Internet NAC AFM Edge Device Control Data Management Internet Intelligent Edge Device

48. 48 Possible Implementation on Optical Network User signaling Control plane= IP network Data Plane = Optical Transport Network Transfer data Client SourceDest Find node

49. 49 TCP - Large RTT - Responsiveness CaseCRTT (ms)MSS (Byte)Responsiveness Typical LAN in 198810 Mb/s[ 2 ; 20 ]1460[ 1.7 ms ; 171 ms ] Typical LAN today1 Gb/s2 (worst case)146096 ms Futur LAN10 Gb/s2 (worst case)14601.7s WAN Geneva Sunnyvale 1 Gb/s120146010 min WAN Geneva Sunnyvale 1 Gb/s180146023 min WAN Geneva Tokyo 1 Gb/s30014601 h 04 min WAN Geneva Sunnyvale 2.5 Gb/s180146058 min Futur WAN CERN Starlight 10 Gb/s12014601 h 32 min Futur WAN link CERN Starlight 10 Gb/s1208960 (Jumbo Frame) 15 min Source – S. Ravot Caltecth & CERN 10Gbs CERN Tokyo > 3 Hours

50. 50 FAST Bandwidth Records 9/01 105 Mbps 30 Streams: SLAC-IN2P3; –102 Mbps 1 Stream CIT-CERN 1/09/02 190 Mbps for One stream shared on Two 155 Mbps links 5/20/02 450-600 Mbps SLAC-Manchester on OC12 with ~100 Streams 6/1/02 290 Mbps Chicago-CERN –One Stream on OC12 (mod. Kernel) 9/02 850, 1350, 1900 Mbps Chicago-CERN –1,2,3 GbE Streams, 2.5G Link 11-12/02: 930 Mbps in 1 Stream California-CERN, &CA – AMS –FAST TCP 9.4 Gbps in 10 Streams California-Chicago 2/03 2.38 Gbps in 1 Stream California-Geneva –(99% Link Utilization) 5/03 935 Mbps IPv6 in 1 Stream Chicago-CERN 10/03 5.44 Gbs Chicago- CERN –Memory to memory

51. 51 How L3 transfers data? GigaByte – Not easy TeraByte – Hard PetaByte (2003) – Not scalable ExaByte (2013) – Not on the radar of L3 Inexpensive disk: 1TB - $700 1PB - $700k

52. 52 Layered Grid Architecture Application Fabric “Controlling things locally”: Access to, & control of, resources Connectivity “Talking to things”: communicat’n (Internet protocols) & security Resource “Sharing single resources”: nego- tiating access, controlling use Collective “ Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services Internet Transport Application Link Internet Protocol Architecture “The Anatomy of the Grid: Enabling Scalable Virtual Organizations”, Foster, Kesselman, Tuecke, Intl J. High Performance Computing Applications, 15(3), 2001.

53. 53 Layered Architecture Lambda Data Grid - Globus Services CONNECTION Fabric SABUL UDP ODIN OMNInet Storage Bricks Grid FTP GRAM GSI Particle Physics Multidisciplinary Simulation SOAP TCP/HTTP Grid Layered Architecture NRS Storage Service DTS IP Connectivity Application Resource Collaborative GARA

54. 54 BW requirements #users#users C A B ADSL GigE A.Lightweight users, browsing, mailing, home use Need full Internet routing, one to many B.Business applications, multicast, streaming, VPN’s, mostly LAN Need VPN services and full Internet routing, several to several + uplink C.Special scientific applications, computing, data grids, virtual-presence Need very fat pipes, limited multiple Virtual Organizations, few to few  A ≈ 20 Gb/s  B ≈ 40 Gb/s  C ≈ 100 Gb/s (9 of 15) Source: Cees De Laat

55. 55 The 13.6 TF TeraGrid: Computing at 40 Gb/s 26 24 8 4 HPSS 5 UniTree External Networks Site Resources NCSA/PACI 8 TF 240 TB SDSC 4.1 TF 225 TB CaltechArgonne TeraGrid/DTF: NCSA, SDSC, Caltech, Argonne www.teragrid.org Not a Scalable Solution

56. 56 GRAM ( Grid Resource Allocation Management ) GRAM-1  HTTP-based RPC GRAM-1.5 (Reliability improvements)  Once-and-only-once submission  Reliable termination detection  GRAM-2  towards integration with web-services (SOAP)  OGSA (Open Grid Services Architecture)  Gate Keeper  Single point of entry  “secure inetd”  Job Manager  Layers on top of local resource management system (e.g., PBS,LSF, etc)  Handles remote interaction with the job Source: Globus web site

57. 57 GRAM LSFCondorPBS Application RSL Simple ground RSL Information Service Local resource managers RSL specialization Broker RSL Co-allocator Queries & Info Source: Globus web site Resource Management Architecture

58. 58 GARA General-purpose Architecture for Reservation and Allocation (GARA) –2nd generation resource management services Two contributions –Generalize to support various resource types CPU, storage, network, devices, etc. –Advance reservation of resources, in addition to allocation GRAM 2 and Globus 3.0 Source: Globus web site

59. 59 GARA Architecture Application Information service Co-reservation agent Co-allocation agent RM Resource RM Resource RM Resource Resources discovery CreateReservation calls Resource spec. res. handles res. handles + computation spec. obj. handles Source: Globus web site

60. 60 Resource Allocation

61. 61 Flexibility- Allows Heuristic "Optimization" Problem is NP-Complete, but... Rerouting in topology network –provides flexibility for new requests that conflict over one or more segments and keep same times Rescheduling within allowed/requested window –provides flexibility for new or higher priority requests Measures of "success" –ability to service requests & their timely completion Under-constrained allows variety of solutions, allows reroute and/or reschedule, as required

62. 62 Example 1: Time Shift Request for 1/2 hour between 4:00 and 5:30 on Segment D granted to User W at 4:00 New request from User X for same segment for 1 hour between 3:30 and 5:00 Reschedule user W to 4:30; user X to 3:30. Everyone is happy. Route allocated for a time slot; new request comes in; 1st route can be rescheduled for a later slot within window to accommodate new request 4:305:005:304:003:30 D 4:305:005:304:003:30 W 4:305:005:304:003:30 D W

63. 63 Example 2: Reroute Request for 1 hour between nodes A and B between 7:00 and 8:30 is granted using Segment X (and other segments) is granted for 7:00 New request for 2 hours between nodes C and D between 7:00 and 9:30 This route needs to use Segment E to be satisfied Reroute the first request to take another path thru the topology to free up Segment E for the 2nd request. Everyone is happy A D B C X 7:00-8:00 A D B C X Y Route allocated; new request comes in for a segment in use; 1st route can be altered to use different path to allow 2nd to also be serviced in its time window

64. 64 Extension of Under-Constrained Concepts Initially, we use simple time windows More complex extensions –any time after 7:30 –within 3 hours after Event B happens –cost function (time) –numerical priorities for job requests Extend (eventually) concept of under- constrained to user-specified utility functions for costing, priorities, callbacks to request scheduled jobs to be rerouted/rescheduled (client can say yea or nay)

65. 65 OMNInet Characteristics A 4-node multi-site optical metro 10GE testbed with dynamically switched lambdas Connected to other research optical networks Programmable via ODIN/THOR, OGSI interfaces Enhanced UNI-C, UNI-N –topology discovery –segment-level allocation & deallocation of paths End-to-end path allocation and deallocation Out-of-band control plane Data transfers thru optical data plane

66. 66 Architecture Applications Replication, Disk, Accounting, Authentication, Etc. ftp, GridFTP, Sabul, fast, etc Architecture, Page 2 OGSI provided for all application layer interfaces DRS NRS Other Svcs DTS Other dwdm … ODIN omninet ’s