Presentation is loading. Please wait.

Presentation is loading. Please wait.

Future Directions in Grid Computing Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science.

Similar presentations


Presentation on theme: "Future Directions in Grid Computing Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science."— Presentation transcript:

1 Future Directions in Grid Computing Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science The University of Chicago http://www.mcs.anl.gov/~foster Invited Talk, Terena 2002 Conference, Limerick, June 5, 2002

2 2 foster@mcs.anl.gov ARGONNE CHICAGO The Grid Vision Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations –On-demand, ubiquitous access to computing, data, and services –New capabilities constructed dynamically and transparently from distributed services When the network is as fast as the computer's internal links, the machine disintegrates across the net into a set of special purpose appliances (George Gilder)

3 3 foster@mcs.anl.gov ARGONNE CHICAGO Why the Grid? (1) Evolution of the Scientific Process l Pre-electronic –Theorize &/or experiment, alone or in small teams; publish paper l Post-electronic –Construct and mine very large databases of observational or simulation data –Develop computer simulations & analyses –Access specialized devices remotely –Exchange information quasi-instantaneously within distributed multidisciplinary teams Need to manage dynamic, distributed infrastructures, services, and applications

4 4 foster@mcs.anl.gov ARGONNE CHICAGO eScience Application: Sloan Digital Sky Survey Analysis

5 5 foster@mcs.anl.gov ARGONNE CHICAGO Size distribution of galaxy clusters? eScience Application: Sloan Digital Sky Survey Analysis Galaxy cluster size distribution Chimera Virtual Data System + iVDGL Data Grid (many CPUs)

6 6 foster@mcs.anl.gov ARGONNE CHICAGO Challenging Technical Requirements l Dynamic formation and management of virtual organizations l Online negotiation of access to services: who, what, why, when, how l Configuration of applications and systems able to deliver multiple qualities of service l Autonomic management of distributed infrastructures, services, and applications l Management of distributed state as a fundamental issue

7 7 foster@mcs.anl.gov ARGONNE CHICAGO State of the Art: Globus Toolkit TM (since 1996) l Small, standards-based set of protocols for distributed system management –Authentication, delegation; resource discovery; reliable invocation; etc. l Information-centric design –Data models; publication, discovery protocols l Open source implementation –Large international user community l Successful enabler of higher-level services and applications

8 8 foster@mcs.anl.gov ARGONNE CHICAGO Grid Projects in eScience

9 9 foster@mcs.anl.gov ARGONNE CHICAGO A Large Virtual Organization: CERNs Large Hadron Collider 1800 Physicists, 150 Institutes, 32 Countries 100 PB of data by 2010; 50,000 CPUs?

10 10 foster@mcs.anl.gov ARGONNE CHICAGO Data Grids for High Energy Physics Tier2 Centre ~1 TIPS Online System Offline Processor Farm ~20 TIPS CERN Computer Centre FermiLab ~4 TIPS France Regional Centre Italy Regional Centre Germany Regional Centre Institute Institute ~0.25TIPS Physicist workstations ~100 MBytes/sec ~622 Mbits/sec ~1 MBytes/sec There is a bunch crossing every 25 nsecs. There are 100 triggers per second Each triggered event is ~1 MByte in size Physicists work on analysis channels. Each institute will have ~10 physicists working on one or more channels; data for these channels should be cached by the institute server Physics data cache ~PBytes/sec ~622 Mbits/sec or Air Freight (deprecated) Tier2 Centre ~1 TIPS Caltech ~1 TIPS ~622 Mbits/sec Tier 0 Tier 1 Tier 2 Tier 4 1 TIPS is approximately 25,000 SpecInt95 equivalents

11 11 foster@mcs.anl.gov ARGONNE CHICAGO Lift Capabilities Drag Capabilities Responsiveness Deflection capabilities Responsiveness Thrust performance Reverse Thrust performance Responsiveness Fuel Consumption Braking performance Steering capabilities Traction Dampening capabilities Crew Capabilities - accuracy - perception - stamina - re-action times - SOPs Engine Models Airframe Models Wing Models Landing Gear Models Stabilizer Models Human Models Grids at NASA: Aviation Safety

12 12 foster@mcs.anl.gov ARGONNE CHICAGO NETWORK IMAGING INSTRUMENTS COMPUTATIONAL RESOURCES LARGE DATABASES DATA ACQUISITION PROCESSING, ANALYSIS ADVANCED VISUALIZATION Life Sciences: Telemicroscopy

13 13 foster@mcs.anl.gov ARGONNE CHICAGO Why the Grid: (2) And What About Business? l Fragmentation of enterprise infrastructure –Specialized platforms -> commodity servers –Intelligence embedded in networks l The rise of the eUtility (IBM, HP, …) –Outsourcing, economies of scale l Business-to-business computing –Especially complex virtual organizations l Ever more challenging QoS requirements –Green-screen -> ubiquitious web presence

14 14 foster@mcs.anl.gov ARGONNE CHICAGO Todays Enterprise Computing Environment

15 15 foster@mcs.anl.gov ARGONNE CHICAGO Grids and Industry: Early Examples Butterfly.net: Grid for multi-player games Entropia: Distributed computing (BMS, Novartis, …)

16 16 foster@mcs.anl.gov ARGONNE CHICAGO The Business Opportunity l On-demand computing, storage, services –Significant savings due to reduced build-out, economies of scale, reduced admin costs –Greater flexibility => greater productivity l Entirely new applications and services –Based on high-speed resource integration l Solution to enterprise computing crisis –Render distributed infrastructures manageable

17 17 foster@mcs.anl.gov ARGONNE CHICAGO Grid Computing

18 18 foster@mcs.anl.gov ARGONNE CHICAGO Realizing the Promise Requires Significant Innovation l Automation of infrastructure operation to achieve economies of scale l Management and component models for distributed service provisioning l New applications and tools powered by distributed services and resources l Business and service models to support specialization of function

19 19 foster@mcs.anl.gov ARGONNE CHICAGO Grid Evolution: Open Grid Services Architecture l Refactor Globus protocol suite to enable common base and expose key capabilities l Service orientation to virtualize resources and unify resources/services/information l Embrace key Web services technologies for standard IDL, leverage commercial efforts l Result: standard interfaces & behaviors for distributed system management: the Grid service

20 20 foster@mcs.anl.gov ARGONNE CHICAGO Open Grid Services Architecture: Transient Service Instances l Web services address discovery & invocation of persistent services –Interface to persistent state of entire enterprise l In Grids, must also support transient service instances, created/destroyed dynamically –Interfaces to the states of distributed activities –E.g. workflow, video conf., dist. data analysis l Significant implications for how services are managed, named, discovered, and used –In fact, much of OGSA (and Grid) is concerned with the management of service instances

21 21 foster@mcs.anl.gov ARGONNE CHICAGO Open Grid Services Architecture l Defines fundamental (WSDL) interfaces and behaviors that define a Grid Service –Required + optional interfaces = WS profile –A unifying framework for interoperability & establishment of total system properties l Defines WSDL extensibility elements –E.g., serviceType (a group of portTypes) l Delivery via open source Globus Toolkit 3.0 –Leverage GT experience, code, community l And commercial implementations

22 22 foster@mcs.anl.gov ARGONNE CHICAGO The Grid Service = Interfaces/Behaviors + Service Data Service data element Service data element Service data element Implementation GridService (required) Service data access Explicit destruction Soft-state lifetime … other interfaces … (optional) Standard: - Notification - Authorization - Service creation - Service registry - Manageability - Concurrency + application- specific interfaces Binding properties: - Reliable invocation - Authentication Hosting environment/runtime (C, J2EE,.NET, …)

23 23 foster@mcs.anl.gov ARGONNE CHICAGO Service Data l A Grid service instance maintains a set of service data elements –XML fragments encapsulated in standard containers –Includes basic introspection information, interface-specific data, and application data l FindServiceData operation (GridService interface) queries this information –Extensible query language support l See also notification interfaces –Allows notification of service existence and changes in service data

24 24 foster@mcs.anl.gov ARGONNE CHICAGO Grid Service Example: Database Service l A DBaccess Grid service will support at least two portTypes –GridService –DBaccess l Each has service data –GridService: basic introspection information, lifetime, … –DBaccess: database type, query languages supported, current load, …, … l Maybe other portTypes as well –E.g., NotificationSource (SDE = subscribers) Grid Service DBaccess DB info Name, lifetime, etc.

25 25 foster@mcs.anl.gov ARGONNE CHICAGO Example: Data Mining for Bioinformatics User Application BioDB n Storage Service Provider Mining Factory Community Registry Database Service BioDB 1 Database Service...... Compute Service Provider I want to create a personal database containing data on e.coli metabolism...... Database Factory

26 26 foster@mcs.anl.gov ARGONNE CHICAGO Example: Data Mining for Bioinformatics User Application BioDB n Storage Service Provider Mining Factory Community Registry Database Service BioDB 1 Database Service...... Compute Service Provider...... Find me a data mining service, and somewhere to store data Database Factory

27 27 foster@mcs.anl.gov ARGONNE CHICAGO Example: Data Mining for Bioinformatics User Application BioDB n Storage Service Provider Mining Factory Community Registry Database Service BioDB 1 Database Service...... Compute Service Provider...... GSHs for Mining and Database factories Database Factory

28 28 foster@mcs.anl.gov ARGONNE CHICAGO Example: Data Mining for Bioinformatics User Application BioDB n Storage Service Provider Mining Factory Community Registry Database Service BioDB 1 Database Service...... Compute Service Provider...... Create a data mining service with initial lifetime 10 Create a database with initial lifetime 1000 Database Factory

29 29 foster@mcs.anl.gov ARGONNE CHICAGO Example: Data Mining for Bioinformatics User Application BioDB n Storage Service Provider Database Factory Mining Factory Community Registry Database Service BioDB 1 Database Service...... Compute Service Provider...... Database Miner Create a data mining service with initial lifetime 10 Create a database with initial lifetime 1000

30 30 foster@mcs.anl.gov ARGONNE CHICAGO Example: Data Mining for Bioinformatics User Application BioDB n Storage Service Provider Database Factory Mining Factory Community Registry Database Service BioDB 1 Database Service...... Compute Service Provider...... Database Miner Query

31 31 foster@mcs.anl.gov ARGONNE CHICAGO Example: Data Mining for Bioinformatics User Application BioDB n Storage Service Provider Database Factory Mining Factory Community Registry Database Service BioDB 1 Database Service...... Compute Service Provider...... Database Miner Query Keepalive

32 32 foster@mcs.anl.gov ARGONNE CHICAGO Example: Data Mining for Bioinformatics User Application BioDB n Storage Service Provider Database Factory Mining Factory Community Registry Database Service BioDB 1 Database Service...... Compute Service Provider...... Database Miner Keepalive Results

33 33 foster@mcs.anl.gov ARGONNE CHICAGO Example: Data Mining for Bioinformatics User Application BioDB n Storage Service Provider Database Factory Mining Factory Community Registry Database Service BioDB 1 Database Service...... Compute Service Provider...... Database Miner Keepalive

34 34 foster@mcs.anl.gov ARGONNE CHICAGO Example: Data Mining for Bioinformatics User Application BioDB n Storage Service Provider Database Factory Mining Factory Community Registry Database Service BioDB 1 Database Service...... Compute Service Provider...... Database Keepalive

35 35 foster@mcs.anl.gov ARGONNE CHICAGO GT3: An Open Source OGSA- Compliant Globus Toolkit l GT3 Core –Implements Grid service interfaces & behaviors –Reference impln of evolving standard –Multiple hosting envs: Java/J2EE, C, C#/.NET? l GT3 Base Services –Evolution of current Globus Toolkit capabilities l Many other Grid services GT3 Core GT3 Base Services Other Grid Services GT3 Data Services

36 36 foster@mcs.anl.gov ARGONNE CHICAGO Summary: Grids and Globus Toolkit l The Grid: Resource sharing & coordinated problem solving in dynamic, multi- institutional virtual organizations l Considerable impact within eScience, growing interest & adoption within eBusiness l Globus Toolkit an open source, defacto standard source of protocol and API definitionsand reference implementations l A strong community organization: the Global Grid Forum

37 37 foster@mcs.anl.gov ARGONNE CHICAGO Summary: Open Grid Services Architecture l Open Grid Services Architecture represents (we hope!) next step in Grid evolution l Service orientation enables unified treatment of resources, data, and services l Standard interfaces and behaviors (the Grid service) for managing distributed state l Deeply integrated information model for representing and disseminating service data l Open source Globus Toolkit implementation (and commercial value adds)

38 38 foster@mcs.anl.gov ARGONNE CHICAGO For More Information l Grid Book (somewhat old) –www.mkp.com/grids l Survey + research articles –www.mcs.anl.gov/~foster l The Globus Project –www.globus.org l GriPhyN project –www.griphyn.org l Global Grid Forum –www.gridforum.org –www.gridforum.org/ogsi-wg –Edinburgh, July 22-24 –Chicago, Oct 15-17


Download ppt "Future Directions in Grid Computing Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science."

Similar presentations


Ads by Google