Presentation is loading. Please wait.

Presentation is loading. Please wait.

Grid Business Symposium 2005, Seoul, Korea

Similar presentations

Presentation on theme: "Grid Business Symposium 2005, Seoul, Korea"— Presentation transcript:

1 Grid Business Symposium 2005, Seoul, Korea
Recent Advances in Grid Computing and Business Models: A Gridbus Perspective WW Grid Grid Business Symposium 2005, Seoul, Korea Rajkumar Buyya Grid and Distributed Systems (GRIDS) Laboratory Dept. of Computer Science and Software Engineering The University of Melbourne Melbourne, Australia

2 Outline Introduction Global Grids and Challenges Grid Initiatives
Utility Networks and Grid Computing Global Grids and Challenges Grid Initiatives World-wide with Australia and India Perspective Introduction to Gridbus Project and Grid Economy Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Economy-based Scheduling in Data Grids Summary

3 4 Essential Utilities and Delivery Networks
(1) Water (2) Electricity (3) Gas (4) Telephone

4 (5) IT services as the fifth utility (water, electricity, gas, telephone, IT)
eScience eBusiness eGovernment eHealth Multilingual eEducation

5 A Bird Eye View of World-Wide Grid Environment
Grid Information Service Grid Resource Broker database Application R2 R3 R4 R5 RN Grid Resource Broker R6 R1 Resource Broker Grid Information Service

6 Grid Resource Sharing & Synergies
Computational Services – CPU cycles NASA IPG, TeraGrid, I-Grid,… Data Services Data replication, management, secure access--LHC Grid/Napster Application Services Access to remote software/libraries and license management—NetSolve Interaction Services eLearning, Virtual Tables, Group Communication (Access Grid), Gaming Knowledge Services The way knowledge is acquired and managed—data mining. Utility Computing Services Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities. Utility Grid Knowledge Grid Interaction Grid ASP Grid Data Grid Computational Grid

7 Grid Challenges Application Construction Security
Uniform Access Security System Management Computational Economy Resource Discovery Resource Allocation & Scheduling Data locality Network Management Application Construction

8 Some Grid Initiatives Worldwide
Australia Nimrod-G Gridbus DISCWorld GrangeNet. APACGrid ARC eResearch Brazil OurGrid, EasyGrid LNCC-Grid + many others China ChinaGrid – Education CNGrid - application Europe UK eScience EU Grids.. and many more... India I-Grid Japan NAGERI Korea... N*Grid Singapore NGP USA Globus NASA IPG AccessGrid TeraGrid Cyberinfrasture Industry Initiatives IBM On Demand Computing HP Adaptive Computing Sun N1 Microsoft - .NET Oracle 10g Satyam – Grid Practice Infosys, Wipro, TCS StorageTek –Grid.. Public Forums Global Grid Forum Australian Grid Forum Conferences: CCGrid Grid HPDC E-Science 27 million 1.3 billion – 3 yrs 2? billion 120million – 5 yrs 450million – 5 yrs 486million – 5 yrs 1.3 billion (Rs) 1 billion – 5 yrs

9 Grid (Market) Participant Types and Application Category
free trading Public computing Alchemi, UD) Private enterprises (IBM, Sun, Satyam) Sharing Model National provider (K*Grid, TeraGrid, Garuda/IndiaGrid, UKGrid) regulation scientific commercial Application Category

10 Grid Computing in Australia (Courtesy: Jihyoun Park, SNU Visitor to Melbourne)
Academia Government Collaboration Industry

11 Academic activities 1 University laboratories for Grid computing
- Uni. of Melbourne(GRIDS lab): Gridbus (GridSim, GMD, GridBank, Alchemi, ..), Master of Engineering in Distributed Computing - Monash Uni.: GriddlsS (Legacy SW to the computational grid), Nimrod-G - Australian national Uni. (Internet Futures Group) - Sydney Uni.(ViSLAB): high performance visualization &computing - Uni. of Adelaide (DHPC Group): DISCWorld - Queensland Uni. of Technology (PLAS): G2 (.NET based) 2 Grid Infrastructure Projects APACGrid, National Neurosciece Facility, Australian Virtual Observatory, several state level facilities (VPAC, TPAC, SAPAC, QPSF, IVEC) 3 Grid Applications * Asia Pacific Bioinformatics Network/ Virtual Drug Design: Molecular Modeling for Drug Design on P2P Grid/ HEPGrid: High Energy Physics and the Grid Network/ Access Grid/Australian Computational Earth Systems Simulator/. * Recently 30 more applications are funded as part of ARC e-Research * Govt. has formed “National e-Research Coordination Committee”.

12 Grid Computing in India
Academia Government Collaboration Industry (majority focus on Grid integration)

13 Grid Computing in India: Academic and Industrial Activities
Academic and Government Initiatives: TIFR, IITM, Anna University, IITD, UoH, etc. C-DAC’s Garuda – Ministry of IT Software Companies in India: Top 4 Indian IT Companies: Satyam, Infosys, TCS (Tata Consultancy Service), and Wipro. Oracle 10g, IBM, HP, Sun ertc. have a large Grid development centers in Bangalore, India. Satyam is leading the pack in Grid Business push: Grid Practice Centre with top management support. Singned MoU with Melbourne University and extensively using Gridbus in powering applications. Also contributing the development of Gridbus technologies (e.g., Alchemi) – SEI CMM Level 5 principles. Application Verticals: Manufacturing, Security, Life Sciences, Finance


15 Australian and Indian Grid Efforts Compared
Korea: Is it like Australia or India?

16 On Demand Utility Computing
The Gridbus Melbourne: Enable Leasing of ICT Services on Demand Distributed Data WWG Gridbus World Wide Grid! On Demand Utility Computing

17 The Gridbus Project:
A multi-institutional “Open Source” R&D Project with focus on: Architecture, Specification, and Open Source Reference Implementation. Service-Oriented Grid, Utility Computing & Distributed Data and Computation Economy Scaling from Desktops, Clusters, Cluster Federation, Enterprise Grids to Global Grids. Alchemi: Harnessing .NET/Windows-based Resources Grid Market Directory and Web Services Grid Bank: Accounting and Transaction Management Visual Tools for Creation of Distributed Applications Workflow Composition and Deployment Services Data Grid Brokering and Grid Economy Services Data Replication Strategies GridSim Toolkit: Enhanced to support Data Grid, Reservation, etc. Libra: SLA-based Allocation of Cluster Resources Coupling of Clusters and Computational Economy WWG: Global Data Intensive Grid Testbed Application Enabler Projects: High-Energy Physics , Astronomy, Brain Activity Analysis – Osaka U., Natural Language Processing, Portfolio Analysis – Spain, BioGrid - WEHI (via APACGrid), SensorGrid (NICTA), Medical Imaging (HFI) Supported by:

18 Grid Economy: Methodology for Sustained Resourced Sharing and Managing Supply-and-Demand for Resources For resource providers, they provide service with different goals For consumers, they have different requirments How to manage grid service supply-and-demand is hard task So how can consumers find a resource depending on their requirements. And what type of resource that providers provide and how can they serve consumers within certain QoS. For example, when many consumers want to access same resource, who should be served first How to manage supply-demand balance is another challenge.

19 New challenges of Grid Economy
Resource Owners How do I decide prices ? (economic models?) How do I specify them ? How do I translate price to resource allocation ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments? ….. Resource Consumers How do I decide expenses ? How do I express QoS requirements ? How do I trade between timeframe & cost ? How do I map jobs to resources to meet my QoS needs? They need mechanisms and technologies for value expression, value translation, and value enforcement.

20 Grid Entities and Architecture
Grid consumer GSP site scheduler Resource owners broker GSP global scheduler GSP site scheduler Resource owners Market Maker accounting End users Private enterprises National providers

21 A Reference Service-Oriented Architecture for Utility Grids
Data Catalogue Grid Bank Grid Market Services Information Service Sign-on Health Monitor Info ? Grid Explorer Grid Node N Programming Environments Secure Job Control Agent Applications Schedule Advisor Grid Node1 QoS Pricing Algorithms Trade Server Trading Trade Manager Accounting Resource Reservation Misc. services Deployment Agent JobExec Resource Allocation Grid Resource Broker Storage R1 R2 Rm Grid Middleware Services Grid Consumer Grid Service Providers

22 Gridbus and Complementary Technologies – realizing Utility Grid
Applications Science Commerce Engineering Collaboratories Portals MPI ExcellGrid Gridscape Workflow X-Parameter Sweep Lang. User-Level Middleware (Grid Tools) Grid Brokers: Nimrod-G Workflow Engine Gridbus Data Broker Core Grid Middleware Globus Unicore Grid Storage Economy Grid Exchange & Federation Grid Market Directory Grid Bank Alchemi NorduGrid XGrid GRIDSIM Grid Economy .NET JVM Condor PBS SGE Libra Tomcat Grid Fabric Software Windows Solaris Linux AIX IRIX OSF1 Mac CDB Grid Fabric Hardware PDB Worldwide Grid

23 Alchemi: .NET-based Enterprise Grid Platform & Web Services
Alchemi Manager Web Services Internet Alchemi Users Internet like Model General Purpose Dedicated/Non-dedicate workers Role-based Security .NET and Web Services C# Implementation GridThread and Job Model Programming Easy to setup and use Widely in use! Alchemi Worker Agents

24 Many users in Universities: See next for an example.
Some Users of Alchemi Tier Technologies, USA Large scale document processing using Alchemi framework Satyam Computers Applied Research Laboratory, India Micro-array data processing using Alchemi framework CSIRO, Australia Natural Resource Modeling The University of Sao Paulo, Brazil The Alchemi Executor as a Windows Service stochastix GmbH, Germany Asynchronous Excel Tasks using ManagedXLL and Alchemi .Net Grid Computing framework. The Friedrich Miescher Institute (FMI) for Biomedical Research, Switzerland Patterns of transcription factors in mammalian genes Many users in Universities: See next for an example.

25 On Demand Assembly of Services: Putting Them All Together
Visual Application Composer Application Code Explore data 1 Data Source Data Results+ Cost Info 10 Grid Resource Broker 2 Bill 12 (Instruments/distributed sources) Data Replicator (GDMP) Data Catalogue 5 ASP Catalogue Grid Info Service Grid Market Directory 4 6 3 Job 8 Results 9 7 GSP (e.g., UofM) PE (e.g., VPAC) (e.g., IBM) CPU or PE Grid Service (GS) (Globus) Alchemi GS GTS Cluster Scheduler GSP (Accounting Service) Gridbus GridBank 11 Cluster Scheduler PE Grid Service Provider (GSP) (e.g., CERN)

26 The Gridbus Grid Service Broker for Data Grid Applications
Builds on the Nimrod-G Computational Grid Broker and Computational Economy [Buyya, Abramson, Giddy, Monash University, ] And Extends its notion for Data and Service Grids

27 Gridbus Broker Architecture
Gridbus Client Gribus Client Gridbus Client (Bag of Tasks Applications) App, T, $, Opt (Data Grid Scheduler) Gridbus Farming Engine Schedule Advisor Record Keeper Trading Manager Grid Dispatcher Grid Explorer Grid Middleware TM TS $ GE GIS, NWS Grid Info Server RM & TS G $ Data Catalog Data Node C $ U G Unicore enabled node. Globus enabled node. L A RM: Local Resource Manager, TS: Trade Server Alchemi enabled node.

28 Gridbus Broker and Remote Service Access Enablers
Home Node/Portal Gridbus Broker fork() batch() -PBS -Condor -SGE -Alchemi -XGrid Credential Repository MyProxy Portlets Data Catalog Alchemi -PBS -Condor -SGE Globus Job manager fork() batch() Gridbus agent Data Store Gateway Unicore -PBS -Condor -SGE -XGrid SSH fork() batch() Gridbus agent Access Technology SRB Grid FTP

29 Gridbus Services for eScience applications
Application Development Environment: XML-based language for composition of task farming (legacy) applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e.g., Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow engine. Resource Allocation and Scheduling Dynamic discovery of optional computational and data nodes that meet user QoS requirements. Hide Low-Level Grid Middleware interfaces Globus, Alchemi, Unicore, NorduGrid, XGrid, etc.

30 Figure 3 : Logging into the portal.
Click Here for Demo Figure 3 : Logging into the portal. Drug Design Made Easy!

31 Economy-based Data Grid Scheduling
CLICK HERE TO SKIP IF RUNNING OUT of TIME High Energy Physics as eScience Application Case Study

32 Australian Belle Data Grid Testbed

33 Case Study: Event Simulation and Analysis
B0->D*+D*-Ks Simulation and Analysis Package - Belle Analysis Software Framework (BASF) Experiment in 2 parts – Generation of Simulated Data and Analysis of the distributed data Analyzed 100 data files (30MB each) were distributed among the five nodes

34 Resources Used and their Service Price
Organization Node details Role Cost (in G$/CPU-sec) CS,UniMelb 4 CPU, 2GB RAM, 40 GB HD, Linux Broker host, Data host, NWS server N.A. (Not used as a compute resource) Physics, UniMelb 1 CPU, 512 MB RAM, 40 GB HD, Linux Replica Catalog host, Data host, Compute resource, NWS sensor 2 CS, University of Adelaide 4 CPU (only 1 available) , 2GB RAM, 40 GB HD, Linux Data host, NWS sensor ANU, Canberra 4 CPU, 2GB RAM, 40 GB HD, Linux Data host, Compute resource, NWS sensor 4 Dept of Physics, USyd 4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux VPAC, Melbourne 180 node cluster (only head node used), Linux Compute resource, NWS sensor 6

35 Network Cost (in Grid $/Currency!)

36 Deploying Application Scenario
A data grid scenario with 100 jobs and each accessing remote data of ~30MB Deadline: 3hrs. Budget: G$ 60K Scheduling Optimisation Scenario: Minimise Time Minimise Cost Results:

37 Grid and Gridbus Technologies for Various Grid (Market) Types
free trading Public computing (Alchemi) Private enterprises (Libra, Gridbus, Globus) Sharing Model National provider (Globus, Gridbus,..) regulation scientific commercial Application Category

38 (5) IT services as the fifth utility (water, electricity, gas, telephone, IT)
eScience eBusiness eGovernment eHealth Multilingual eEducation

39 Summary and Conclusion
Grids exploit synergies that result from cooperation of autonomous entities: Resource sharing, dynamic provisioning, and aggregation at global level. Grid Economy provides incentive needed for sustained cooperation. Grid Network has potential to serve as Cyberinfrastructure for Utility Computing Grids offer enormous opportunities for realizing eScience and eBusiness at global level.

40 Gridbus Project -
Any Questions ? Gridbus Project -

41 Thanks for your attention!
The Gridbus Cooperation!

42 Backup Slides

43 Some Open Research Challenges
Value expression, translation, and enforcement mechanisms and supporting Grid technologies for: different economic models for spot markets and futures application models Dynamic Pricing Schemes Interaction Protocols for Service Negotiation Micro payments and Digital Currencies Scheduling Algorithms Programming Environments for Building Information Utility Applications Last, but not least: Dispute Managements and Legal Issues Taxation (consult your National Taxation Office) State, national, and international boundaries Tax returns!

44 This talk is designed to answer:
How can Grid technologies support the emergence and operation of virtual enterprises? How can Grid shared resources be treated, brokered, and marketed as ICT ‘commodities’ or ‘futures’ among networked organisations? What kind of Grid architecture is needed for handling such market mechanisms in an automated fashion? How can Grid economies map the evolution of networked business models?

45 What do Grids aim for and how to support them.
Grids aim at exploiting synergies that result from cooperation of autonomous distributed entities. Synergies include: Resource sharing “On-demand” Virtual Enterprises creation Aggregation of resources on demand. For this cooperation to be sustainable, participants needs to have (economic) incentive. Therefore, “incentive” mechanisms should be considered as one of key design parameters of Grid computing.

46 Grid Market (Participant) Types and Application Category
free trading Public computing Private enterprises Sharing Model National provider regulation scientific commercial Application Category

47 Appropriate Market Model for different market types
high Variable price auction Commodity market Willingness to Pay Posted price oligopoly low weak strong Demand elasticity

48 Realising Market-based Grid: Minimal New Components
Grid Market Directory Services Grid Trading Services – for different economic models Grid Metering Services Grid Accounting and Payment Services Grid Service Broker

49 Deadline (D) and Budget (B) Constrained Scheduling Algorithms
Execution Time (D) Execution Cost (B) Compute Grid Data Grid Cost Opt Limited by D Minimize Yes Cost-Time Opt Minimize if possible Time Opt Limited by B Conservative-Time Opt Limited by B, jobs have guaranteed minimum budget

Download ppt "Grid Business Symposium 2005, Seoul, Korea"

Similar presentations

Ads by Google