Presentation on theme: "Recent Advances in Grid Computing and Business Models: A Gridbus Perspective Rajkumar Buyya Grid and Distributed Systems (GRIDS) Laboratory Dept. of Computer."— Presentation transcript:
Recent Advances in Grid Computing and Business Models: A Gridbus Perspective Rajkumar Buyya Grid and Distributed Systems (GRIDS) Laboratory Dept. of Computer Science and Software Engineering The University of Melbourne Melbourne, Australia www.gridbus.org WW Grid Grid Business Symposium 2005, Seoul, Korea
2 Outline Introduction Utility Networks and Grid Computing Global Grids and Challenges Grid Initiatives World-wide with Australia and India Perspective Introduction to Gridbus Project and Grid Economy Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Economy-based Scheduling in Data Grids Summary
3 4 Essential Utilities and Delivery Networks (1) Water (2) Electricity (3) Gas (4) Telephone
4 (5) IT services as the fifth utility (water, electricity, gas, telephone, IT) eScience eBusiness eGovernment eHealth Multilingual eEducation …
5 A Bird Eye View of World-Wide Grid Environment Grid Resource Broker Resource Broker Application Grid Information Service Grid Resource Broker database R2R2 R3R3 RNRN R1R1 R4R4 R5R5 R6R6 Grid Information Service
6 Grid Resource Sharing & Synergies Computational Services – CPU cycles SETI@Home, NASA IPG, TeraGrid, I-Grid, … Data Services Data replication, management, secure access-- LHC Grid/Napster Application Services Access to remote software/libraries and license management NetSolve Interaction Services eLearning, Virtual Tables, Group Communication (Access Grid), Gaming Knowledge Services The way knowledge is acquired and managed data mining. Utility Computing Services Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities. Computational Grid Data Grid ASP Grid Interaction Grid Knowledge Grid Utility Grid
7 Grid Challenges Security Resource Allocation & Scheduling Data locality Network Management System Management Resource Discovery Uniform Access Computational Economy Application Construction
8 Some Grid Initiatives Worldwide Australia Nimrod-G Gridbus DISCWorld GrangeNet. APACGrid ARC eResearch Brazil OurGrid, EasyGrid LNCC-Grid + many others China ChinaGrid – Education CNGrid - application Europe UK eScience EU Grids.. and many more... India I-Grid Japan NAGERI Korea... N*Grid Singapore NGP USA Globus NASA IPG AccessGrid TeraGrid Cyberinfrasture Industry Initiatives IBM On Demand Computing HP Adaptive Computing Sun N1 Microsoft -.NET Oracle 10g Satyam – Grid Practice Infosys, Wipro, TCS StorageTek – Grid.. Public Forums Global Grid Forum Australian Grid Forum Conferences: CCGrid Grid HPDC E-Science http://www.gridcomputing.com 1.3 billion – 3 yrs 1 billion – 5 yrs 450million – 5 yrs 486million – 5 yrs 1.3 billion (Rs) 27 million 2? billion 120million – 5 yrs
9 Grid (Market) Participant Types and Application Category commercial scientific free trading regulation Public computing (SETI@Home, Alchemi, UD) National provider (K*Grid, TeraGrid, Garuda/IndiaGrid, UKGrid) Private enterprises (IBM, Sun, Satyam ) Application Category Sharing Model
10 Grid Computing in Australia (Courtesy: Jihyoun Park, SNU Visitor to Melbourne) Academia Government Collaboration Indus try
11 Academic activities 1 University laboratories for Grid computing - Uni. of Melbourne(GRIDS lab): Gridbus (GridSim, GMD, GridBank, Alchemi,..), Master of Engineering in Distributed Computing - Monash Uni.: GriddlsS (Legacy SW to the computational grid), Nimrod-G - Australian national Uni. (Internet Futures Group) - Sydney Uni.(ViSLAB): high performance visualization &computing - Uni. of Adelaide (DHPC Group): DISCWorld - Queensland Uni. of Technology (PLAS): G2 (.NET based) 2 Grid Infrastructure Projects APACGrid, National Neurosciece Facility, Australian Virtual Observatory, several state level facilities (VPAC, TPAC, SAPAC, QPSF, IVEC) 3 Grid Applications * Asia Pacific Bioinformatics Network/ Virtual Drug Design: Molecular Modeling for Drug Design on P2P Grid/ HEPGrid: High Energy Physics and the Grid Network/ Access Grid/Australian Computational Earth Systems Simulator/. * Recently 30 more applications are funded as part of ARC e-Research * Govt. has formed National e-Research Coordination Committee.
12 Grid Computing in India Academia Governm ent Collabor ation Industry (majority focus on Grid integration)
13 Grid Computing in India: Academic and Industrial Activities Academic and Government Initiatives: TIFR, IITM, Anna University, IITD, UoH, etc. C-DAC s Garuda – Ministry of IT Software Companies in India: Top 4 Indian IT Companies: Satyam, Infosys, TCS (Tata Consultancy Service), and Wipro. Oracle 10g, IBM, HP, Sun ertc. have a large Grid development centers in Bangalore, India. Satyam is leading the pack in Grid Business push: Grid Practice Centre with top management support. Singned MoU with Melbourne University and extensively using Gridbus in powering applications. Also contributing the development of Gridbus technologies (e.g., Alchemi) – SEI CMM Level 5 principles. Application Verticals: Manufacturing, Security, Life Sciences, Finance
15 Australian and Indian Grid Efforts Compared Australia India Korea: Is it like Australia or India?
16 The Gridbus Project @ Melbourne: Enable Leasing of ICT Services on Demand WWG World Wide Grid! On Demand Utility Computing Gridbus Distributed Data
17 The Gridbus Project: http://www.gridbus.org A multi-institutional Open Source R&D Project with focus on: Architecture, Specification, and Open Source Reference Implementation. Service-Oriented Grid, Utility Computing & Distributed Data and Computation Economy Scaling from Desktops, Clusters, Cluster Federation, Enterprise Grids to Global Grids. Alchemi: Harnessing.NET/Windows-based Resources Grid Market Directory and Web Services Grid Bank: Accounting and Transaction Management Visual Tools for Creation of Distributed Applications Workflow Composition and Deployment Services Data Grid Brokering and Grid Economy Services Data Replication Strategies GridSim Toolkit: Enhanced to support Data Grid, Reservation, etc. Libra: SLA-based Allocation of Cluster Resources Coupling of Clusters and Computational Economy WWG: Global Data Intensive Grid Testbed Application Enabler Projects: High-Energy Physics, Astronomy, Brain Activity Analysis – Osaka U., Natural Language Processing, Portfolio Analysis – Spain, BioGrid - WEHI (via APACGrid), SensorGrid (NICTA), Medical Imaging (HFI) Supported by:
18 Grid Economy: Methodology for Sustained Resourced Sharing and Managing Supply-and-Demand for Resources
19 New challenges of Grid Economy Resource Owners How do I decide prices ? (economic models?) How do I specify them ? How do I translate price to resource allocation ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments? ….. Resource Consumers How do I decide expenses ? How do I express QoS requirements ? How do I trade between timeframe & cost ? How do I map jobs to resources to meet my QoS needs? ….. They need mechanisms and technologies for value expression, value translation, and value enforcement.
20 Grid Entities and Architecture GSP site scheduler accounting Grid consumer Market Maker GSP global scheduler broker Resource owners End users Private enterprises National providers GSP site scheduler Resource owners
21 Grid Node N A Reference Service-Oriented Architecture for Utility Grids Grid Consumer Programming Environments Grid Resource Broker Grid Service Providers Grid Explorer Schedule Advisor Trade Manager Job Control Agent Deployment Agent Trade Server Resource Allocation Resource Reservation R1R1 Misc. services Information Service R2R2 RmRm … Pricing Algorithms Accounting Grid Node1 … Grid Middleware Services … … Health Monitor Grid Market Services JobExec Info ? Secure Trading QoS Storage Sign-on Grid Bank Applications Data Catalogue
23 Alchemi:.NET-based Enterprise Grid Platform & Web Services Internet Alchemi Worker Agents Alchemi Manager Alchemi Users Web Services SETI@Home like Model General Purpose Dedicated/Non-dedicate workers Role-based Security.NET and Web Services C# Implementation GridThread and Job Model Programming Easy to setup and use Widely in use!
24 Some Users of Alchemi Tier TechnologiesTier Technologies, USA Large scale document processing using Alchemi framework CSIROCSIRO, Australia Natural Resource Modeling The Friedrich Miescher Institute (FMI) for Biomedical ResearchThe Friedrich Miescher Institute (FMI) for Biomedical Research, Switzerland Patterns of transcription factors in mammalian genes Satyam Computers Applied Research LaboratorySatyam Computers Applied Research Laboratory, India Micro-array data processing using Alchemi framework The University of Sao PauloThe University of Sao Paulo, Brazil The Alchemi Executor as a Windows Service stochastix GmbHstochastix GmbH, Germany Asynchronous Excel Tasks using ManagedXLL and Alchemi.Net Grid Computing framework. Many users in Universities: See next for an example.
25 On Demand Assembly of Services: Putting Them All Together Data Source (Instruments/dis tributed sources) Data Replicator (GDMP) ASP Catalogue Grid Info Service Grid Market Directory GSP (Accounting Service) Gridbus GridBank Data GSP (e.g., UofM) PE GSP (e.g., VPAC) PE GSP (e.g., IBM) CPU or PE Grid Service (GS) (Globus) Alchemi GS GTS Cluster Scheduler Grid Service Provider (GSP) (e.g., CERN) PE Cluster Scheduler Job 8 Grid Resource Broker 2 Visual Application Composer Application Code Explore data 1 36 45 Results 97 Results+ Cost Info 10 11 Bill 12 Data Catalogue
The Gridbus Grid Service Broker for Data Grid Applications Builds on the Nimrod-G Computational Grid Broker and Computational Economy [Buyya, Abramson, Giddy, Monash University, 1999-2001] And Extends its notion for Data and Service Grids
27 Gridbus Broker Architecture Grid Middleware Gridbus Client Gribus Client Grid Info Server Schedule Advisor Trading Manager Gridbus Farming Engine Record Keeper Grid Explorer GE GIS, NWS TM TS RM & TS Grid Dispatcher RM: Local Resource Manager, TS: Trade Server G G C U Globus enabled node. A L Alchemi enabled node. (Data Grid Scheduler) Data Catalog Data Node Unicore enabled node. $ $ $ App, T, $, Opt (Bag of Tasks Applications)
29 Gridbus Services for eScience applications Application Development Environment: XML-based language for composition of task farming (legacy) applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e.g., Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow engine. Resource Allocation and Scheduling Dynamic discovery of optional computational and data nodes that meet user QoS requirements. Hide L ow-Level Grid Middleware interfaces Globus, Alchemi, Unicore, NorduGrid, XGrid, etc.
30 Figure 3 : Logging into the portal. Drug Design Made Easy! Click Here for Demo
Economy-based Data Grid Scheduling High Energy Physics as eScience Application Case Study CLICK HERE TO SKIP IF RUNNING OUT of TIME
33 Case Study: Event Simulation and Analysis B0->D*+D*-Ks Simulation and Analysis Package - Belle Analysis Software Framework (BASF) Experiment in 2 parts – Generation of Simulated Data and Analysis of the distributed data Analyzed 100 data files (30MB each) were distributed among the five nodes
34 Resources Used and their Service Price Organization Node detailsRoleCost (in G$/CPU- sec) CS,UniMelbbelle.cs.mu.oz.au 4 CPU, 2GB RAM, 40 GB HD, Linux Broker host, Data host, NWS server N.A. (Not used as a compute resource) Physics, UniMelbfleagle.ph.unimelb.edu.au 1 CPU, 512 MB RAM, 40 GB HD, Linux Replica Catalog host, Data host, Compute resource, NWS sensor 2 CS, University of Adelaide belle.cs.adelaide.edu.au 4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux Data host, NWS sensor N.A. (Not used as a compute resource) ANU, Canberrabelle.anu.edu.au 4 CPU, 2GB RAM, 40 GB HD, Linux Data host, Compute resource, NWS sensor 4 Dept of Physics, USyd belle.physics.usyd.edu.au 4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux Data host, Compute resource, NWS sensor 4 VPAC, Melbourne brecca-2.vpac.org 180 node cluster (only head node used), Linux Compute resource, NWS sensor 6
36 Deploying Application Scenario A data grid scenario with 100 jobs and each accessing remote data of ~30MB Deadline: 3hrs. Budget: G$ 60K Scheduling Optimisation Scenario: Minimise Time Minimise Cost Results:
37 Grid and Gridbus Technologies for Various Grid (Market) Types commercial scientific free trading regulation Public computin g (Alchemi) National provider (Globus, Gridbus,..) Private enterprises (Libra, Gridbus, Globus) Application Category Sharing Model
38 (5) IT services as the fifth utility (water, electricity, gas, telephone, IT) eScience eBusiness eGovernment eHealth Multilingual eEducation …
39 Summary and Conclusion Grids exploit synergies that result from cooperation of autonomous entities: Resource sharing, dynamic provisioning, and aggregation at global level. Grid Economy provides incentive needed for sustained cooperation. Grid Network has potential to serve as Cyberinfrastructure for Utility Computing Grids offer enormous opportunities for realizing eScience and eBusiness at global level.
40 Any Questions ? Gridbus Project - http://www.gridbus.org
41 Thanks for your attention! The Gridbus Cooperation! http://www.gridbus.com
43 Some Open Research Challenges Value expression, translation, and enforcement mechanisms and supporting Grid technologies for: different economic models for spot markets and futures application models Dynamic Pricing Schemes Interaction Protocols for Service Negotiation Micro payments and Digital Currencies Scheduling Algorithms Programming Environments for Building Information Utility Applications Last, but not least: Dispute Managements and Legal Issues Taxation (consult your National Taxation Office) State, national, and international boundaries Tax returns!
44 This talk is designed to answer: How can Grid technologies support the emergence and operation of virtual enterprises? How can Grid shared resources be treated, brokered, and marketed as ICT commodities or futures among networked organisations? What kind of Grid architecture is needed for handling such market mechanisms in an automated fashion? How can Grid economies map the evolution of networked business models?
45 What do Grids aim for and how to support them. Grids aim at exploiting synergies that result from cooperation of autonomous distributed entities. Synergies include: Resource sharing On-demand Virtual Enterprises creation Aggregation of resources on demand. For this cooperation to be sustainable, participants needs to have (economic) incentive. Therefore, incentive mechanisms should be considered as one of key design parameters of Grid computing.
46 Grid Market (Participant) Types and Application Category commercial scientific free trading regulation Public computin g National provider Private enterprises Application Category Sharing Model
47 Appropriate Market Model for different market types strong weak high low Variable price auction Posted price oligopoly Commodity market Demand elasticity Willingn ess to Pay
48 Realising Market-based Grid: Minimal New Components Grid Market Directory Services Grid Trading Services – for different economic models Grid Metering Services Grid Accounting and Payment Services Grid Service Broker
49 Deadline (D) and Budget (B) Constrained Scheduling Algorithms AlgorithmExecution Time (D) Execution Cost (B) Compute Grid Data Grid Cost OptLimited by DMinimize Yes Cost-Time Opt Minimize if possible Minimize Yes Time OptMinimizeLimited by B Yes Conservative -Time Opt MinimizeLimited by B, jobs have guaranteed minimum budget Yes