Presentation is loading. Please wait.

Presentation is loading. Please wait.

Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

Similar presentations


Presentation on theme: "Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer."— Presentation transcript:

1 Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer Science and Software Engineering The University of Melbourne, Australia Gridbus Sponsors

2 2 The GRIDS Melbourne Youngest and one of the rapidly growing research labs in our School/University: Founded in 2002 Houses 20+ researchers consisting of: Research Fellows/PostDocs Software Engineers PhD candidates Honours/Masters students Funding National and International organizations Australian Research Council & DEST Many industries (Sun, StorageTek, Microsoft, IBM, Microsoft) University-wide collaboration: Faculties of Science, Engineering, and Medicine Many national and international collaborations. Academics Industries Software: Widely in academic and industrial users. Publication: My research team produces over 20% of our Depts research output. EducationR & D + Community Services: e.g., IEEE TC for Scalable Computing

3 3 Agenda Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services Global Grids and Challenges Security, resource management, pricing models, … Service-Oriented Grid Architecture and Gridbus Solutions Market-based Management, GMD, Grid Bank, Aneka Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Summary and Conclusion

4 4 Computer Utilities Vision: Implications of the Internet 1969 – Leonard Kleinrock, ARPANET project As of now, computer networks are still in their infancy, but as they grow up and become sophisticated, we will probably see the spread ofcomputer utilities, which, like present electric and telephone utilities, will service individual homes and offices across the country Computers Redefined 1984 – John Gage, Sun Microsystems The network is the computer 2008 – David Patterson, U. C. Berkeley The data center is the computer. There are dramatic differences between of developing software for millions to use as a service versus distributing software for millions to run their PCs 2008 – Cloud is the computer – Buyya!

5 5 Computing Paradigms and Attributes: Realizing the Computer Utilities Vision Web Data Centres Utility Computing Service Computing Grid Computing P2P Computing Market-Oriented Computing Cloud Computing … -Ubiquitous access -Reliability -Scalability -Autonomic -Dynamic discovery - Composability -QoS -SLA - … } + Paradigms Attributes/Capabilities ? -Trillion $ business - Who will own it?

6 * Since Grids have been around for sometime (early 2000), do we have a unified vision of what Grids can do? * And did we make sufficient advances to turn vision of computer utilities into a reality? - - Let us take a look at views of - industrial practitioners & academics

7 7 Industrial vision of Grid computing IBM On Demand Computing Microsoft.NET Oracle 10g Sun N1 – Sun Grid Engine HP Adaptive Enterprise Amazon Elastic Compute Cloud Services Manjrasoft Aneka for building enterprise Grids and Clouds.

8 8 Most academics view: Cyberinfrastructure for conducting collaborative (e-)Science

9 9 How do Grids look like? A Bird Eye View of a Global Grid Grid Resource Broker Resource Broker Application Grid Information Service Grid Resource Broker database R2R2 R3R3 RNRN R1R1 R4R4 R5R5 R6R6 Grid Information Service

10 10 How do Grids look like? A Bird Eye View of a Global Grid Grid Resource Broker Resource Broker Application Grid Information Service Grid Resource Broker database R2R2 R3R3 RNRN R1R1 R4R4 R5R5 R6R6 Grid Information Service

11 11 How Are Grids Used? High-performance computing Collaborative data-sharing Collaborative design Drug discovery Financial modeling Data center automation High-energy physics Life sciences E-Business E-Science Natural language processing Utility computing Business Intelligence (Data Mining)

12 12 Agenda Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services Global Grids and Challenges Security, resource management, pricing models, … Service-Oriented Grid Architecture and Gridbus Solutions Market-based Management, GMD, Grid Bank, Aneka Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Summary and Conclusion

13 13 Grid Challenges Security Resource Allocation & Scheduling Data locality Network Management System Management Resource Discovery Uniform Access Computational Economy Application Construction

14 14 Some Grid Initiatives Worldwide Australia Nimrod-G Gridbus DISCWorld GrangeNet. APACGrid ARC eResearch Brazil OurGrid, EasyGrid LNCC-Grid + many others China ChinaGrid – Education CNGrid - application Europe UK eScience EU Grids.. and many more... India Garuda Japan NAREGI Korea... N*Grid Singapore NGP USA Globus TeraGrid Cyberinfrasture AutoMate and many more... Industry Initiatives IBM On Demand Computing HP Adaptive Computing Sun N1 Microsoft -.NET Oracle 10g Amzon – Elastic Compute Cloud Infosys – Enterprise Grid Satyam – Business Grid Manjrasoft – enterprise Clouds and Grids and many more Public Forums Open Grid Forum Conferences: CCGrid Grid HPDC E-Science 1.3 billion – 3 yrs 1 billion – 5 yrs 450million – 5 yrs 486million – 5 yrs 1.3 billion (Rs) 27 million 2? billion 120million – 5 yrs

15 15 Open-Source Grid Middleware Projects Slide by Hiro

16 16 Driving Theme: Community vs. Utility Grids Type Feature Community GridsUtility Grids (Now Clouds) User QoSBest effortContract/SLA Service Pricing Not considered / free access Usage, QoS level, Market supply and demand Example Middleware Globus, Condor, OMII, Unicore Nimrod-G, Gridbus, & many inspired efforts (IBM Business Grid, Sun Grid Market).. Amazon EC2..

17 17 The Gridbus Melbourne: Enable Leasing of ICT Services on Demand WWG Pushes Grid computing into mainstream computing Gridbus

18 18

19 19 Agenda Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services Global Grids and Challenges Security, resource management, pricing models, … Service-Oriented Grid Architecture and Gridbus Solutions Market-based Management, GMD, Grid Bank, Aneka Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Summary and Conclusion

20 20 What do Grid players want & require? Grid Service Consumers (GSCs): - minimize expenses, meet QoS How do I express QoS requirements ? How do I trade between timeframe & cost ? How do I discover services and map jobs to meet my QoS needs? How do I manage Grid dynamics and get my work done? … Grid Service Providers (GSPs):– maximise ROI How do I decide service pricing models ? How do I specify them ? How do I translate them into resource allocations ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments? … They need mechanisms, tools and technologies that help them in value expression, value translation, and value enforcement.

21 21 Grid Node N Service-Oriented Grid Architecture Grid Service Consumer Programming Environments Grid Resource Broker Grid Explorer Schedule Advisor Trade Manager Job Control Agent Deployment Agent Information Service Pricing Algorithms Grid Node1 … Core Middleware Services … … Health Monitor Grid Market Services JobExec Info ? Secure Trading QoS Storage Sign-on Grid Bank Applications Data Catalogue Grid Service Providers Trade Server Resource Allocation Resource Reservation R1R1 Misc. services R2R2 RmRm … Accounting

22 22 Market-Oriented Grid Software: A union of Gridbus and other technologies AIX Solaris WindowsLinux.NET Grid Fabric Software Grid Applications Core Grid Middleware User-Level Middleware Grid Bank Grid Exchange & Federation JVM Grid Scheduling: Task, Parametric, and Components Programming Gridbus Resource Broker MPI CondorSGETomcatPBS Aneka Cloud (WS- based access + SLA Workflow APIs IRIXOSF1 Mac Libra GlobusUnicore … … Grid Market Directory PDB Worldwide Grid Grid Fabric Hardware Grid PortalsScienceCommerceEngineering … … Collaboratories … … Grid Storage Economy Grid Economy NorduGridXGrid ExcellGrid Grid Workflow Engine APIs/Tools: CDB

23 23 On Demand Assembly of Services in Market-Oriented Grid Environments ASP Catalogue Grid Info Service Grid Market Directory GSP (Accounting Service) Gridbus GridBank GSP (e.g., Microsoft) PE GSP (e.g., Amazon) PE GSP (e.g., IBM) CPU or PE Grid Service (GS) (Globus) Aneka EC2 GTS Resource Allocation Job 8 Grid Resource Broker 2 Visual Application Composer Application Code Explore data Results 97 Results+ Cost Info Bill 12 Data Catalogue

24 24 On Demand Assembly of Services in Market-Oriented Grid Environments

25 25 Cloud Services Infrastructure as a Services (IaaS) CPU, Storage: Amazon.com et. al Platform as a Services (PaaS) Google App Engine, Microsoft Azure, and Manjrasoft Aneka Software as a Service (SaaS) SalesForce.Com Clouds Enterprise/Private Clouds Public/Internet Clouds

26 26 Layered view of services within a Cloud stack Infrastructure as a Service (IaaS) e.g., Amazon, Nirvanix Software as a Service (SaaS) e.g.,..SalesForce.com Platform as a Service (PaaS) e.g.,..Aneka

27 Aneka A Software Platform for Building and Managing Enterprise Grids and Clouds

28 28 Aneka: A 3 rd Gen enterprise Grid Technology Cloud model GenerationPropertiesTechnologies First (-2001) Application-specific platforms distributed.net, Second ( ) Single programming model, rigid architecture, no QoS UD, XtremeWeb, Alchemi, Digipede, DataSynapse, BOINC Third ( ?) SOA, extensible architecture, multiple programming models, multi-tenancy, enterprise QoS, SLAs, market-based resource allocation Aneka

29 29 ANEKA – Product Overview (Alpha).NET based service-oriented platform for grid / cloud computing Development and Run Time Environment Includes Development and Management Tools Suitable for Development of Enterprise Grid / Cloud Applications Grid / Cloud enabling legacy applications Ideal for Corporate Developers, Software, SaaS, Hosting Vendors and Application / System Integrators ANEKA Product Architecture

30 30 Aneka Deployment Models Enterprise/Private Harness LAN connected resources Application Development, Testing, Execution Teaching and Learning Sensitive applications Public Hosted by a 3 rd party service provider owning a large Data Center (1000s of servers) Offers subscription-based services to their shared infrastructure on pay-as go model.to many users from different organisations. Amazon.com, Microsoft Azure Aneka SDK + Execution Manger Aneka Enterprise/Private Clouds Public Clouds

31 31 Aneka: components public DumbTask: ITask { … public void Execute() { …… } for(int i=0; i

32 32 How does it solve the problem? An Illustratioin Application Manager Executor Manager / Executor GThreads/Tasks Divide the problem in to multiple small tasks and distribute them run in parallel on multiple computers within a Cloud.

33 33 User scenario: GoFront (unit of China Southern Railway Group) Aneka utilizes idle desktops (30) to decrease task time from days to hours Time (in hrs) Single Server Aneka Cloud Raw Locomotive Design Files (Using AutoDesk Maya) Using Maya Graphical Mode Directly Case 1: Single Server 4 cores server Aneka Maya Renderer Use private Aneka Cloud GoFront Private Aneka Cloud LAN network (Running Maya Batch Mode on demand) Case 2: Aneka Enterprise Cloud Application: Locomotive design CAD rendering

34 34 Aneka: How can get it? Available to Download: Software: Manual: Setting up Cloud using your LAN-network computers Teaching material parallel and distributed computing and programming, List of possible assignments for students Possible Projects for Final year students.. Price – highly affordable = Fee you charge to 1 student (each year) and all students/teachers in entire college/university can use it! Applications Other Departments (Physics, Chemistry, Biology, Finance, Engineering) can use it for their applications.

35 35 On Demand Assembly of Services in Market-Oriented Grid Environments

36 36 Agenda Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services Global Grids and Challenges Security, resource management, pricing models, … Service-Oriented Grid Architecture and Gridbus Solutions Market-based Management, GMD, Grid Bank, Aneka Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Summary and Conclusion

37 37 A resource broker for scheduling task farming data Grid applications with static or dynamic parameter sweeps on global Grids. It uses computational economy paradigm for optimal selection of computational and data services depending on their quality, cost, and availability, and users QoS requirements (deadline, budget, & T/C optimisation) Key Features A single window to manage & control experiment Programmable Task Farming Engine Resource Discovery and Resource Trading Optimal Data Source Discovery Scheduling & Predications Generic Dispatcher & Grid Agents Transportation of data & sharing of results Accounting Grid Service Broker (GSB)

38 38 Core Middleware Gridbus User Console/Portal/Application Interface Grid Info Server Schedule Advisor Trading Manager Gridbus Farming Engine Record Keeper Grid Explorer GE GIS, NWS TM TS RM & TS Grid Dispatcher G G C U Globus enabled node. A L Data Catalog Data Node Amazon EC2/S3 Cloud. $ $ $ App, T, $, Optimization Preference workload Gridbus Broker

39 39 Gridbus Broker: Separating applications from different remote service access enablers and schedulers Aneka AMI Amazon EC2Data Store Access Technology Grid FTP SRB -PBS -Condor -SGE Globus Job manager fork()batch() Gridbus agent Data Catalog -PBS -Condor -SGE -XGrid SSH fork() batch() Gridbus agent Single-sign on security Home Node/Portal Gridbus Broker fork() batch() -PBS -Condor -SGE -Aneka -XGrid Application Development Interface Scheduling Interfaces Alogorithm1 AlogorithmN Plugin Actuators

40 40 Gridbus Services for eScience applications Application Development Environment: XML-based language for composition of task farming (legacy) applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e.g., Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow engine. … Grid Superscalar – in cooperation with BSC/UPC Resource Allocation and Scheduling Dynamic discovery of optional computational and data nodes that meet user QoS requirements. Hide L ow-Level Grid Middleware interfaces Globus (v2, v4), SRB, Aneka, Unicore, and ssh-based access to local/remote resources managed by XGrid, PBS, Condor, SGE.

41 41 Drug Design Made Easy! Click Here for Demo

42 42 s

43 43 Agenda Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services Global Grids and Challenges Security, resource management, pricing models, … Service-Oriented Grid Architecture and Gridbus Solutions Market-based Management, GMD, Grid Bank, Aneka Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Summary and Conclusion

44 44 Case Study: High Energy Physics and Data Grid The Belle Experiment KEK B-Factory, Japan Investigating fundamental violation of symmetry in nature (Charge Parity) which may help explainwhy do we have more antimatter in the universe OR imbalance of matter and antimatter in the universe?. Collaboration 1000 people, 50 institutes 100s TB data currently

45 45 Case Study: Event Simulation and Analysis B0->D*+D*-Ks Simulation and Analysis Package - Belle Analysis Software Framework (BASF) Experiment in 2 parts – Generation of Simulated Data and Analysis of the distributed data Analyzed 100 data files (30MB each) that were distributed among the five nodes within Australian Belle DataGrid platform.

46 46 Australian Belle Data Grid Testbed VPAC Melbourne

47 47 Belle Data Grid (GSP CPU Service Price: G$/sec) NA G$4 Data node G$6 VPAC Melbourne G$2

48 48 Belle Data Grid (Bandwidth Price: G$/MB) NA G$4 Data node G$6 VPAC Melbourne G$

49 49 Deploying Application Scenario A data grid scenario with 100 jobs and each accessing remote data of ~30MB Deadline: 3hrs. Budget: G$ 60K Scheduling Optimisation Scenario: Minimise Time Minimise Cost Results:

50 50 Time Minimization in Data Grids Time (in mins.) Number of jobs completed fleagle.ph.unimelb.edu.aubelle.anu.edu.aubelle.physics.usyd.edu.aubrecca-2.vpac.org

51 51 Results : Cost Minimization in Data Grids Time(in mins.) Number of jobs completed fleagle.ph.unimelb.edu.aubelle.anu.edu.aubelle.physics.usyd.edu.aubrecca-2.vpac.org

52 52 Observation Organization Node detailsCost (in G$/CPU-sec)Total Jobs Executed TimeCost CS,UniMelbbelle.cs.mu.oz.au 4 CPU, 2GB RAM, 40 GB HD, Linux N.A. (Not used as a compute resource) -- Physics, UniMelbfleagle.ph.unimelb.edu.au 1 CPU, 512 MB RAM, 40 GB HD, Linux CS, University of Adelaide belle.cs.adelaide.edu.au 4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux N.A. (Not used as a compute resource) -- ANU, Canberrabelle.anu.edu.au 4 CPU, 2GB RAM, 40 GB HD, Linux 42 2 Dept of Physics, USyd belle.physics.usyd.edu.au 4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux VPAC, Melbournebrecca-2.vpac.org 180 node cluster (only head node used), Linux 623 2

53 53 Agenda Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services Global Grids and Challenges Security, resource management, pricing models, … Service-Oriented Grid Architecture and Gridbus Solutions Market-based Management, GMD, Grid Bank, Aneka Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Summary and Conclusion

54 54 Summary and Conclusion Grids exploit synergies that result from cooperation of autonomous entities: Resource sharing, dynamic provisioning, and aggregation at global level Great Science and Great Business! Grids have emerged as enabler for Cyberinfrastructure that powers e-Science and e-Business applications. SOA + Market-based Grid Management = Utility Grids Grids allow users to dynamically lease Grid services at runtime based on their quality, cost, availability, and users QoS requirements. Delivering ICT services as computing utilities. Clouds are rapidly emerging, but more work is required Federation of Clouds, Cloud Exchange, and Application Scaling

55 55 Convergence of Competing Paradigms/Communities Needed Web Data Centres Utility Computing Service Computing Grid Computing P2P Computing Cloud Computing Market-Oriented Computing … Ubiquitous access Reliability Scalability Autonomic Dynamic discovery Composability QoS SLA … } + Paradigms Attributes/Capabilities ? -Trillion $ business - Who will own it?

56 56 Thanks for your attention! Are there any Questions? Comments/ Suggestions We Welcome Cooperation in R&D and Business! | |


Download ppt "Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer."

Similar presentations


Ads by Google