1. Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Laboratory Dept. of Computer Science and Software Engineering The University of Melbourne, Australia www.gridbus.org www.gridbus.org Gridbus Sponsors

2. 2 GRIDS Lab @ Melbourne Youngest and one of the rapidly growing research labs in our School/University: Founded in 2002 Houses: Research Fellows/PostDocs Research Programmers PhD candidates Honours/Masters students Funding National and International organizations Australian Research Council & DEST Many industries (Sun, StorageTek, Microsoft, IBM, Microsoft) University-wide collaboration: Faculties of Science, Engineering, and Medicine Many national and international collaborations. Academics Industries Software: Widely in academic and industrial users. Publication: My research team produces over 20% of our Dept’s research output. EducationR & D + Community Services: e.g., IEEE TC for Scalable Computing

3. 3 (some) Books: Co-authored & edited

4. 4 Agenda Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services Global Grids and Challenges Security, resource management, pricing models, … Service-Oriented Grid Architecture and Gridbus Solutions Market-based Management, GMD, Grid Bank, Alchemi Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Summary and Conclusion

5. 5 4 Essential Utilities and Delivery Networks (1) Water Water Distribution Network (4) Telephone Telecom Networks (3) Gas (2) Electricity Power Grid

6. 6 (5) Computing Grid: Delivering IT services as the 5 th utility (Power Grid inspiration) eScience eBusiness eGovernment eHealth Multilingual eEducation …

7. 7 Power Grid Inspiration: Seamlessly delivering electricity as a utility to users

8. 8 Grid-like Vision In 1969, Leonard Kleinrock, one of the chief scientists of the original ARPA project which seeded the Internet, wrote: "As of now, computer networks are still in their infancy, but as they grow up and become sophisticated, we will probably see the spread of "computer utilities", which, like present electric and telephone utilities, will service individual homes and offices across the country“ Despite major advances in hardware and software systems over the past 35 years, we are yet to realize this vision. How far are we still from delivering computing as a utility? Let us look into the ICT evolution and project the future.

9. 9 Why Grid Computing Now? Let us look at the Evolution of ICT

10. 10 Computing and Communication Technologies Evolution: 1960-2010! * Sputnik 19601970197519801985199019952000 * ARPANET * Email * Ethernet * TCP/IP * IETF * Internet Era* WWW Era * Mosaic * XML *PC Clusters *Crays *MPPs *Mainframes * HTML * W3C *P2P *Grids *XEROX PARC worm COMPUTING Communication * Web Services * Minicomputers *PCs *WS Clusters *PDAs *Workstations *HTC 2010 * e-Science * Computing as Utility * e-Business * SocialNet Control Centralised Decentralised

11. 11 2100 Personal DeviceSMPs or SuperComputers Local Cluster Global Grid S E R V I C E S + P E R F O R M A N C E Inter Planet Grid Individual Group Department Campus State National Globe Inter Planet Universe Administrative Barriers Enterprise Cluster/Grid Computing is Scaling: Towards Inter-Planetary Level

12. 12 What is Grid? (It means different things to different people) IBM On Demand Computing Microsoft.NET Oracle 10g Sun N1 – Sun Grid Engine HP Adaptive Enterprise United Devices and related companies: Harvesting Unused Desktop resources

13. 13 What is Grid? [Buyya et. al.] A type of parallel and distributed system that enables the sharing, exchange, selection, & aggregation of geographically distributed “autonomous” resources: Computers – PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc; Software – e.g., ASPs renting expensive special purpose applications on demand; Catalogued data and databases – e.g. transparent access to human genome database; Special devices/instruments – e.g., radio telescope – SETI@Home searching for life in galaxy. People/collaborators. depending on their availability, capability, cost, and user QoS requirements. Wide area

14. 14 How does Grids look like? A Bird Eye View of a Global Grid Grid Resource Broker Resource Broker Application Grid Information Service Grid Resource Broker database R2R2 R3R3 RNRN R1R1 R4R4 R5R5 R6R6 Grid Information Service

15. 15 Classes of Grid Services / Types of Grids Computational Services – CPU cycles Pooling computing power: SETI@Home, TeraGrid, AusGrid, ChinaGrid, IndiaGrid, UK Grid,… Data Services Collaborative data sharing generated by instruments, sensors, persons: LHC Grid, Napster Application Services Access to remote software/libraries and license management—NetSolve Interaction Services eLearning, Virtual Tables, Group Communication (Access Grid), Gaming Knowledge Services The way knowledge is acquired, processed and managed—data mining. Utility Computing Services Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities. Computational Grid Data Grid ASP Grid Interaction Grid Knowledge Grid Utility Grid infrastructure Users

16. 16 How Are Grids Used? High-performance computing Collaborative data-sharing Collaborative design Drug discovery Financial modeling Data center automation High-energy physics Life sciences E-Business E-Science Natural language processing & Data Mining Utility computing

17. 17 e-Science Environment: Supporting Collaborative Science Distributed instruments Distributed computation Distributed data Peers sharing ideas and collaborative interpretation of data/results 2100 2100 2100 Remote Visualization Data & Compute Service Cyberinfrastructure E-Scientist

18. 18 Agenda Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services Global Grids and Challenges Security, resource management, pricing models, … Service-Oriented Grid Architecture and Gridbus Solutions Market-based Management, GMD, Grid Bank, Alchemi Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Summary and Conclusion

19. 19 Grid Challenges Security Resource Allocation & Scheduling Data locality Network Management System Management Resource Discovery Uniform Access Computational Economy Application Construction

20. 20 Some Grid Initiatives Worldwide Australia Nimrod-G Gridbus DISCWorld GrangeNet. APACGrid ARC eResearch Brazil OurGrid, EasyGrid LNCC-Grid + many others China ChinaGrid – Education CNGrid - application Europe UK eScience EU Grids.. and many more... India Garuda  Japan NAGERI Korea... N*Grid Singapore NGP USA Globus GridSec AccessGrid TeraGrid Cyberinfrasture and many more... Industry Initiatives IBM On Demand Computing HP Adaptive Computing Sun N1 Microsoft -.NET Oracle 10g Infosys – Enterprise Grid Satyam – Business Grid StorageTek –Grid.. and many more Public Forums Global Grid Forum Australian Grid Forum Conferences: CCGrid Grid HPDC E-Science http://www.gridcomputing.com 1.3 billion – 3 yrs 1 billion – 5 yrs 450million – 5 yrs 486million – 5 yrs 1.3 billion (Rs) 27 million 2? billion 120million – 5 yrs

21. 21 Open-Source Grid Middleware Projects

22. 22 The Gridbus Project @ Melbourne: Enable Leasing of ICT Services on Demand WWG Pushes Grid computing into mainstream computing Gridbus

23. 23

24. 24 Agenda Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services Global Grids and Challenges Security, resource management, pricing models, … Service-Oriented Grid Architecture and Gridbus Solutions Market-based Management, GMD, Grid Bank, Alchemi Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Summary and Conclusion

25. 25 What do Grid players want? Grid Consumers Execute jobs for solving varying problem size and complexity Benefit by utilizing distributed resources wisely Tradeoff timeframe and cost Strategy: minimise expenses Grid Providers Contribute resources for executing consumer jobs Benefit by maximizing resource utilisation Tradeoff local requirements & market opportunity Strategy: maximise return on investment

26. 26 What do Grid players require? They need tools and technologies that help them in value expression, value translation, and value enforcement. Grid Service Consumers (GSCs): How do I express QoS requirements ? How do I trade between timeframe & cost ? How do I map jobs to resources to meet my QoS needs? How do I manage Grid dynamics and get my work done? … Grid Service Providers (GSPs) How do I decide service pricing models ? How do I specify them ? How do I translate them into resource allocations ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments? …

27. 27 Principle 1: Service Oriented Architecture (SOA) A SOA is a contractual architecture for offering and consuming software as services. There are four entities that make up an SOA service provider, service registry, and service consumer (also known as service requestor). The functions or tasks that the service provider offers, along with other functional and technical information required for consumption, are defined in the service definition or contract. provider registry consumer contract

28. 28 Principle 2: Market-Oriented (Grid) Computing- (a) Sustained Resourced Sharing and (b) Effective Management of Shared Resources Grid Economy

29. 29 Market-based Systems = Self-managed and Self-regulated systems. Complexity present in Grid systems is similar to one present in human economies.

30. 30 Grid Node N Service-Oriented Grid Architecture Grid Service Consumer Programming Environments Grid Resource Broker Grid Service Providers Grid Explorer Schedule Advisor Trade Manager Job Control Agent Deployment Agent Trade Server Resource Allocation Resource Reservation R1R1 Misc. services Information Service R2R2 RmRm … Pricing Algorithms Accounting Grid Node1 … Core Middleware Services … … Health Monitor Grid Market Services JobExec Info ? Secure Trading QoS Storage Sign-on Grid Bank Applications Data Catalogue

31. 31 Gridbus and Complementary Technologies – realizing Utility Grid AIX Solaris WindowsLinux.NET Grid Fabric Software Grid Applications Core Grid Middleware User-Level Middleware Grid Bank Grid Exchange & Federation JVM Grid Brokers: X-Parameter Sweep Lang. Gridbus Data Broker MPI CondorSGETomcatPBS Alchemi Workflow IRIXOSF1 Mac Libra GlobusUnicore … … Grid Market Directory PDBCDB Worldwide Grid Grid Fabric Hardware … … PortalsScienceCommerceEngineering … … Collaboratories … … Workflow Engine Grid Storage Economy Grid Economy NorduGridXGrid ExcellGrid Nimrod-G Gridscape

32. 32 On Demand Assembly of Services: Putting Them All Together ASP Catalogue Grid Info Service Grid Market Directory GSP (Accounting Service) Gridbus GridBank GSP (e.g., UofM) PE GSP (e.g., VPAC) PE GSP (e.g., IBM) CPU or PE Grid Service (GS) (Globus) Alchemi GS GTS Cluster Scheduler Job 8 Grid Resource Broker 2 Visual Application Composer Application Code Explore data 1 36 45 Results 97 Results+ Cost Info 10 11 Bill 12 Data Catalogue

33. 33 On Demand Assembly of Services: Putting Them All Together ASP Catalogue Grid Info Service Grid Market Directory GSP (Accounting Service) Gridbus GridBank GSP (e.g., UofM) PE GSP (e.g., VPAC) PE GSP (e.g., IBM) CPU or PE Grid Service (GS) (Globus) Alchemi GS GTS Cluster Scheduler Job 8 Grid Resource Broker 2 Visual Application Composer Application Code Explore data 1 36 45 Results 97 Results+ Cost Info 10 11 Bill 12 Data Catalogue

34. 34 Alchemi:.NET-based Enterprise Grid Platform & Web Services Internet Alchemi Worker Agents Alchemi Manager Alchemi Users Web Services SETI@Home like Model General Purpose Dedicated/Non-dedicate workers Role-based Security.NET and Web Services C# Implementation GridThread and Job Model Programming Easy to setup and use Widely in use!

35. 35 Some Users of Alchemi Tier TechnologiesTier Technologies, USA Large scale document processing using Alchemi framework CSIROCSIRO, Australia Natural Resource Modeling The Friedrich Miescher Institute (FMI) for Biomedical ResearchThe Friedrich Miescher Institute (FMI) for Biomedical Research, Switzerland Patterns of transcription factors in mammalian genes Satyam Computers Applied Research LaboratorySatyam Computers Applied Research Laboratory, India Micro-array data processing using Alchemi framework The University of Sao PauloThe University of Sao Paulo, Brazil The Alchemi Executor as a Windows Service stochastix GmbHstochastix GmbH, Germany Serving clients in International Banking/Finance sector Many users in Universities: See next for an example.

36. 36 Students' project gives old computers new life - 1/25/2005

37. 37 On Demand Assembly of Services: Putting Them All Together ASP Catalogue Grid Info Service Grid Market Directory GSP (Accounting Service) Gridbus GridBank GSP (e.g., UofM) PE GSP (e.g., VPAC) PE GSP (e.g., IBM) CPU or PE Grid Service (GS) (Globus) Alchemi GS GTS Cluster Scheduler Job 8 Grid Resource Broker 2 Visual Application Composer Application Code Explore data 1 36 45 Results 97 Results+ Cost Info 10 11 Bill 12 Data Catalogue

38. 38 Agenda Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services Global Grids and Challenges Security, resource management, pricing models, … Service-Oriented Grid Architecture and Gridbus Solutions Market-based Management, GMD, Grid Bank, Alchemi Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Summary and Conclusion

39. 39 A resource broker for scheduling task farming data Grid applications with static or dynamic parameter sweeps on global Grids. It uses computational economy paradigm for optimal selection of computational and data services depending on their quality, cost, and availability, and users’ QoS requirements (deadline, budget, & T/C optimisation) Key Features A single window to manage & control experiment Programmable Task Farming Engine Resource Discovery and Resource Trading Optimal Data Source Discovery Scheduling & Predications Generic Dispatcher & Grid Agents Transportation of data & sharing of results Accounting Grid Service Broker (GSB)

40. 40 Gridbus Broker Architecture Grid Middleware Gridbus Client Gribus Client Grid Info Server Schedule Advisor Trading Manager Gridbus Farming Engine Record Keeper Grid Explorer GE GIS, NWS TM TS RM & TS Grid Dispatcher RM: Local Resource Manager, TS: Trade Server G G C U Globus enabled node. A L Alchemi enabled node. (Data Grid Scheduler) Data Catalog Data Node Unicore enabled node. $ $ $ App, T, $, Opt (Bag of Tasks Applications)

41. 41 Gridbus Broker: Separating “applications” from “different” remote service access enablers and schedulers Alchemi Gateway UnicoreData Store Access Technology Grid FTP SRB -PBS -Condor -SGE Globus Job manager fork()batch() Gridbus agent Data Catalog -PBS -Condor -SGE -XGrid SSH fork() batch() Gridbus agent Single-sign on security Home Node/Portal Gridbus Broker fork() batch() -PBS -Condor -SGE -Alchemi -XGrid Application Development Interface Scheduling Interfaces Alogorithm1 AlogorithmN Plugin Actuators

42. 42 Gridbus Services for eScience applications Application Development Environment: XML-based language for composition of task farming (legacy) applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e.g., Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow engine. Resource Allocation and Scheduling Dynamic discovery of optional computational and data nodes that meet user QoS requirements. Hide L ow-Level Grid Middleware interfaces Globus (v2, v4), SRB, Alchemi, Unicore, and ssh-based access to local/remote resources managed by XGrid, Condor, SGE.

43. 43 Figure 3 : Logging into the portal. Drug Design Made Easy! Click Here for Demo

44. 44 Excel Plugin to Access Gridbus Services Excel ExcelGrid Add-In ExcelGrid Runner ExcelGridJob ExcelGrid MiddlewareGridbus BrokerEnterprise Grid 210 0

45. 45 Discover Resources Distribute Jobs Establish Rates Meet requirements ? Remaining Jobs, Deadline, & Budget ? Evaluate & Reschedule Discover More Resources Compose & Schedule Adaptive Scheduling Steps

46. 46 Deadline (D) and Budget (B) Constrained Scheduling Algorithms AlgorithmExecution Time (D) Execution Cost (B) Compute Grid Data Grid Cost OptLimited by DMinimize Yes Cost-Time OptMinimize if possible Minimize Yes Time OptMinimizeLimited by B Yes Conservative- Time Opt MinimizeLimited by B, jobs have guaranteed minimum budget Yes

47. 47

48. 48 Agenda Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services Global Grids and Challenges Security, resource management, pricing models, … Service-Oriented Grid Architecture and Gridbus Solutions Market-based Management, GMD, Grid Bank, Alchemi Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Summary and Conclusion

49. 49 Case Study: High Energy Physics and Data Grid The Belle Experiment KEK B-Factory, Japan Investigating fundamental violation of symmetry in nature (Charge Parity) which may help explain “why do we have more antimatter in the universe OR imbalance of matter and antimatter in the universe?”. Collaboration 1000 people, 50 institutes 100’s TB data currently

50. 50 Case Study: Event Simulation and Analysis B0->D*+D*-Ks Simulation and Analysis Package - Belle Analysis Software Framework (BASF) Experiment in 2 parts – Generation of Simulated Data and Analysis of the distributed data  Analyzed 100 data files (30MB each) that were distributed among the five nodes within Australian Belle DataGrid platform.

51. 51 Australian Belle Data Grid Testbed VPAC Melbourne

52. 52 Belle Data Grid (GSP CPU Service Price: G$/sec) NA G$4 Data node G$6 VPAC Melbourne G$2

53. 53 Belle Data Grid (Bandwidth Price: G$/MB) NA G$4 Data node G$6 VPAC Melbourne G$2 34 31 38 31 30 33 36 32

54. 54 Deploying Application Scenario A data grid scenario with 100 jobs and each accessing remote data of ~30MB Deadline: 3hrs. Budget: G$ 60K Scheduling Optimisation Scenario: Minimise Time Minimise Cost Results:

55. 55 Observation Organization Node detailsCost (in G$/CPU-sec)Total Jobs Executed TimeCost CS,UniMelbbelle.cs.mu.oz.au 4 CPU, 2GB RAM, 40 GB HD, Linux N.A. (Not used as a compute resource) -- Physics, UniMelbfleagle.ph.unimelb.edu.au 1 CPU, 512 MB RAM, 40 GB HD, Linux 23 94 CS, University of Adelaide belle.cs.adelaide.edu.au 4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux N.A. (Not used as a compute resource) -- ANU, Canberrabelle.anu.edu.au 4 CPU, 2GB RAM, 40 GB HD, Linux 42 2 Dept of Physics, USyd belle.physics.usyd.edu.au 4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux 472 2 VPAC, Melbournebrecca-2.vpac.org 180 node cluster (only head node used), Linux 623 2

56. 56 Agenda Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services Global Grids and Challenges Security, resource management, pricing models, … Service-Oriented Grid Architecture and Gridbus Solutions Market-based Management, GMD, Grid Bank, Alchemi Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Summary and Conclusion

57. Alessandro Volta in Paris in 1801 inside French National Institute shows the battery while in the presence of Napoleon I Fresco by N. Cianfanelli (1841) (Zoological Section "La Specula" of National History Museum of Florence University)

58. 58 ….and in the future, I imagine a Worldwide Power (Electrical) Grid …... What ?!?! This is a mad man… Oh, mon Dieu !

59. 59 2006 - 1801 = 205 Years (Recent R&D: Delivering Internet services via Electric cables).

60. 60 When are we going to get IT as the 5 th utility? (water, electricity, gas, telephone, IT) eScience eBusiness eGovernment eHealth Multilingual eEducation …

61. 61 Summary and Conclusion Grids exploit synergies that result from cooperation of autonomous entities: Resource sharing, dynamic provisioning, and aggregation at global level  Great Science and Great Business! Grids have emerged as enabler for Cyberinfrastructure that powers e-Science and e-Business applications. SOA + Market-based Grid Management = Utility Grids Grids allow users to dynamically lease Grid services at runtime based on their quality, cost, availability, and users QoS requirements. Delivering ICT services as computing utilities. Grids offer enormous opportunities for realizing e-Science and e-Business at global level.

62. 62 Thanks for your attention! We Welcome Cooperation in Research and Development! http:/www.gridbus.org

63. Gridbus Middleware: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Laboratory Dept. of Computer Science and Software Engineering The University of Melbourne, Australia www.gridbus.org www.gridbus.org Gridbus Sponsors

64. Backup Slides

65. 65 Some Characteristics of Grids Numerous resources Different security requirements & policies Resources are heterogeneous Geographically distributed Different resource management policies Connected by heterogeneous, multi-level networks Owned by multiple organizations & individuals Unreliable resources and environments Slide by Hiro

66. 66 What are Grid benefits? Resource sharing across multiple administrative boundaries Effective utilisation of the (existing) resources Dynamic provisioning Application Acceleration Scalability Reliability Virtualisation: applications, services, resources,…

67. 67 Time Minimization in Data Grids 0 10 20 30 40 50 60 70 80 123456789101112131415161718192021222324252627282930313233343536373839404142 Time (in mins.) Number of jobs completed fleagle.ph.unimelb.edu.aubelle.anu.edu.aubelle.physics.usyd.edu.aubrecca-2.vpac.org

68. 68 Results : Cost Minimization in Data Grids 0 10 20 30 40 50 60 70 80 90 100 13579111315171921232527293133353739414345474951535557596163 Time(in mins.) Number of jobs completed fleagle.ph.unimelb.edu.aubelle.anu.edu.aubelle.physics.usyd.edu.aubrecca-2.vpac.org

69. Utility Grids: Autonomic and Market-Oriented Global Grids for Powering e-Science and e-Business Applications Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Laboratory Dept. of Computer Science and Software Engineering The University of Melbourne, Australia www.gridbus.org www.gridbus.org Gridbus Sponsors