Grid Business Symposium 2005, Seoul, Korea Recent Advances in Grid Computing and Business Models: A Gridbus Perspective WW Grid Grid Business Symposium 2005, Seoul, Korea Rajkumar Buyya Grid and Distributed Systems (GRIDS) Laboratory Dept. of Computer Science and Software Engineering The University of Melbourne Melbourne, Australia www.gridbus.org

Outline Introduction Global Grids and Challenges Grid Initiatives Utility Networks and Grid Computing Global Grids and Challenges Grid Initiatives World-wide with Australia and India Perspective Introduction to Gridbus Project and Grid Economy Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Economy-based Scheduling in Data Grids Summary

4 Essential Utilities and Delivery Networks (1) Water (2) Electricity (3) Gas (4) Telephone

(5) IT services as the fifth utility (water, electricity, gas, telephone, IT) eScience eBusiness eGovernment eHealth Multilingual eEducation …

A Bird Eye View of World-Wide Grid Environment Grid Information Service Grid Resource Broker database Application R2 R3 R4 R5 RN Grid Resource Broker R6 R1 Resource Broker Grid Information Service

Grid Resource Sharing & Synergies Computational Services – CPU cycles SETI@Home, NASA IPG, TeraGrid, I-Grid,… Data Services Data replication, management, secure access--LHC Grid/Napster Application Services Access to remote software/libraries and license management—NetSolve Interaction Services eLearning, Virtual Tables, Group Communication (Access Grid), Gaming Knowledge Services The way knowledge is acquired and managed—data mining. Utility Computing Services Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities. Utility Grid Knowledge Grid Interaction Grid ASP Grid Data Grid Computational Grid

Grid Challenges Application Construction Security Uniform Access Security System Management Computational Economy Resource Discovery Resource Allocation & Scheduling Data locality Network Management Application Construction

Some Grid Initiatives Worldwide Australia Nimrod-G Gridbus DISCWorld GrangeNet. APACGrid ARC eResearch Brazil OurGrid, EasyGrid LNCC-Grid + many others China ChinaGrid – Education CNGrid - application Europe UK eScience EU Grids.. and many more... India I-Grid Japan NAGERI Korea... N*Grid Singapore NGP USA Globus NASA IPG AccessGrid TeraGrid Cyberinfrasture Industry Initiatives IBM On Demand Computing HP Adaptive Computing Sun N1 Microsoft - .NET Oracle 10g Satyam – Grid Practice Infosys, Wipro, TCS StorageTek –Grid.. Public Forums Global Grid Forum Australian Grid Forum Conferences: CCGrid Grid HPDC E-Science 27 million 1.3 billion – 3 yrs 2? billion 120million – 5 yrs 450million – 5 yrs 486million – 5 yrs 1.3 billion (Rs) 1 billion – 5 yrs http://www.gridcomputing.com

Grid (Market) Participant Types and Application Category free trading Public computing (SETI@Home, Alchemi, UD) Private enterprises (IBM, Sun, Satyam) Sharing Model National provider (K*Grid, TeraGrid, Garuda/IndiaGrid, UKGrid) regulation scientific commercial Application Category

Grid Computing in Australia (Courtesy: Jihyoun Park, SNU Visitor to Melbourne) Academia Government Collaboration Industry

Academic activities 1 University laboratories for Grid computing - Uni. of Melbourne(GRIDS lab): Gridbus (GridSim, GMD, GridBank, Alchemi, ..), Master of Engineering in Distributed Computing - Monash Uni.: GriddlsS (Legacy SW to the computational grid), Nimrod-G - Australian national Uni. (Internet Futures Group) - Sydney Uni.(ViSLAB): high performance visualization &computing - Uni. of Adelaide (DHPC Group): DISCWorld - Queensland Uni. of Technology (PLAS): G2 (.NET based) 2 Grid Infrastructure Projects APACGrid, National Neurosciece Facility, Australian Virtual Observatory, several state level facilities (VPAC, TPAC, SAPAC, QPSF, IVEC) 3 Grid Applications * Asia Pacific Bioinformatics Network/ Virtual Drug Design: Molecular Modeling for Drug Design on P2P Grid/ HEPGrid: High Energy Physics and the Grid Network/ Access Grid/Australian Computational Earth Systems Simulator/. * Recently 30 more applications are funded as part of ARC e-Research * Govt. has formed “National e-Research Coordination Committee”.

Grid Computing in India Academia Government Collaboration Industry (majority focus on Grid integration)

Grid Computing in India: Academic and Industrial Activities Academic and Government Initiatives: TIFR, IITM, Anna University, IITD, UoH, etc. C-DAC’s Garuda – Ministry of IT Software Companies in India: Top 4 Indian IT Companies: Satyam, Infosys, TCS (Tata Consultancy Service), and Wipro. Oracle 10g, IBM, HP, Sun ertc. have a large Grid development centers in Bangalore, India. Satyam is leading the pack in Grid Business push: Grid Practice Centre with top management support. Singned MoU with Melbourne University and extensively using Gridbus in powering applications. Also contributing the development of Gridbus technologies (e.g., Alchemi) – SEI CMM Level 5 principles. Application Verticals: Manufacturing, Security, Life Sciences, Finance

Australian and Indian Grid Efforts Compared Korea: Is it like Australia or India?

On Demand Utility Computing The Gridbus Project @ Melbourne: Enable Leasing of ICT Services on Demand Distributed Data WWG Gridbus World Wide Grid! On Demand Utility Computing

The Gridbus Project: http://www.gridbus.org A multi-institutional “Open Source” R&D Project with focus on: Architecture, Specification, and Open Source Reference Implementation. Service-Oriented Grid, Utility Computing & Distributed Data and Computation Economy Scaling from Desktops, Clusters, Cluster Federation, Enterprise Grids to Global Grids. Alchemi: Harnessing .NET/Windows-based Resources Grid Market Directory and Web Services Grid Bank: Accounting and Transaction Management Visual Tools for Creation of Distributed Applications Workflow Composition and Deployment Services Data Grid Brokering and Grid Economy Services Data Replication Strategies GridSim Toolkit: Enhanced to support Data Grid, Reservation, etc. Libra: SLA-based Allocation of Cluster Resources Coupling of Clusters and Computational Economy WWG: Global Data Intensive Grid Testbed Application Enabler Projects: High-Energy Physics , Astronomy, Brain Activity Analysis – Osaka U., Natural Language Processing, Portfolio Analysis – Spain, BioGrid - WEHI (via APACGrid), SensorGrid (NICTA), Medical Imaging (HFI) Supported by:

Grid Economy: Methodology for Sustained Resourced Sharing and Managing Supply-and-Demand for Resources For resource providers, they provide service with different goals For consumers, they have different requirments How to manage grid service supply-and-demand is hard task So how can consumers find a resource depending on their requirements. And what type of resource that providers provide and how can they serve consumers within certain QoS. For example, when many consumers want to access same resource, who should be served first How to manage supply-demand balance is another challenge.

New challenges of Grid Economy Resource Owners How do I decide prices ? (economic models?) How do I specify them ? How do I translate price to resource allocation ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments? ….. Resource Consumers How do I decide expenses ? How do I express QoS requirements ? How do I trade between timeframe & cost ? How do I map jobs to resources to meet my QoS needs? They need mechanisms and technologies for value expression, value translation, and value enforcement.

Grid Entities and Architecture Grid consumer GSP site scheduler Resource owners broker GSP global scheduler GSP site scheduler Resource owners Market Maker accounting End users Private enterprises National providers

A Reference Service-Oriented Architecture for Utility Grids Data Catalogue Grid Bank Grid Market Services Information Service Sign-on Health Monitor Info ? Grid Explorer Grid Node N … … Programming Environments Secure Job Control Agent Applications Schedule Advisor Grid Node1 QoS Pricing Algorithms Trade Server Trading Trade Manager Accounting Resource Reservation Misc. services … Deployment Agent JobExec Resource Allocation Grid Resource Broker Storage R1 R2 … Rm Grid Middleware Services Grid Consumer Grid Service Providers

Gridbus and Complementary Technologies – realizing Utility Grid Applications … Science Commerce Engineering Collaboratories Portals … MPI ExcellGrid Gridscape Workflow X-Parameter Sweep Lang. User-Level Middleware (Grid Tools) … Grid Brokers: Nimrod-G Workflow Engine Gridbus Data Broker Core Grid Middleware Globus Unicore Grid Storage Economy Grid Exchange & Federation Grid Market Directory … Grid Bank Alchemi NorduGrid XGrid GRIDSIM Grid Economy .NET JVM Condor PBS SGE Libra Tomcat Grid Fabric Software Windows Solaris Linux AIX IRIX OSF1 Mac CDB Grid Fabric Hardware PDB Worldwide Grid

Alchemi: .NET-based Enterprise Grid Platform & Web Services Alchemi Manager Web Services Internet Alchemi Users Internet SETI@Home like Model General Purpose Dedicated/Non-dedicate workers Role-based Security .NET and Web Services C# Implementation GridThread and Job Model Programming Easy to setup and use Widely in use! Alchemi Worker Agents

Many users in Universities: See next for an example. Some Users of Alchemi Tier Technologies, USA Large scale document processing using Alchemi framework Satyam Computers Applied Research Laboratory, India Micro-array data processing using Alchemi framework CSIRO, Australia Natural Resource Modeling The University of Sao Paulo, Brazil The Alchemi Executor as a Windows Service stochastix GmbH, Germany Asynchronous Excel Tasks using ManagedXLL and Alchemi .Net Grid Computing framework. The Friedrich Miescher Institute (FMI) for Biomedical Research, Switzerland Patterns of transcription factors in mammalian genes Many users in Universities: See next for an example.

On Demand Assembly of Services: Putting Them All Together Visual Application Composer Application Code Explore data 1 Data Source Data Results+ Cost Info 10 Grid Resource Broker 2 Bill 12 (Instruments/distributed sources) Data Replicator (GDMP) Data Catalogue 5 ASP Catalogue Grid Info Service Grid Market Directory 4 6 3 Job 8 Results 9 7 GSP (e.g., UofM) PE (e.g., VPAC) (e.g., IBM) CPU or PE Grid Service (GS) (Globus) Alchemi GS GTS Cluster Scheduler GSP (Accounting Service) Gridbus GridBank 11 Cluster Scheduler PE Grid Service Provider (GSP) (e.g., CERN)

The Gridbus Grid Service Broker for Data Grid Applications Builds on the Nimrod-G Computational Grid Broker and Computational Economy [Buyya, Abramson, Giddy, Monash University, 1999-2001] And Extends its notion for Data and Service Grids

Gridbus Broker Architecture Gridbus Client Gribus Client Gridbus Client (Bag of Tasks Applications) App, T, $, Opt (Data Grid Scheduler) Gridbus Farming Engine Schedule Advisor Record Keeper Trading Manager Grid Dispatcher Grid Explorer Grid Middleware TM TS $ GE GIS, NWS Grid Info Server RM & TS G $ Data Catalog Data Node C $ U G Unicore enabled node. Globus enabled node. L A RM: Local Resource Manager, TS: Trade Server Alchemi enabled node.

Gridbus Broker and Remote Service Access Enablers Home Node/Portal Gridbus Broker fork() batch() -PBS -Condor -SGE -Alchemi -XGrid Credential Repository MyProxy Portlets Data Catalog Alchemi -PBS -Condor -SGE Globus Job manager fork() batch() Gridbus agent Data Store Gateway Unicore -PBS -Condor -SGE -XGrid SSH fork() batch() Gridbus agent Access Technology SRB Grid FTP

Gridbus Services for eScience applications Application Development Environment: XML-based language for composition of task farming (legacy) applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e.g., Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow engine. Resource Allocation and Scheduling Dynamic discovery of optional computational and data nodes that meet user QoS requirements. Hide Low-Level Grid Middleware interfaces Globus, Alchemi, Unicore, NorduGrid, XGrid, etc.

Figure 3 : Logging into the portal. Click Here for Demo Figure 3 : Logging into the portal. Drug Design Made Easy!

Economy-based Data Grid Scheduling CLICK HERE TO SKIP IF RUNNING OUT of TIME High Energy Physics as eScience Application Case Study

Australian Belle Data Grid Testbed

Case Study: Event Simulation and Analysis B0->D*+D*-Ks Simulation and Analysis Package - Belle Analysis Software Framework (BASF) Experiment in 2 parts – Generation of Simulated Data and Analysis of the distributed data Analyzed 100 data files (30MB each) were distributed among the five nodes

Resources Used and their Service Price Organization   Node details Role Cost (in G$/CPU-sec) CS,UniMelb belle.cs.mu.oz.au 4 CPU, 2GB RAM, 40 GB HD, Linux Broker host, Data host, NWS server N.A. (Not used as a compute resource) Physics, UniMelb fleagle.ph.unimelb.edu.au 1 CPU, 512 MB RAM, 40 GB HD, Linux Replica Catalog host, Data host, Compute resource, NWS sensor 2 CS, University of Adelaide belle.cs.adelaide.edu.au 4 CPU (only 1 available) , 2GB RAM, 40 GB HD, Linux Data host, NWS sensor ANU, Canberra belle.anu.edu.au 4 CPU, 2GB RAM, 40 GB HD, Linux Data host, Compute resource, NWS sensor 4 Dept of Physics, USyd belle.physics.usyd.edu.au 4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux VPAC, Melbourne brecca-2.vpac.org 180 node cluster (only head node used), Linux Compute resource, NWS sensor 6

Network Cost (in Grid $/Currency!)

Deploying Application Scenario A data grid scenario with 100 jobs and each accessing remote data of ~30MB Deadline: 3hrs. Budget: G$ 60K Scheduling Optimisation Scenario: Minimise Time Minimise Cost Results:

Grid and Gridbus Technologies for Various Grid (Market) Types free trading Public computing (Alchemi) Private enterprises (Libra, Gridbus, Globus) Sharing Model National provider (Globus, Gridbus,..) regulation scientific commercial Application Category

(5) IT services as the fifth utility (water, electricity, gas, telephone, IT) eScience eBusiness eGovernment eHealth Multilingual eEducation …

Summary and Conclusion Grids exploit synergies that result from cooperation of autonomous entities: Resource sharing, dynamic provisioning, and aggregation at global level. Grid Economy provides incentive needed for sustained cooperation. Grid Network has potential to serve as Cyberinfrastructure for Utility Computing Grids offer enormous opportunities for realizing eScience and eBusiness at global level.

Gridbus Project - http://www.gridbus.org Any Questions ? Gridbus Project - http://www.gridbus.org

Thanks for your attention! The Gridbus Cooperation! http://www.gridbus.com

Backup Slides

Some Open Research Challenges Value expression, translation, and enforcement mechanisms and supporting Grid technologies for: different economic models for spot markets and futures application models Dynamic Pricing Schemes Interaction Protocols for Service Negotiation Micro payments and Digital Currencies Scheduling Algorithms Programming Environments for Building Information Utility Applications Last, but not least: Dispute Managements and Legal Issues Taxation (consult your National Taxation Office) State, national, and international boundaries Tax returns!

This talk is designed to answer: How can Grid technologies support the emergence and operation of virtual enterprises? How can Grid shared resources be treated, brokered, and marketed as ICT ‘commodities’ or ‘futures’ among networked organisations? What kind of Grid architecture is needed for handling such market mechanisms in an automated fashion? How can Grid economies map the evolution of networked business models?

What do Grids aim for and how to support them. Grids aim at exploiting synergies that result from cooperation of autonomous distributed entities. Synergies include: Resource sharing “On-demand” Virtual Enterprises creation Aggregation of resources on demand. For this cooperation to be sustainable, participants needs to have (economic) incentive. Therefore, “incentive” mechanisms should be considered as one of key design parameters of Grid computing.

Grid Market (Participant) Types and Application Category free trading Public computing Private enterprises Sharing Model National provider regulation scientific commercial Application Category

Appropriate Market Model for different market types high Variable price auction Commodity market Willingness to Pay Posted price oligopoly low weak strong Demand elasticity

Realising Market-based Grid: Minimal New Components Grid Market Directory Services Grid Trading Services – for different economic models Grid Metering Services Grid Accounting and Payment Services Grid Service Broker

Deadline (D) and Budget (B) Constrained Scheduling Algorithms Execution Time (D) Execution Cost (B) Compute Grid Data Grid Cost Opt Limited by D Minimize Yes Cost-Time Opt Minimize if possible Time Opt Limited by B Conservative-Time Opt Limited by B, jobs have guaranteed minimum budget