1. Grid Business Symposium 2005, Seoul, Korea
Recent Advances in Grid Computing and Business Models: A Gridbus Perspective
WW Grid
Grid Business Symposium 2005, Seoul, Korea
Rajkumar Buyya
Grid and Distributed Systems (GRIDS) Laboratory Dept. of Computer Science and Software Engineering The University of Melbourne Melbourne, Australia

2. Outline Introduction Global Grids and Challenges Grid Initiatives
Utility Networks and Grid Computing
Global Grids and Challenges
Grid Initiatives
World-wide with Australia and India Perspective
Introduction to Gridbus Project and Grid Economy
Grid Service Broker
Architecture, Design and Implementation
Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids
A Case Study in High Energy Physics
Economy-based Scheduling in Data Grids
Summary

3. 4 Essential Utilities and Delivery Networks
(1) Water
(2) Electricity
(3) Gas
(4) Telephone

4. (5) IT services as the fifth utility (water, electricity, gas, telephone, IT)
eScience
eBusiness
eGovernment
eHealth
Multilingual
eEducation …

5. A Bird Eye View of World-Wide Grid Environment
Grid Information Service
Grid Resource Broker
database
Application R2 R3 R4 R5 RN
Grid Resource Broker R6 R1
Resource Broker
Grid Information Service

6. Grid Resource Sharing & Synergies
Computational Services – CPU cycles
NASA IPG, TeraGrid, I-Grid,…
Data Services
Data replication, management, secure access--LHC Grid/Napster
Application Services
Access to remote software/libraries and license management—NetSolve
Interaction Services
eLearning, Virtual Tables, Group Communication (Access Grid), Gaming
Knowledge Services
The way knowledge is acquired and managed—data mining.
Utility Computing Services
Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities.
Utility Grid
Knowledge Grid
Interaction Grid
ASP Grid
Data Grid
Computational Grid

7. Grid Challenges Application Construction Security
Uniform Access
Security
System Management
Computational Economy
Resource Discovery
Resource Allocation
& Scheduling
Data locality
Network Management
Application Construction

8. Some Grid Initiatives Worldwide
Australia
Nimrod-G
Gridbus
DISCWorld
GrangeNet.
APACGrid
ARC eResearch
Brazil
OurGrid, EasyGrid
LNCC-Grid + many others
China
ChinaGrid – Education
CNGrid - application
Europe
UK eScience
EU Grids..
and many more...
India
I-Grid
Japan
NAGERI
Korea...
N*Grid
Singapore
NGP
USA
Globus
NASA IPG
AccessGrid
TeraGrid
Cyberinfrasture
Industry Initiatives
IBM On Demand Computing
HP Adaptive Computing
Sun N1
Microsoft - .NET
Oracle 10g
Satyam – Grid Practice
Infosys, Wipro, TCS
StorageTek –Grid..
Public Forums
Global Grid Forum
Australian Grid Forum
Conferences:
CCGrid
Grid
HPDC
E-Science
27 million
1.3 billion – 3 yrs
2? billion
120million – 5 yrs
450million – 5 yrs
486million – 5 yrs
1.3 billion (Rs)
1 billion – 5 yrs

9. Grid (Market) Participant Types and Application Category
free trading
Public computing
Alchemi, UD)
Private enterprises (IBM, Sun, Satyam)
Sharing Model
National provider (K*Grid, TeraGrid, Garuda/IndiaGrid, UKGrid)
regulation
scientific
commercial
Application Category

10. Grid Computing in Australia (Courtesy: Jihyoun Park, SNU Visitor to Melbourne)
Academia
Government
Collaboration
Industry

11. Academic activities 1 University laboratories for Grid computing
- Uni. of Melbourne(GRIDS lab): Gridbus (GridSim, GMD, GridBank, Alchemi, ..), Master of Engineering in Distributed Computing
- Monash Uni.: GriddlsS (Legacy SW to the computational grid), Nimrod-G
- Australian national Uni. (Internet Futures Group)
- Sydney Uni.(ViSLAB): high performance visualization &computing
- Uni. of Adelaide (DHPC Group): DISCWorld
- Queensland Uni. of Technology (PLAS): G2 (.NET based)
2 Grid Infrastructure Projects
APACGrid, National Neurosciece Facility, Australian Virtual Observatory, several state level facilities (VPAC, TPAC, SAPAC, QPSF, IVEC)
3 Grid Applications
* Asia Pacific Bioinformatics Network/ Virtual Drug Design: Molecular Modeling for Drug Design on P2P Grid/ HEPGrid: High Energy Physics and the Grid Network/ Access Grid/Australian Computational Earth Systems Simulator/.
* Recently 30 more applications are funded as part of ARC e-Research
* Govt. has formed “National e-Research Coordination Committee”.

12. Grid Computing in India
Academia
Government
Collaboration
Industry
(majority focus on Grid integration)

13. Grid Computing in India: Academic and Industrial Activities
Academic and Government Initiatives:
TIFR, IITM, Anna University, IITD, UoH, etc.
C-DAC’s Garuda – Ministry of IT
Software Companies in India:
Top 4 Indian IT Companies: Satyam, Infosys, TCS (Tata Consultancy Service), and Wipro.
Oracle 10g, IBM, HP, Sun ertc. have a large Grid development centers in Bangalore, India.
Satyam is leading the pack in Grid Business push:
Grid Practice Centre with top management support.
Singned MoU with Melbourne University and extensively using Gridbus in powering applications.
Also contributing the development of Gridbus technologies (e.g., Alchemi) – SEI CMM Level 5 principles.
Application Verticals: Manufacturing, Security, Life Sciences, Finance

15. Australian and Indian Grid Efforts Compared
Korea: Is it like Australia or India?

16. On Demand Utility Computing
The Gridbus Melbourne: Enable Leasing of ICT Services on Demand
Distributed Data
WWG
Gridbus
World Wide Grid!
On Demand Utility Computing

17. The Gridbus Project: http://www.gridbus.org
A multi-institutional “Open Source” R&D Project with focus on:
Architecture, Specification, and Open Source Reference Implementation.
Service-Oriented Grid, Utility Computing & Distributed Data and Computation Economy
Scaling from Desktops, Clusters, Cluster Federation, Enterprise Grids to Global Grids.
Alchemi: Harnessing .NET/Windows-based Resources
Grid Market Directory and Web Services
Grid Bank: Accounting and Transaction Management
Visual Tools for Creation of Distributed Applications
Workflow Composition and Deployment Services
Data Grid Brokering and Grid Economy Services
Data Replication Strategies
GridSim Toolkit: Enhanced to support Data Grid, Reservation, etc.
Libra: SLA-based Allocation of Cluster Resources
Coupling of Clusters and Computational Economy
WWG: Global Data Intensive Grid Testbed
Application Enabler Projects:
High-Energy Physics , Astronomy, Brain Activity Analysis – Osaka U., Natural Language Processing, Portfolio Analysis – Spain, BioGrid - WEHI (via APACGrid), SensorGrid (NICTA), Medical Imaging (HFI)
Supported by:

18. Grid Economy: Methodology for Sustained Resourced Sharing and Managing Supply-and-Demand for Resources
For resource providers, they provide service with different goals
For consumers, they have different requirments
How to manage grid service supply-and-demand is hard task
So how can consumers find a resource depending on their requirements.
And what type of resource that providers provide and how can they serve consumers within certain QoS.
For example, when many consumers want to access same resource, who should be served first
How to manage supply-demand balance is another challenge.

19. New challenges of Grid Economy
Resource Owners
How do I decide prices ? (economic models?)
How do I specify them ?
How do I translate price to resource allocation ?
How do I enforce them ?
How do I advertise & attract consumers ?
How do I do accounting and handle payments?
…..
Resource Consumers
How do I decide expenses ?
How do I express QoS requirements ?
How do I trade between timeframe & cost ?
How do I map jobs to resources to meet my QoS needs?
They need mechanisms and technologies for value expression, value translation, and value enforcement.

20. Grid Entities and Architecture
Grid consumer
GSP site scheduler
Resource owners
broker
GSP global scheduler
GSP site scheduler
Resource owners
Market
Maker
accounting
End users
Private enterprises
National providers

21. A Reference Service-Oriented Architecture for Utility Grids
Data Catalogue
Grid Bank
Grid Market Services
Information Service
Sign-on
Health Monitor
Info ?
Grid Explorer
Grid Node N … …
Programming Environments
Secure
Job Control
Agent
Applications
Schedule Advisor
Grid Node1
QoS
Pricing Algorithms
Trade Server
Trading
Trade Manager
Accounting
Resource
Reservation
Misc. services …
Deployment Agent
JobExec
Resource Allocation
Grid Resource Broker
Storage R1 R2 … Rm
Grid Middleware
Services
Grid Consumer
Grid Service Providers

22. Gridbus and Complementary Technologies – realizing Utility Grid
Applications …
Science
Commerce
Engineering
Collaboratories
Portals …
MPI
ExcellGrid
Gridscape
Workflow
X-Parameter Sweep Lang.
User-Level Middleware
(Grid Tools) …
Grid Brokers:
Nimrod-G
Workflow Engine
Gridbus Data Broker
Core Grid
Middleware
Globus
Unicore
Grid Storage Economy
Grid Exchange & Federation
Grid Market Directory …
Grid Bank
Alchemi
NorduGrid
XGrid
GRIDSIM
Grid Economy
.NET
JVM
Condor
PBS
SGE
Libra
Tomcat
Grid
Fabric
Software
Windows
Solaris
Linux
AIX
IRIX
OSF1
Mac
CDB
Grid
Fabric
Hardware
PDB
Worldwide Grid

23. Alchemi: .NET-based Enterprise Grid Platform & Web Services
Alchemi Manager
Web Services
Internet
Alchemi Users
Internet
like Model
General Purpose
Dedicated/Non-dedicate workers
Role-based Security
.NET and Web Services
C# Implementation
GridThread and Job Model Programming
Easy to setup and use
Widely in use!
Alchemi Worker Agents

24. Many users in Universities: See next for an example.
Some Users of Alchemi
Tier Technologies, USA
Large scale document processing using Alchemi framework
Satyam Computers Applied Research Laboratory, India
Micro-array data processing using Alchemi framework
CSIRO, Australia
Natural Resource Modeling
The University of Sao Paulo, Brazil
The Alchemi Executor as a Windows Service
stochastix GmbH, Germany
Asynchronous Excel Tasks using ManagedXLL and Alchemi .Net Grid Computing framework.
The Friedrich Miescher Institute (FMI) for Biomedical Research, Switzerland
Patterns of transcription factors in mammalian genes
Many users in Universities: See next for an example.

25. On Demand Assembly of Services: Putting Them All Together
Visual Application Composer
Application Code
Explore data 1
Data Source
Data
Results+ Cost Info 10
Grid Resource Broker 2
Bill 12
(Instruments/distributed sources)
Data Replicator
(GDMP)
Data Catalogue 5
ASP Catalogue
Grid Info Service
Grid Market Directory 4 6 3
Job 8
Results 9 7
GSP
(e.g., UofM) PE
(e.g., VPAC)
(e.g., IBM)
CPU
or PE
Grid Service (GS)
(Globus)
Alchemi GS
GTS
Cluster Scheduler
GSP
(Accounting Service)
Gridbus GridBank 11
Cluster Scheduler PE
Grid Service Provider (GSP) (e.g., CERN)

26. The Gridbus Grid Service Broker for Data Grid Applications
Builds on the Nimrod-G Computational Grid Broker and Computational Economy [Buyya, Abramson, Giddy, Monash University, ]
And
Extends its notion for Data and Service Grids

27. Gridbus Broker Architecture
Gridbus Client
Gribus Client
Gridbus Client
(Bag of Tasks Applications)
App, T, $, Opt
(Data Grid Scheduler)
Gridbus Farming Engine
Schedule Advisor
Record Keeper
Trading Manager
Grid Dispatcher
Grid Explorer
Grid Middleware
TM TS $
GE GIS, NWS
Grid Info Server
RM & TS G $
Data
Catalog
Data
Node C $ U G
Unicore enabled node.
Globus enabled node. L A
RM: Local Resource Manager, TS: Trade Server
Alchemi enabled node.

28. Gridbus Broker and Remote Service Access Enablers
Home Node/Portal
Gridbus
Broker
fork()
batch()
-PBS
-Condor
-SGE
-Alchemi
-XGrid
Credential Repository
MyProxy
Portlets
Data Catalog
Alchemi
-PBS
-Condor
-SGE
Globus
Job manager
fork()
batch()
Gridbus
agent
Data Store
Gateway
Unicore
-PBS
-Condor
-SGE
-XGrid
SSH
fork()
batch()
Gridbus
agent
Access Technology
SRB
Grid FTP

29. Gridbus Services for eScience applications
Application Development Environment:
XML-based language for composition of task farming (legacy) applications as parameter sweep applications.
Task Farming APIs for new applications.
Web APIs (e.g., Portlets) for Grid portal development.
Threads-based Programming Interface
Workflow interface and Gridbus-enabled workflow engine.
Resource Allocation and Scheduling
Dynamic discovery of optional computational and data nodes that meet user QoS requirements.
Hide Low-Level Grid Middleware interfaces
Globus, Alchemi, Unicore, NorduGrid, XGrid, etc.

30. Figure 3 : Logging into the portal.
Click Here for Demo
Figure 3 : Logging into the portal.
Drug Design
Made Easy!

31. Economy-based Data Grid Scheduling
CLICK HERE TO SKIP IF RUNNING OUT of TIME
High Energy Physics as eScience Application Case Study

32. Australian Belle Data Grid Testbed

33. Case Study: Event Simulation and Analysis
B0->D*+D*-Ks
Simulation and Analysis Package - Belle Analysis Software Framework (BASF)
Experiment in 2 parts – Generation of Simulated Data and Analysis of the distributed data
Analyzed 100 data files (30MB each) were distributed among the five nodes

34. Resources Used and their Service Price
Organization
Node details
Role
Cost (in G$/CPU-sec)
CS,UniMelb
belle.cs.mu.oz.au 4 CPU, 2GB RAM, 40 GB HD, Linux
Broker host, Data host, NWS server
N.A. (Not used as a compute resource)
Physics, UniMelb
fleagle.ph.unimelb.edu.au 1 CPU, 512 MB RAM, 40 GB HD, Linux
Replica Catalog host, Data host, Compute resource, NWS sensor 2
CS, University of Adelaide
belle.cs.adelaide.edu.au
4 CPU (only 1 available) , 2GB RAM, 40 GB HD, Linux
Data host, NWS sensor
ANU, Canberra
belle.anu.edu.au 4 CPU, 2GB RAM, 40 GB HD, Linux
Data host, Compute resource, NWS sensor 4
Dept of Physics, USyd
belle.physics.usyd.edu.au 4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux
VPAC, Melbourne
brecca-2.vpac.org
180 node cluster (only head node used), Linux
Compute resource,
NWS sensor 6

35. Network Cost (in Grid $/Currency!)

36. Deploying Application Scenario
A data grid scenario with 100 jobs and each accessing remote data of ~30MB
Deadline: 3hrs.
Budget: G$ 60K
Scheduling Optimisation Scenario:
Minimise Time
Minimise Cost
Results:

37. Grid and Gridbus Technologies for Various Grid (Market) Types
free trading
Public computing
(Alchemi)
Private enterprises
(Libra, Gridbus, Globus)
Sharing Model
National provider (Globus, Gridbus,..)
regulation
scientific
commercial
Application Category

38. (5) IT services as the fifth utility (water, electricity, gas, telephone, IT)
eScience
eBusiness
eGovernment
eHealth
Multilingual
eEducation …

39. Summary and Conclusion
Grids exploit synergies that result from cooperation of autonomous entities:
Resource sharing, dynamic provisioning, and aggregation at global level.
Grid Economy provides incentive needed for sustained cooperation.
Grid Network has potential to serve as Cyberinfrastructure for Utility Computing
Grids offer enormous opportunities for realizing eScience and eBusiness at global level.

40. Gridbus Project - http://www.gridbus.org
Any Questions ?
Gridbus Project -

41. Thanks for your attention!
The Gridbus Cooperation!

42. Backup Slides

43. Some Open Research Challenges
Value expression, translation, and enforcement mechanisms and supporting Grid technologies for:
different economic models for spot markets and futures
application models
Dynamic Pricing Schemes
Interaction Protocols for Service Negotiation
Micro payments and Digital Currencies
Scheduling Algorithms
Programming Environments for Building Information Utility Applications
Last, but not least:
Dispute Managements and Legal Issues
Taxation (consult your National Taxation Office)
State, national, and international boundaries
Tax returns!

44. This talk is designed to answer:
How can Grid technologies support the emergence and operation of virtual enterprises?
How can Grid shared resources be treated, brokered, and marketed as ICT ‘commodities’ or ‘futures’ among networked organisations?
What kind of Grid architecture is needed for handling such market mechanisms in an automated fashion?
How can Grid economies map the evolution of networked business models?

45. What do Grids aim for and how to support them.
Grids aim at exploiting synergies that result from cooperation of autonomous distributed entities. Synergies include:
Resource sharing
“On-demand” Virtual Enterprises creation
Aggregation of resources on demand.
For this cooperation to be sustainable, participants needs to have (economic) incentive.
Therefore, “incentive” mechanisms should be considered as one of key design parameters of Grid computing.

46. Grid Market (Participant) Types and Application Category
free trading
Public computing
Private enterprises
Sharing Model
National provider
regulation
scientific
commercial
Application Category

47. Appropriate Market Model for different market types
high
Variable price auction
Commodity market
Willingness to Pay
Posted price oligopoly
low
weak
strong
Demand elasticity

48. Realising Market-based Grid: Minimal New Components
Grid Market Directory Services
Grid Trading Services –
for different economic models
Grid Metering Services
Grid Accounting and Payment Services
Grid Service Broker

49. Deadline (D) and Budget (B) Constrained Scheduling Algorithms
Execution Time (D)
Execution Cost (B)
Compute Grid
Data Grid
Cost Opt
Limited by D
Minimize
Yes
Cost-Time Opt
Minimize if possible
Time Opt
Limited by B
Conservative-Time Opt
Limited by B, jobs have guaranteed minimum budget