A Case for Economy Grid Architecture for Service Oriented Grid Computing Rajkumar Buyya, David Abramson, Jon Giddy School of Computer Science and Software Engineering, Monash University, Melbourne, Australia

Overview A brief introduction to Grid computing Resource Management issues A Glance at Approaches to Grid computing. Grid Architecture for Computational Economy Economy Grid = Globus + GRACE Nimrod-G: A Grid Resource Broker Scheduling Experiments Conclusions SchedulingEconomics Grid Economy Grid

2100 DesktopSMPs or SuperComputers Local Cluster Global Cluster/Grid PERFORMANCEPERFORMANCE Inter Planet Cluster/Grid ?? Individual Group Department Campus State National Globe Inter Planet Universe Administrative Barriers Enterprise Cluster/Grid ? Scalable HPC: Breaking Administrative Barriers

Why Grids ? Large Scale Exploration needs themKiller Applications. Solving grand challenge applications using computer modeling, simulation and analysis Life Sciences CAD/CAM Aerospace Military Applications Digital Biology Military Applications Internet & Ecommerce

What is Grid ? An infrastructure that couples: Computers – PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc; Software – e.g., ASPs renting expensive special purpose applications on demand; Catalogued data and databases – e.g. transparent access to human genome database; Special devices – e.g., radio telescope – searching for life in galaxy. People/collaborators. Potentially Offers a simple, consistent, dependable, and pervasive access across wide- area networks and presents users with an integrated global resource.

Grid Applications-Drivers Distributed HPC (Supercomputing): Computational science. High-throughput computing: Large scale simulation/chip design & parameter studies. Content Sharing Sharing digital contents among peers (e.g., Napster) Remote software access/renting services: Application service provides (ASPs). Data-intensive computing: Data mining, particle physics (CERN), Drug Design. On-demand computing: Medical instrumentation & network-enabled solvers. Collaborative: Collaborative design, data exploration, education.

Building and Using Grids requires... Services that make our systems Grid Ready! Security mechanisms that permit resources to be accessed only by authorized users. (New) programming tools that make our applications Grid Ready!. Tools that can translate the requirements of an application into requirements for computers, networks, and storage. Tools that perform resource discovery, trading, composition, scheduling and distribution of jobs and collects results.

Players in Grid Computing

What users want ? Users in Grid Economy & Strategy Grid Consumers Execute jobs for solving varying problem size and complexity Benefit by selecting and aggregating resources wisely Tradeoff timeframe and cost Strategy: minimise expenses Grid Providers Contribute idle resource for executing consumer jobs Benefit by maximizing resource utilisation Tradeoff local requirements & market opportunity Strategy: maximise returns on services

Sources of Complexity in Resource Management for World Wide Computing Size (large number of nodes, providers, consumers) Heterogeneity of resources (PCs, Workstatations, clusters, and supercomputers) Heterogeneity of fabric management systems (single system image OS, queuing systems, etc.) Heterogeneity of fabric management polices Heterogeneity of applications (scientific, engineering, and commerce) Heterogeneity of application requirements (CPU, I/O, memory, and/or network intensive) Heterogeneity in demand patters Geographic distribution and different time zones Differing goals (producers and consumers have different objectives and strategies) Unsecure and Unreliable environment

Traditional approaches to resource management are NOT useful for Grid ? They use centralised policy that need complete state-information and common fabric management policy or decentralised consensus-based policy. Due to too many heterogenous parameters in the Grid it is impossible to define: system-wide performance matrix and common fabric management policy that is acceptable to all. So, we propose the usage of economics paradigm for managing resources proved successful in managing decentralization and heterogeneity that is present in human economies! We can easy leverage proven Economic principles and techniques Easy to regulate demand and supply User-centric, scalable, adaptable, value-driven costing, etc. Offers incentive (money?) for being part of the grid!

mix-and-match Object-oriented Internet-WWW Problem Solving Approach Market/Computational Economy

Grid RMS to support Ack: Globus.. Authentication (once). Specify (code, resources, etc.). Discover resources. Negotiate authorization, acceptable use, Cost, etc. Acquire resources. Schedule Jobs. Initiate computation. Steer computation. Access remote data-sets. Collaborate with results. Account for usage. Discover resources. Negotiate authorisation, acceptable use, Cost, etc. Acquire resources. Schedule jobs. Initiate computation. Steer computation. Domain 2 Domain 1

Building an Economy Grid brokerage system….. Foundation for the Grid Economy

Economic Models for Resource Trading Commodity Market Model Posted Prices Models Bargaining Model Tendering (Contract Net) Model Auction Model English, first-price sealed-bid, second-price sealded-bid (Vickrey), and Dutch. Proportional Resource Sharing Model Shareholder Model Partnership Model

Grid Node N Grid Architecture for Computational Economy Grid User Application Grid Resource Broker Grid Service Providers Grid Explorer Schedule Advisor Trade Manager Job Control Agent Deployment Agent Trade Server Resource Allocation Resource Reservation R1R1 Misc. services Information Server(s) R2R2 RmRm … Pricing Algorithms Accounting Grid Node1 … Grid Middleware Services … … Health Monitor Grid Market Services JobExec Info ? Secure Trading QoS Storage Sign-on

Economy Grid = Globus + GRACE Applications MDS GRAM Globus Security Interface Heartbeat Monitor Nexus Local Services LSF Condor GRDQBank PBS TCP SolarisIrixLinux UDP High-level Services and Tools DUROCglobusrunMPI-G Nimrod/G MPI-IOCC++ GlobusViewGrid Status GASS GRACE-TS GARA Grid Fabric Grid Apps. Grid Middleware Grid Tools GBank GMD eCash JVM DUROC Core Services ScienceEngineeringCommercePortalsActiveSheet … …

GRACE components A resource broker (e.g., Nimrod/G) Resource trading protocols A mediator for negotiating between users and grid service providers (Grid Market Directory) A deal template for specifying resource requirements and services offers A trade server A pricing policy specification Accounting (e.g., QBank) and payment management (GBank)

Grid Open Trading Protocols Get Connected Call for Bid(DT) Reply to Bid (DT) Negotiate Deal(DT) Confirm Deal(DT, Y/N) …. Cancel Deal(DT) Change Deal(DT) Get Disconnected Trade Manager Trade Server Pricing Rules DT - Deal Template - resource requirements (BM) - resource profile (BS) - price (any one can set) - status - change the above values - negotiation can continue - accept/decline - validity period API

Open Trading Finite State Machine DT Offer TS DT DNDA Offer TM > > DT - Deal Template TM - Trade Manager TM - Trade Server DA - Deal Accepted DN - Deal Not accepted

Pricing, Accounting, Allocations and Job Scheduling each site/Grid Level QBank Resource Manager 4 IBM-LL/PBS/… Compute Resources clusters/SGI/SP/ Make Deposits, Transfers, Refunds, Queries/Reports 1. Clients negotiates for access cost. 2. Negotiation is performed per owner defined policies. 3. If client is happy, TS informs QB about access deal. 4. Job is Submitted 5. Check with QB for go ahead 6. Job Starts 7. Job Completes 8. Inform QB about resource resource utilization. Trade Server 3 1 Pricing Policy 2 Site GRID Bank (digital transactions) 0

Service Items to be Charged CPU - User and System time Memory: maximum resident set size - page size amount of memory used page faults: with/without physical I/O Storage: size, r/w/block IO operations Network: msgs sent/received Signals received, context switches Software and Libraries accessed Data Sources (e.g. Protein Data Bank)

How to decide Price ? Fixed price model (like todays Internet) Dynamic/Demand and Supply (like tomorrows Internet) Usage Period Loyalty of Customers (like Airlines favoring frequent flyers!) Historical data Advance Agreement (high discount for corporations) Usage Timing (peak, off-peak, lunch time) Calendar based (holiday/vacation period) Bulk Purchase (register 100.com domains at once!) Voting -- trade unions decide pricing structure Resource capability as benchmarked in the market! Academic R&D/public-good application users can be offered at cheaper rate compared to commercial use. Customer Type – Quality or price sensitive buyers. Can be Prescribed by Regulating (Govt.) authorities

Payments- Options & Automation Buy credits in advance / GSPs bill the user later--pay as you go Pay by Electronic Currency via Grid Bank NetCash (anonymity), NetCheque, and Paypal NetCheque: - Users register with NC accounting servers, can write electronic cheques and send (e.g ). When deposited, balance is transferred from sender to receiver account. NetCash - It supports anonymity and it uses the NetCheque system to clear payments between currency servers. Paypal.com– account+ is linked to credit card. Enter the recipients address and the amount you wish to request. The recipient gets an notification and pays you at

A Glance at Nimrod-G Broker Grid Middleware Nimrod/G Client Grid Information Server(s) Schedule Advisor Trading Manager Nimrod/G Engine Grid Store Grid Explorer GE GIS TM TS RM & TS Grid Dispatcher RM: Local Resource Manager, TS: Trade Server Globus,Legion, Condor-g,, Ninf,etc. G G C L Globus enabled node. Legion enabled node. C L Condor enabled node. RM & TS

A resource broker for managing and steering task farming (parametric sweep) applications on computational Grids based on deadline and computational economy. Key Features A single window to manage & control experiment Resource Discovery Trade for Resources Resource Composition & Scheduling Steering & data management It allows to study the behaviour of some of the output variables against a range of different input scenarios. Nimrod/G : A Grid Resource Broker

GlobusLegion Fabric Nimrod Broker Nimrod Clients P-Tools (GUI/Scripting) (parameter_modeling) Legacy Applications Condor-GGRACE-TS Farming Engine Dispatcher (transport and execution management) Schedule Advisor Trading Manager Grid Explorer Customised Apps (Active Sheet) Monitoring and Steering Portals Algorithm1 AlgorithmN Middleware... ComputersStorageNetworksInstrumentsLocal Schedulers G-Bank... VariablesResource Programmable Entities Management JobTask... ResourceSchedulerJobServer PC/WS/ClustersRadio TelescopeCondor/LL/Mosix/... Database Meta-Scheduler Nimrod/G Grid Broker Architecture

A Nimrod/G Client CostDeadline Legion hosts Globus Hosts Bezek is in both Globus and Legion Domains

Nimrod/G Interactions Grid Info servers Resource Discovery Queuing System Process server Resource allocation (local) User process File access I/O server Gatekeeper node Job Wrapper Computational node Dispatcher Root node Scheduler Farming Engine Trade Server

Discover Resources Distribute Jobs Establish Rates Meet requirements ? Remaining Jobs, Deadline, & Budget ? Evaluate & Reschedule Discover More Resources Adaptive Scheduling algorithms Compose & Schedule

Globus+Legion +Condor/G Australia Monash Uni.: Linux cluster Solaris WS Nimrod/G Globus + GRACE_TS Europe ZIB/FUB: T3E/Mosix Cardiff: Sun E6500 Paderborn: HPCLine Lecce: Compaq SC CNR: Cluster Calabria: Cluster CERN: Cluster Pozman: SGI/SP2 Globus + GRACE_TS Asia/Japan Tokyo I-Tech.: ETL, Tuskuba Linux cluster Globus/Legion GRACE_TS North America ANL: SGI/Sun/SP2 USC-ISI: SGI UVa: Linux Cluster Internet Inter-Continental Grid

Experiment-1 Setup Workload: 165 jobs, each need 5 minute of cpu time Deadline: 1 hrs. and budget: 800,000 units Strategy: minimise cost and meet deadline Execution Cost with cost optimisation AU Peaktime: (G$) AU Offpeak time: (G$)

Resources Selected & Price/CPU-sec. Resource Type & Size Owner and Location Grid servicesPeaktime Cost (G$) Offpeak cost Linux cluster (60 nodes) Monash, Australia Globus/Condor205 IBM SP2 (80 nodes) ANL, Chicago, US Globus/LL510 Sun (8 nodes)ANL, Chicago, US Globus/Fork510 SGI (96 nodes)ANL, Chicago, US Globus/Condor-G15 SGI (10 nodes)ISI, LA, USGlobus/Fork1020

AU Peak Time

AU Offpeak Time

AU peak: Resources/Cost in Use After the calibration phase, note the difference in pattern of two graphs. This is when scheduler stopped using expensive resources.

AU offpeak: Resources/Cost in Use

Data Intensive Computing on Grid A Virtual Laboratory for Molecular Modelling for Drug Design" on Peer-to-Peer Grid. It provides tools for examining millions of chemical compounds (molecules) in the Protein Data Bank (PDB) to identify those having potential use in drug design. In collaboration with: Kim Branson, Structural Biology, Walter and Eliza Hall Institute (WEHI)

Active Sheet: Spreadsheet Processing on Grid NimrodProxy Nimrod/G

Related Works (contd) Mariposa-Distributed Database system (UCB) query with budget, creates sub-query & divides budget, trades with (remote) servers UCB Millennium clusters remote execution environment on clusters and supports computational economy rexec for clusters - proportional resource sharing UNSW Mungi Storage management: allocation of backing store and garbage collection of unwanted memory segments depending available credit. Amount of credit required to store increases as available storage space becomes minimum.

Related Works JaWS - Java based Webcomputing system offers market oriented programming and computing mechanisms on the Web. Xenoservers - Accounted execution of untrusted code DAgents - Agents and computational economy MOSIX - cost based cluster load balancing A number of theoretical works on pricing. FIPA standard Agents Interaction Protocols (for trading) - we plan to explore this!

I think there is a world market for about five computers. Thomas J. Watson Sr., IBM Founder, 1943 Can we Predict its Future ?

Conclusions The HPC will be dominated by Peer-to-Peer Grid of clusters. Adaptive, scalable, and easy to use Systems and End-User applications will be prominent. Access electricity, internet, entertainment (music, movie,…), etc. from the wall socket! An Economics –based Service Oriented Grid Computing computing needed for eventual success of Grids! The impact of World-Wide Grid on 21 st century economy will be the same as electricity on 20 th century economy.

Thank You… Any ??