Presentation on theme: "1 Architectural Models for Resource Management in the Grid Rajkumar Buyya Monash University, Australia Steve Chapin, Syracuse University, USA David DiNucci."— Presentation transcript:
1 Architectural Models for Resource Management in the Grid Rajkumar Buyya Monash University, Australia Steve Chapin, Syracuse University, USA David DiNucci Elepar.com, USA
3 Computing Power (HPC) Drivers Solving grand challenge applications using computer modeling, simulation and analysis Life Sciences CAD/CAM Aerospace Military Applications Digital Biology Military Applications Internet & Ecommerce
4 2100 DesktopSMPs or SuperComputers Local Cluster Global Cluster/Grid PERFORMANCEPERFORMANCE Inter Planet Cluster/Grid ?? Individual Group Department Campus State National Globe Inter Planet Universe Breaking Administrative Barriers Enterprise Cluster/Grid ? Computing Platforms
5 Towards Grid Computing Unification of geographically distributed resources
6 What is Grid ? An infrastructure that couples: ã Computers – PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc; ã Software – e.g., ASPs renting expensive special purpose applications on demand; ã Catalogued data and databases – e.g. transparent access to human genome database; ã Special devices – e.g., radio telescope – SETI@Home searching for life in galaxy, Austrophysics@Swinburne for pulsars) ã People/collaborators. & offers dependable, consistent, & pervasive access to resources.
8 Sources of Complexity in Grid Resource Management No single administrative control. No single ownership policy: ã Each resource owner has their own policies or scheduling mechanisms; ã Users must honour them (particularly external Grid users). Heterogeneity of resources. Dynamic availability – may appear and disappear…
9 Sources of Complexity in Grid Resource Management Unreliable resource – disappear from view! No uniform cost model - varies from one users resource to another and from time of day. No single access mechanism – Web, custom interfaces, command line…
11 Architectural Models MODELREMARKSSystems HierarchicalIt captures model followed in most contemporary systems. Globus, Legion, CCS, Apples, NetSolve, Ninf. Abstract Owner (AO) Order and delivery model and focuses on long term goals. Expected to emerge and most peer-2- peer computing systems likely to be based on this. Market ModelIt follows economic model for resource discover, sharing, & scheduling. GRACE, Nimrod/G, JavaMarket, Mariposa.
12 Hierarchical Resource Management Connection Cloud User Global Scheduler Control Domain Monitor Resource Grid Information Service Access/Admission Control Agent Domain Resource Manager or Control Agent Global Scheduler Local Scheduler - Task Persistent Job Control Agent Global Scheduler Global Scheduler Global Scheduler Deployment Agent
13 Who owns the GRID? Talk to people Power appliances Use GRID resources Phone co.Electric co. Abstract Owner (AO) I want to: My interface is: I arrange service and payments with a: (may be many choices) But resources I get may belong to others: Generators Power lines Transformers Antennae Cable/fiber Switches HPC Networks Instruments People
14 OrderPickup AO is owner or broker User negotiates with AO through order window That AO may own some resources, and/or it may broker with other AOs for those resources After negotiation, resources are delivered through pickup window Order Window Pickup Window Physical Resource User RequestsResources AO Order Pickup Resource Manager AO1 Manager DeliverySales AO2 AO3
15 AO Resources Resources are objects Classes are ã Instrument »Data source, sink, transform »e.g. programs, people, files, data collection devices ã Channel »Moves data among instruments ã Complexes of above Attributes define sizes, times, connections, etc. Instrument (File) Instrument (Program) Instrument (File) Instrument (Program) Channels Instrument (Telescope) Instrument (Person)
16 Negotiating with an AO Make dummy resource (with attributes set to constants, variables, or dont care) + bid + delivery plan + variable constraints Resource candidates (values for variables/attributes + asking price for each) Pick one, Try again, Or give up Delivery Window Resource Order Window Assign tasks to resource, use, relinquish Perhaps later...
17 Many Testbeds ? & who pays ? GUSTO Legion Testbed NASA IPG EcoGrid
18 Testbeds so far -- observations Who contributed to resources & why ? ã Volunteers: for fun, challenge, fame, public good like SETI@Home & distributed.net projects.SETI@Home ã Collaborators: sharing resources while developing new technologies of common interest – Globus, Legion, Ecogrid. How long ? ã Short duration: GUSTO decommissioned. What do we need ? Grid Marketplace! ã Regulates demand and supply, offers incentive for being players, simple, scalable solution, quasi-deterministic – proven model in real-world.
19 Users in Grid Economy & Strategy Grid Consumers ã Execute jobs for solving varying problem size and complexity ã Benefit by selecting and aggregating resources wisely ã Tradeoff timeframe and cost »Strategy: minimise expenses Grid Providers ã Contribute idle resource for executing consumer jobs ã Benefit by maximizing resource utilisation ã Tradeoff local requirements & market opportunity »Strategy: maximise returns on services
20 Building of a Economy Grid brokerage system….. Foundation for the Grid Economy
21 Grid Node N Grid Architecture for Computational Economy Grid User Application Grid Resource Broker Grid Service Providers Grid Explorer Schedule Advisor Trade Manager Job Control Agent Deployment Agent Trade Server Resource Allocation Resource Reservation R1R1 Misc. services Information Server(s) R2R2 RmRm … Pricing Algorithms Accounting Grid Node1 … Grid Middleware Services … … Health Monitor Grid Market Services JobExec Info ? Secure Trading QoS Storage Sign-on
22 Economic Models for Trading Commodity Market Model Posted Prices Models Bargaining Model Tendering (Contract Net) Model Auction Model ã English, first-price sealed-bid, second-price sealded-bid (Vickrey), and Dutch. Proportional Resource Sharing Model Shareholder Model Partnership Model
24 A resource broker for managing and steering task farming (parametric sweep) applications on computational Grids based on deadline and computational economy. Key Features ã A single window to manage & control experiment ã Resource Discovery ã Trade for Resources ã Scheduling ã Steering & data management It allows to study the behaviour of some of the output variables against a range of different input scenarios. What is Nimrod/G ?
25 Nimrod/G Grid Broker Architecture Grid Middleware Nimrod/G Client Grid Information Server(s) Schedule Advisor Trading Manager Nimrod/G Engine Grid Store Grid Explorer GE GIS TM TS RM & TS Grid Dispatcher RM: Local Resource Manager, TS: Trade Server Globus,Legion, Condor-g,, Ninf,etc. G G C L Globus enabled node. Legion enabled node. C L Condor enabled node. RM & TS
26 A Nimrod/G Client CostDeadline Legion hosts Globus Hosts Bezek is in both Globus and Legion Domains
28 Globus+Legion +Condor/G Australia Monash Uni.: Linux cluster Solaris WS Nimrod/G Globus + GRACE_TS Europe ZIB/FUB: T3E/Mosix Cardiff: Sun E6500 Paderborn: HPCLine Lecce: Compaq SC CNR: Cluster CERN: Cluster Globus + GRACE_TS Asia/Japan Tokyo I-Tech.: ETL, Tuskuba Linux cluster Globus/Legion GRACE_TS North America ANL: SGI/Sun/SP2 USC-ISI: SGI UVa: Linux Cluster Manitoba: Cluster Internet Global Economy Grid
30 Conclusions Proposed three models for Grid resource management architecture ã Hierarchical, AO, & Market-model The future systems are likely follow a model that combines all these models. The future computing (HPC) infrastructure is going to be a Grid of Clusters. Peer-to-Peer/Grid has already become a darling of venture capitalists. The impact of Grid on 21 st century economy will be the same as electricity on 20 th century economy.