Presentation is loading. Please wait.

Presentation is loading. Please wait.

Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia

Similar presentations

Presentation on theme: "Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia"— Presentation transcript:

1 Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia WW Grid

2 2

3 3 Need Honest Answers! I want to have access to your Grid resources & want to know how many of you are willing to give me access ? (following cases) I am unable to give you access our Australian machines, but I want to have access to yours! [social] Want to solve academic problems Want to solve business problems I am willing to gift you Kangaroos! [bartering] I am willing to give you access to my machines, if you want. (sharing, but no measure & no QoS) [bartering] I am willing to pay you dollars on usage basis. [economic incentive, market-based, and QoS] WW Grid

4 4 Overview A quick glance at today s Grid computing Resource Management challenges for next generation Grid computing A Glance at Approaches to Grid computing. Grid Architecture for Computational Economy Economy Grid = Globus + GRACE Nimrod-G -- Grid Resource Broker Scheduling Experiments Case Study: Drug Design Application on Grid Conclusions SchedulingEconomics Grid Economy Grid

5 5 2100 DesktopSMPs or SuperComputers Local Cluster Global Cluster/Grid PERFORMANCEPERFORMANCE Inter Planetary Grid! Individual Group Department Campus State National Globe Inter Planet Universe Administrative Barriers Enterprise Cluster/Grid ? Scalable HPC: Breaking Administrative Barriers & new challenges

6 6 Why Grids? Large Scale Explorations need them Killer Applications. Solving grand challenge applications using modeling, simulation and analysis Life Sciences CAD/CAM Aerospace Military Applications Digital Biology Military Applications Internet & Ecommerce

7 7

8 8 What is Grid ? An infrastructure that logically couples distributed resources: Computers – PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc; Software – e.g., ASPs renting expensive special purpose applications on demand; Catalogued data and databases – e.g. transparent access to human genome database; Special devices – e.g., radio telescope – SETI@Home searching for life in galaxy. People/collaborators. and presents them as an integrated global resource. It enables the creation of virtual enterprises (VEs) for resource sharing. Wide area data archives

9 9 P2P/Grid Applications-Drivers Distributed HPC (Supercomputing): Computational science. High-Capacity/Throughput Computing: Large scale simulation/chip design & parameter studies. Content Sharing (free or paid) Sharing digital contents among peers (e.g., Napster) Remote software access/renting services: Application service provides (ASPs) & Web services. Data-intensive computing: Virtual Drug Design, Particle Physics, Stock Prediction... On-demand, realtime computing: Medical instrumentation & Mission Critical. Collaborative Computing: Collaborative design, Data exploration, education. Service Oriented Computing (SOC): Computing as Utility: New paradigm and new industries.

10 10 Building and Using Grids require Services that make our systems Grid Ready! Security mechanisms that permit resources to be accessed only by authorized users. (New) programming tools that make our applications Grid Ready!. Tools that can translate the requirements of an application/user into the requirements of computers, networks, and storage. Tools that perform resource discovery, trading, selection/allocation, scheduling and distribution of jobs and collects results. Globus ?

11 11 Players in Grid Computing

12 12 What users want ? Users in Grid Economy & Strategy Grid Consumers Execute jobs for solving varying problem size and complexity Benefit by selecting and aggregating resources wisely Tradeoff timeframe and cost Strategy: minimise expenses Grid Providers Contribute idle resource for executing consumer jobs Benefit by maximizing resource utilisation Tradeoff local requirements & market opportunity Strategy: maximise return on investment

13 Challenges for Next Generation Grid Technology Development

14 14 Challenges for Grid Computing Security Resource Allocation & Scheduling Data locality Network Management System Management Resource Discovery Uniform Access Computational Economy Application Development Tools

15 15 Sources of Complexity in Resource Management for World Wide Grid Computing Size (large number of nodes, providers, consumers) Heterogeneity of resources (PCs, Workstations, clusters, and supercomputers, instruments, databases, software) Heterogeneity of fabric management systems (single system image OS, queuing systems, etc.) Heterogeneity of fabric management polices Heterogeneity of application requirements (CPU, I/O, memory, and/or network intensive) Heterogeneity in resource demand patterns (peak, off-peak,...) Applications need different QoS at different times (time critical results). The utility of experimental results varies from time to time. Geographical distribution of users & located different time zones Differing goals (producers and consumers have different objectives and strategies) Unsecure and Unreliable environment

16 16 Traditional approaches to resource management & scheduling are NOT useful for Grid ? They use centralised policy that need complete state-information and common fabric management policy or decentralised consensus-based policy. Due to too many heterogenous parameters in the Grid it is impossible to define/get: system-wide performance matrix and common fabric management policy that is acceptable to all. Economics paradigm proved to effective institution in managing decentralization and heterogeneity that is present in human economies! Fall of USSR & Emergence of US as world superpower! (monopoly?) So, we propose/advocate the use of computational economics principles in management of resources and scheduling computations on world wide Grid. Think locally and act globally approach to grid computing!

17 17 Benefits of Computational Economies It provides a nice paradigm for managing self interested and self- regulating entities (resource owners and consumers) Helps in regulating supply-and-demand for resources. Services can be priced in such a way that equilibrium is maintained. User-centric / Utility driven: Value for money! Scalable: No need of central coordinator (during negotiation) Resources(sellers) and also Users(buyers) can make their own decisions and try to maximize utility and profit. Adaptable It helps in offering different QoS (quality of services) to different applications depending the value users place on them. It improves the utilisation of resources It offers incentive for resource owners for being part of the grid! It offers incentive for resource consumers for being good citizens There is large body of proven Economic principles and techniques available, we can easily leverage it.

18 18 New challenges of Computational Economy Resource Owners How do I decide prices ? (economic models?) How do I specify them ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments? ….. Resource Consumers How do I decide expenses ? How do I express QoS requirements ? How I trade between timeframe & cost ? …. Any tools, traders & brokers available to automate the process ?

19 19 mix-and-match Object-oriented Internet/partial-P2P Network enabled Solvers Market/Computational Economy

20 20 Many Grid Projects & Initiatives Australia Economy Grid Nimrod-G Virtual Lab Active Sheets coming up Europe UNICORE MOL Lecce GRB Poland MC Broker EU Data Grid EuroGrid MetaMPI Dutch DAS XW, JaWS and many more... Japan Ninf DataFarm and many more... USA Globus Legion Javelin AppLeS NASA IPG Condor Harness NetSolve AccessGrid GrADS and many more... Cycle Stealing &.com Initiatives SETI@Home, …. SETI@Home Entropia, UD, Parabon, …. Public Forums Global Grid Forum P2P Working Group IEEE TFCC Grid & CCGrid conferences

21 21 Many Testbeds ? & who pays ?, who regulates supply and demand ? GUSTO (decommissioned) Legion Testbed NASA IPG World Wide Grid WW Grid

22 22 Testbeds so far -- observations Who contributed resources & why ? Volunteers: for fun, challenge, fame, charismatic apps, public good like & SETI@Home projects.SETI@Home Collaborators: sharing resources while developing new technologies of common interest – Globus, Legion, Ninf, Ninf, MC Broker, Lecce GRB,... Unless you know lab. leaders, it is impossible to get access! How long ? Short term: excitement is lost, too much of admin. Overhead (Globus inst+), no incentive, policy change, … What we need ? Grid Marketplace! Regulates supply-and-demand, offers incentive for being players, simple, scalable solution, quasi- deterministic – proven model in real-world.

23 23 Building an Economy Grid (Next Generation Grid Computing!) To enable the creation and promotion of: Grid Marketplace (competitive) ASP Service Oriented Computing... And let users focus on their own work (science, engineering, or commerce)!

24 24 Grid Node N GRACE: A Reference Grid Architecture for Computational Economy Grid User Application Grid Resource Broker Grid Service Providers Grid Explorer Schedule Advisor Trade Manager Job Control Agent Deployment Agent Trade Server Resource Allocation Resource Reservation R1R1 Misc. services Information Server(s) R2R2 RmRm … Pricing Algorithms Accounting Grid Node1 … Grid Middleware Services … … Health Monitor Grid Market Services JobExec Info ? Secure Trading QoS Storage Sign-on Grid Bank See PDPTA 2000 paper!

25 25 Economic Models Price-based: Supply,demand,value, wealth of economic system Commodity Market Model Posted Price Model Bargaining Model Tendering (Contract Net) Model Auction Model English, first-price sealed-bid, second-price sealed-bid (Vickrey), and Dutch (consumer:low,high,rate; producer:high, low, rate) Proportional Resource Sharing Model Monopoly (one provider) and Oligopoly (few players) consumers may not have any influence on prices. Bartering Shareholder Model Partnership Model See SPIE ITCom 2001 paper!: with Heinz Stockinger, CERN!

26 26 Grid Open Trading Protocols Get Connected Call for Bid(DT) Reply to Bid (DT) Negotiate Deal(DT) Confirm Deal(DT, Y/N) …. Cancel Deal(DT) Change Deal(DT) Get Disconnected Trade Manager Trade Server Pricing Rules DT - Deal Template - resource requirements (BM) - resource profile (BS) - price (any one can set) - status - change the above values - negotiation can continue - accept/decline - validity period API

27 27 Grid Fabric Grid Apps. Grid Middleware Grid Tools Networked Resources across Organisations Computers Clusters Data Sources Scientific Instruments Storage Systems Local Resource Managers Operating Systems Queuing Systems TCP/IP & UDP … Libraries & App Kernels … Distributed Resources Coupling Services SecurityInformation … QoS Process Development Environments and Tools Languages Libraries Debuggers … Web tools Resource BrokersMonitoring Applications and Portals Prob. Solving Env. Scientific … Collaboration Engineering Web enabled Apps Resource Trading Grid Components Market Info

28 28 Economy Grid = Globus + GRACE Applications MDS GRAM Globus Security Interface Heartbeat Monitor Nexus Local Services LSF Condor GRDQBank PBS TCP SolarisIrixLinux UDP High-level Services and Tools DUROCglobusrunMPI-G Nimrod/G CC++ Grid Status GASS GRACE-TS GARA Grid Fabric Grid Apps. Grid Middleware Grid Tools GBank GMD eCash JVM DUROC Core Services ScienceEngineeringCommercePortalsActiveSheet … … See IPDPS HWC 2001 paper! … …

29 29 GRACE components A resource broker (e.g., Nimrod/G) Various resource trading protocols for different economic models A mediator for negotiating between users and grid service providers (Grid Market Directory) A deal template for specifying resource requirements and services offers Grid Trading Server Pricing policy specification Accounting (e.g., QBank) and payment management (GridBank, not yet implemented)

30 30 Pricing, Accounting, Allocations and Job Scheduling Flow @ each site/Grid Level QBank Resource Manager 4 IBM-LL/PBS/…. 0 58 67 Compute Resources clusters/SGI/SP/... 0. Make Deposits, Transfers, Refunds, Queries/Reports 1. Clients negotiates for access cost. 2. Negotiation is performed per owner defined policies. 3. If client is happy, TS informs QB about access deal. 4. Job is Submitted 5. Check with QB for go ahead 6. Job Starts 7. Job Completes 8. Inform QB about resource resource utilization. Trade Server 3 1 Pricing Policy 2 DB@Each Site GRID Bank (digital transactions) 0

31 31 Service Items to be Charged CPU - User and System time Memory: maximum resident set size - page size amount of memory used page faults: with/without physical I/O Storage: size, r/w/block IO operations Network: msgs sent/received Signals received, context switches Software and Libraries accessed Data Sources (e.g. Protein Data Bank)

32 32 How to decide Price ? Fixed price model (like today s Internet) Dynamic/Demand and Supply (like tomorrow s Internet) Usage Period Loyalty of Customers (like Airlines favoring frequent flyers!) Historical data Advance Agreement (high discount for corporations) Usage Timing (peak, off-peak, lunch time) Calendar based (holiday/vacation period) Bulk Purchase (register domains at once!) Voting -- trade unions decide pricing structure Resource capability as benchmarked in the market! Academic R&D/public-good application users can be offered at cheaper rate compared to commercial use. Customer Type – Quality or price sensitive buyers. Can be Prescribed by Regulating (Govt.) authorities

33 33 Payments- Options & Automation Buy credits in advance / GSPs bill the user later-- pay as you go Pay by Electronic Currency via Grid Bank NetCash (anonymity), NetCheque, and Paypal NetCheque: - Users register with NC accounting servers, can write electronic cheques and send (e.g email). When deposited, balance is transferred from sender to receiver account. NetCash - It supports anonymity and it uses the NetCheque system to clear payments between currency servers. – account+email is linked to credit card. Enter the recipient s email address and the amount you wish to request. The recipient gets an email notification and pays you at

34 Nimrod-G: The Grid Resource Broker Soft Deadline and Budget-based Economy Grid Resource Broker for Parameter Processing on P2P Grids

35 35 Parametric Computing (What Users think of Nimrod Power) Multiple Runs Same Program Multiple Data Killer Application for the Grid! Parameters Courtesy: Anand Natrajan, University of Virginia Magic Engine See IPDPS 2000 paper!

36 36 P-study Applications -- Characteristics Code (Single Program: sequential or threaded) High Resource Requirements Long-running Instances Numerous Instances (Multiple Data) High Computation-to-Communication Ratio Embarrassingly/Pleasantly Parallel

37 37 Sample P-Sweep Applications Bioinformatics: Drug Design / Protein Modelling Bioinformatics: Drug Design / Protein Modelling Sensitivity experiments on smog formation Combinatorial Optimization: Meta-heuristic parameter estimation Ecological Modelling: Control Strategies for Cattle Tick Electronic CAD: Field Programmable Gate Arrays Computer Graphics: Ray Tracing High Energy Physics: Searching for Rare Events Finance: Investment Risk Analysis VLSI Design: SPICE Simulations Aerospace: Wing Design Network Simulation Automobile: Crash Simulation Data Mining Civil Engineering: Building Design astrophysics

38 38 Thesis Perform parameter sweep (bag of tasks) (utilising distributed resources) within T hours or early and cost not exceeding $M. Three Options/Solutions: Using pure Globus commands Build your own Distributed App & Scheduler Use Nimrod-G (Resource Broker)

39 39 Remote Execution Steps Choose Resource Transfer Input Files Set Environment Start Process Pass Arguments Monitor Progress Read/Write Intermediate Files Transfer Output Files Summary View Job View Event View +Resource Discovery, Trading, Scheduling, Predictions, Rescheduling,...

40 40 Using Pure Globus commands Do all yourself! (manually) Total Cost:$???

41 41 Build Distributed Application & Scheduler Build App case by case basis Complicated Construction E.g., AppLeS/MPI basedTotal Cost:$???

42 42 Use Nimrod-G Aggregate Job Submission Aggregate View Submit & Play!

43 43 Nimrod & Associated Family of Tools P-sweep App. Composition: Nimrod/Enfusion Resource Management and Scheduling: Nimrod-G Broker Design Optimisations: Nimrod-O App. Composition and Online Visualization: Active Sheets Grid Simulation in Java: GridSim Drug Design on Grid: Virtual Lab Remote Execution Server (on demand Nimrod Agent) File Transfer Server Upcoming?: HEPGrid (+U. Melbourne), GAVE(+Rutherford Appleton Lab) Grid (Un)Aware Virtual Engineering

44 44 A resource broker for managing, steering, and executing task farming (parametric sweep/SPMD model) applications on Grid based on deadline and computational economy. Based on users QoS requirements, our Broker dynamically leases services at runtime depending on their quality, cost, and availability. Key Features A single window to manage & control experiment Persistent and Programmable Task Farming Engine Resource Discovery Resource Trading Scheduling & Predications Generic Dispatcher & Grid Agents Transportation of data & results Steering & data management Accounting Nimrod/G : A Grid Resource Broker

45 45 A Glance at Nimrod-G Broker Grid Middleware Nimrod/G Client Grid Information Server(s) Schedule Advisor Trading Manager Nimrod/G Engine Grid Store Grid Explorer GE GIS TM TS RM & TS Grid Dispatcher RM: Local Resource Manager, TS: Trade Server Globus, Legion, Condor, etc. G G C L Globus enabled node. Legion enabled node. G L Condor enabled node. RM & TS CL See HPCAsia 2000 paper!

46 46 GlobusLegion Fabric Nimrod Broker Nimrod Clients P-Tools (GUI/Scripting) (parameter_modeling) Legacy Applications P2PGTS Farming Engine Dispatcher & Actuators Schedule Advisor Trading Manager Grid Explorer Customised Apps (Active Sheet) Monitoring and Steering Portals Algorithm1 AlgorithmN Middleware... ComputersStorageNetworksInstrumentsLocal Schedulers G-Bank... Agents Resources Programmable Entities Management JobsTasks... AgentSchedulerJobServer PC/WS/ClustersRadio TelescopeCondor/LL/Mosix/... Database Meta-Scheduler Nimrod/G Grid Broker Architecture Globus-A Channels Legion-AP2P-A... Database (Postgres) XML CondorGMD XML? IP hourglass ?

47 47 A Nimrod/G Monitor CostDeadline Legion hosts Globus Hosts Bezek is in both Globus and Legion Domains

48 48 User Requirements: Deadline/Budget

49 49 Active Sheet: Spreadsheet Processing on Grid NimrodProxy Nimrod/G See HPC 2001 paper!

50 50

51 51 Nimrod/G Interactions Grid Info servers Resource Discovery Queuing System Process server Resource allocation (local) User process File access I/O server Gatekeeper node Nimrod Agent Computational node Dispatcher Root node Scheduler Farming Engine Grid Trade Server Do this in 30min. for $10?

52 52 Discover Resources Distribute Jobs Establish Rates Meet requirements ? Remaining Jobs, Deadline, & Budget ? Evaluate & Reschedule Discover More Resources Adaptive Scheduling Algorithms Compose & Schedule See HPDC AMS 2001 paper!

53 53 Cost Model Without cost ANY shared system becomes un-managable Charge users more for remote facilities than their own Choose cheaper resources before more expensive ones Cost units (G$) may be Dollars Shares in global facility Stored in bank

54 54 Cost Matrix @ Grid site X Non-uniform costing Encourages use of local resources first Real accounting system can control machine usage 13 21 User 5 Machine 1 User 1 Machine 5 Resource Cost = Function (cpu, memory, disk, network, software, QoS, current demand, etc.) Simple: price based on peaktime, offpeak, discount when less demand,..

55 55 Deadline and Budget-based Cost Minimization Scheduling 1. Sort resources by increasing cost. 2. For each resource in order, assign as many jobs as possible to the resource, without exceeding the deadline. 3. Repeat all steps until all jobs are processed.

56 56 M - Resources, N - Jobs, D - deadline Note: Cost of any R i is less than any of R i+1 …. Or Rm RL: Resource List need to be maintained in increasing order of cost C t - Time when accessed (Time now) T i - Job runtime (average) on Resource i (R i ) [updated periodically] T i is acts as a load profiling parameter. A i - number of jobs assigned to R i, where: A i = Min (No.Unassigned Jobs, No. Jobs R i can complete by remaining deadline) No.UnAssignedJobs i = Diff( N, (A 1 + … +A i-1 )) JobsR i consume = RemainingTime (D- C t ) DIV T i ALG: Invoke Job Assignment() periodically until all jobs done. Job Assignment()/Reassignment(): Establish ( RL, C t, T i, A i ) dynamically – Resource Discovery. For all resources (I = 1 to M) { Assign A i Jobs to R i, if required} Deadline-based Cost- minimization Scheduling

57 57 Deadline and Budget Constraint (DBC) Time Minimization Scheduling 1. For each resource, calculate the next completion time for an assigned job, taking into account previously assigned jobs. 2. Sort resources by next completion time. 3. Assign one job to the first resource for which the cost per job is less than the remaining budget per job. 4. Repeat all steps until all jobs are processed. (This is performed periodically or at each scheduling-event.)

58 58 Deadline and Budget Constraint (DBC) Time+Cost Min. Scheduling 1. Split resources by whether cost per job is less than budget per job. 2. For the cheaper resources, assign jobs in inverse proportion to the job completion time (e.g. a resource with completion time = 5 gets twice as many jobs as a resource with completion time = 10). 3. For the dearer resources, repeat all steps (with a recalculated budget per job) until all jobs are assigned. 4. [Schedule/Reschedule] Repeat all steps until all jobs are processed.

59 Evaluation of Scheduling Heuristics A Hypothetical Application on World Wide Grid WW Grid

60 60 Globus+Legion GRACE_TS Australia Monash Uni.: Linux cluster Solaris WS Nimrod/G Globus + GRACE_TS Europe ZIB/FUB: T3E/Mosix Cardiff: Sun E6500 Paderborn: HPCLine Lecce: Compaq SC CNR: Cluster Calabria: Cluster CERN: Cluster Pozman: SGI/SP2 Globus + GRACE_TS Asia/Japan Tokyo I-Tech.: ETL, Tuskuba Linux cluster Globus/Legion GRACE_TS North America ANL: SGI/Sun/SP2 USC-ISI: SGI UVa: Linux Cluster UD: Linux cluster UTK: Linux cluster Internet World Wide Grid (WWG) Globus + GRACE_TS South America Chile: Cluster WW Grid

61 61 Experiment-1 Setup Workload: 165 jobs, each need 5 minute of cpu time Deadline: 1 hrs. and budget: 800,000 units Strategy: minimise cost and meet deadline Execution Cost with cost optimisation AU Peaktime:471205 (G$) AU Offpeak time: 427155 (G$)

62 62 Resources Selected & Price/CPU-sec. Resource Type & Size Owner and Location Grid services Peaktime Cost (G$) Offpeak cost Linux cluster (60 nodes) Monash, Australia Globus/Condor205 IBM SP2 (80 nodes) ANL, Chicago, US Globus/LL510 Sun (8 nodes)ANL, Chicago, US Globus/Fork510 SGI (96 nodes)ANL, Chicago, US Globus/Condor-G15 SGI (10 nodes)ISI, LA, USGlobus/Fork1020

63 63 Execution @ AU Peak Time

64 64 Execution @ AU Offpeak Time

65 65 AU peak: Resources/Cost in Use After the calibration phase, note the difference in pattern of two graphs. This is when scheduler stopped using expensive resources.

66 66 AU offpeak: Resources/Cost in Use

67 67 Experiment-2 Setup Workload: 165 jobs, each need 5 minute of CPU time Deadline: 2 hrs. and budget: 396000 units Strategy: minimise time / cost Execution Cost with cost optimisation Optimise Cost: 115200 (G$) (finished in 2hrs.) Optimise Time: 237000 (G$) (finished in 1.25 hr.) In this experiment: Time-optimised scheduling run costs double that of Cost-optimised. Users can now trade-off between Time Vs. Cost.

68 68 Resources Selected & Price/CPU-sec. Resource & Location Grid services & Fabric Cost/CPU sec.or unit No. of Jobs Executed Time_OptCost_Op t. Linux Cluster-Monash, Melbourne, Australia Globus, GTS, Condor 264153 Linux-Prosecco-CNR, Pisa, Italy Globus, GTS, Fork 371 Linux-Barbera-CNR, Pisa, Italy Globus, GTS, Fork 461 Solaris/Ultas2 TITech, Tokyo, Japan Globus, GTS, Fork 391 SGI-ISI, LA, US Globus, GTS, Fork 8375 Sun-ANL, Chicago,US Globus, GTS, Fork 7424 Total Experiment Cost (G$)237000115200 Time to Complete Exp. (Min.)70119

69 69 DBC Scheduling for Time Optimization

70 70 DBC Scheduling for Cost Optimization

71 Application Case Study The Virtual Laboratory Project: "Molecular Modelling for Drug Design" on Peer-to-Peer Grid

72 72 Virtual Drug Design: Data Intensive Computing on Grid A Virtual Laboratory for Molecular Modelling for Drug Design on Peer-to-Peer Grid. It provides tools for examining millions of chemical compounds (molecules) in the Protein Data Bank (PDB) to identify those having potential use in drug design. In collaboration with: Kim Branson, Structural Biology, Walter and Eliza Hall Institute (WEHI)

73 73 Virtual Drug Design A Virtual Lab for Molecular Modeling for Drug Design on P2P Grid Screen 2K molecules in 30min. for $10 Grid Market Directory Resource Broker Grid Info. Service GTS Give me list PDBs sources Of type aldrich_300? service cost? (GTS - Grid Trade Server) PDB2 get mol.10 from pdb1 & screen it. Data Replica Catalogue service providers? GTS PDB1 mol.10 please? mol.5 please? (RB maps suitable Grid nodes and Protein DataBank)

74 74 DataGrid Brokering Nimrod/G Computational Grid Broker Data Replica Catalogue PDB Broker Algorithm1 AlgorithmN... PDB Service PDB2 Screen mol.5 please? GSP1GSP2 GSP4GSP3 (Grid Service Provider) GSPm PDB Service GSPn 1 advise PDB source? 2 selection & advise: use GSP4! 5 Grid Info. Service 3 Is GSP4 healthy? 4 mol.5 please? 6 PDB replicas please? Screen 2K molecules in 30min. for $10 7 process & send results

75 75 Software Tools Molecular Modelling Tools (DOCK) Parameter Modelling Tools (Nimrod/enFusion) Grid Resource Broker (Nimrod-G) Data Grid Broker Protein Data Bank (PDB) Management and Intelligent Access Tools PDB databse Lookup/Index Table Generation. PDB and associated index-table Replication. PDB Replica Catalogue (that helps in Resource Discovery). PDB Servers (that serve PDB clients requests). PDB Brokering (Replica Selection). PDB Clients for fetching Molecule Record (Data Movement). Grid Middleware (Globus and GrACE) Grid Fabric Management (Fork/LSF/Condor/Codine/ … )

76 76 DOCK code* (Enhanced by WEHI, U of Melbourne) A program to evaluate the chemical and geometric complementarities between a small molecule and a macromolecular binding site. It explores ways in which two molecules, such as a drug and an enzyme or protein receptor, might fit together. Compounds which dock to each other well, like pieces of a three-dimensional jigsaw puzzle, have the potential to bind. So, why is it important to able to identify small molecules which may bind to a target macromolecule? A compound which binds to a biological macromolecule may inhibit its function, and thus act as a drug. Thus disabling the ability of (HIV) virus attaching itself to molecule/protein! With system specific code changed, we have been able to compile it for Sun-Solaris, PC Linux, SGI IRIX, Compaq Alpha/OSF1 * Original Code: University of California, San Francisco:

77 77 Dock input file score_ligand yes minimize_ligand yes multiple_ligands no random_seed 7 anchor_search no torsion_drive yes clash_overlap 0.5 conformation_cutoff_factor 3 torsion_minimize yes match_receptor_sites no random_search yes...... maximum_cycles 1 ligand_atom_file S_1.mol2 receptor_site_file ece.sph score_grid_prefix ece vdw_definition_file parameter/vdw.defn chemical_definition_file parameter/chem.defn chemical_score_file parameter/chem_score.tbl flex_definition_file parameter/flex.defn flex_drive_file parameter/flex_drive.tbl ligand_contact_file dock_cnt.mol2 ligand_chemical_file dock_chm.mol2 ligand_energy_file dock_nrg.mol2 Molecule to be screened

78 78 score_ligand $score_ligand minimize_ligand $minimize_ligand multiple_ligands $multiple_ligands random_seed $random_seed anchor_search $anchor_search torsion_drive $torsion_drive clash_overlap $clash_overlap conformation_cutoff_factor $conformation_cutoff_factor torsion_minimize $torsion_minimize match_receptor_sites $match_receptor_sites random_search $random_search...... maximum_cycles $maximum_cycles ligand_atom_file ${ligand_number}.mol2 receptor_site_file $HOME/dock_inputs/${receptor_site_file} score_grid_prefix $HOME/dock_inputs/${score_grid_prefix} vdw_definition_file vdw.defn chemical_definition_file chem.defn chemical_score_file chem_score.tbl flex_definition_file flex.defn flex_drive_file flex_drive.tbl ligand_contact_file dock_cnt.mol2 ligand_chemical_file dock_chm.mol2 ligand_energy_file dock_nrg.mol2 Parameterized Dock input file Molecule to be screened

79 79 parameter database_name label "database_name" text select oneof "aldrich" "maybridge" "maybridge_300" "asinex_egc" "asinex_epc" "asinex_pre" "available_chemicals_directory" "inter_bioscreen_s" "inter_bioscreen_n" "inter_bioscreen_n_300" "inter_bioscreen_n_500" "biomolecular_research_institute" "molecular_science" "molecular_diversity_preservation" "national_cancer_institute" "IGF_HITS" "aldrich_300" "molecular_science_500" "APP" "ECE" default "aldrich_300"; parameter score_ligand text default "yes"; parameter minimize_ligand text default "yes"; parameter multiple_ligands text default "no"; parameter random_seed integer default 7; parameter anchor_search text default "no"; parameter torsion_drive text default "yes"; parameter clash_overlap float default 0.5; parameter conformation_cutoff_factor integer default 5; parameter torsion_minimize text default "yes"; parameter match_receptor_sites text default "no"; parameter random_search text default "yes";...... parameter maximum_cycles integer default 1; parameter receptor_site_file text default "ece.sph"; parameter score_grid_prefix text default "ece"; parameter ligand_number integer range from 1 to 200 step 1; Dock PlanFile (contd.) Molecules to be screened

80 80 task nodestart copy./parameter/vdw.defn node:. copy./parameter/chem.defn node:. copy./parameter/chem_score.tbl node:. copy./parameter/flex.defn node:. copy./parameter/flex_drive.tbl node:. copy./dock_inputs/get_molecule node:. copy./dock_inputs/dock_base node:. endtask task main node:substitute dock_base dock_run node:substitute get_molecule get_molecule_fetch node:execute sh./get_molecule_fetch node:execute $HOME/bin/dock.$OS -i dock_run -o dock_out copy node:dock_out./results/dock_out.$jobname copy node:dock_cnt.mol2./results/dock_cnt.mol2.$jobname copy node:dock_chm.mol2./results/dock_chm.mol2.$jobname copy node:dock_nrg.mol2./results/dock_nrg.mol2.$jobname endtask Dock PlanFile

81 81 Nimrod/TurboLinux enFuzion GUI tools for Parameter Modeling

82 82 Docking Experiment Preparation Setup PDB DataGrid Index PDB databases Pre-stage (all) Protein Data Bank (PDB) on replica sites Start PDB Server Create Docking GridScore (receptor surface details) for a given receptor on home node. Pre-Staging Large Files required for Docking: Pre-stage Dock executables and PDB access client on Grid nodes, if required (e.g., dock.Linux, dock.SunOS, dock.IRIX64, and dock.OSF1 on Linux, Sun, SGI, and Compaq machines respectively). Use globus- rcp. Pre-stage/Cache all data files (~3-13MB each) representing receptor details on Grid nodes. This can can be done demand by Nimrod/G for each job, but few input files are too large and they are required for all jobs). So, pre- staging/caching at http-cache or broker level is necessary to avoid the overhead of copying the same input files again and again!

83 83 Protein Data Bank Databases consist of small molecules from commercially available organic synthesis libraries, and natural product databases. There is also the ability to screen virtual combinatorial databases, in their entirety. This methodology allows only the required compounds to be subjected to physical screening and/or synthesis reducing both time and expense.

84 84 Target Testcase The target for the test case: electrocardiogram (ECE) endothelin converting enzyme. This is involved in heart stroke and other transient ischemia. Is · che · mi · a : A decrease in the blood supply to a bodily organ, tissue, or part caused by constriction or obstruction of the blood vessels.

85 85 Nimrod/G in Action: Screening on World-Wide Grid

86 86 Any Scientific Discovery ? Did your collaborator invent new drug for xxxx? Not Yet Anyway, checkout the announcement of Nobel-prize winners for next year ?

87 Conclude with a comparison with the Electrical Grid ……….. Where we are ???? Courtesy: Domenico Laforenza

88 Alessandro Volta in Paris in 1801 inside French National Institute shows the battery while in the presence of Napoleon I Fresco by N. Cianfanelli (1841) (Zoological Section "La Specula" of National History Museum of Florence University)

89 89 ….and in the future, I imagine a worldwide Power (Electrical) Grid …... What ?!?! This is a mad man… Oh, mon Dieu !

90 90 2001 - 1801 = 200 Years

91 91 I think there is a world market for about five computers. Thomas J. Watson Sr., IBM Founder, 1943 Can we Predict its Future ?

92 92 Summary and Conclusions P2P and Grid Computing is emerging as a next generation computing platform for solving large scale problems through sharing of geographically distributed resources. Resource management is a complex undertaking as systems need to be adaptive, scalable, competitive, …, and driven by QoS. We proposed a framework based on computational economies and discussed several economic models for resource allocation and for regulating supply-and-demand for resources. Scheduling experiments on World Wide Grid demonstrate our Nimrod-G broker ability to dynamically lease or rent services at runtime based on their quality, cost, and availability depending on consumers QoS requirements. Economics paradigm for QoS driven resource management is essential to push P2P/Grids into mainstream computing!

93 93 Download Software & Information Nimrod & Parameteric Computing: Economy Grid & Nimrod/G: Virtual Laboratory/Virtual Drug Design: Grid Simulation (GridSim) Toolkit (Java based): World Wide Grid (WWG) testbed: Looking for new volunteers to grow Please contact me to barter your & our machines! Want to build on our work/collaborate: Talk to me now or email:

94 94 Thank You… Any ??

95 95 Further Information Books: High Performance Cluster Computing, V1, V2, R.Buyya (Ed), Prentice Hall, 1999. The GRID, I. Foster and C. Kesselman (Eds), Morgan-Kaufmann, 1999. IEEE Task Force on Cluster Computing Global Grid Forum IEEE/ACM CCGrid xy: CCGrid 2002, Berlin: Grid workshop -

96 96 Further Information Cluster Computing Info Centre: Grid Computing Info Centre: IEEE DS Online - Grid Computing area: Compute Power Market Project

Download ppt "Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia"

Similar presentations

Ads by Google