Presentation on theme: "Grid Business Symposium 2005, Seoul, Korea"— Presentation transcript:
1 Grid Business Symposium 2005, Seoul, Korea Recent Advances in Grid Computing and Business Models: A Gridbus PerspectiveWW GridGrid Business Symposium 2005, Seoul, KoreaRajkumar BuyyaGrid and Distributed Systems (GRIDS) Laboratory Dept. of Computer Science and Software Engineering The University of Melbourne Melbourne, Australia
2 Outline Introduction Global Grids and Challenges Grid Initiatives Utility Networks and Grid ComputingGlobal Grids and ChallengesGrid InitiativesWorld-wide with Australia and India PerspectiveIntroduction to Gridbus Project and Grid EconomyGrid Service BrokerArchitecture, Design and ImplementationPerformance Evaluation: Experiments in Creation and Deployment of Applications on Global GridsA Case Study in High Energy PhysicsEconomy-based Scheduling in Data GridsSummary
10 Grid Computing in Australia (Courtesy: Jihyoun Park, SNU Visitor to Melbourne) AcademiaGovernmentCollaborationIndustry
11 Academic activities 1 University laboratories for Grid computing - Uni. of Melbourne(GRIDS lab): Gridbus (GridSim, GMD, GridBank, Alchemi, ..), Master of Engineering in Distributed Computing- Monash Uni.: GriddlsS (Legacy SW to the computational grid), Nimrod-G- Australian national Uni. (Internet Futures Group)- Sydney Uni.(ViSLAB): high performance visualization &computing- Uni. of Adelaide (DHPC Group): DISCWorld- Queensland Uni. of Technology (PLAS): G2 (.NET based)2 Grid Infrastructure ProjectsAPACGrid, National Neurosciece Facility, Australian Virtual Observatory, several state level facilities (VPAC, TPAC, SAPAC, QPSF, IVEC)3 Grid Applications* Asia Pacific Bioinformatics Network/ Virtual Drug Design: Molecular Modeling for Drug Design on P2P Grid/ HEPGrid: High Energy Physics and the Grid Network/ Access Grid/Australian Computational Earth Systems Simulator/.* Recently 30 more applications are funded as part of ARC e-Research* Govt. has formed “National e-Research Coordination Committee”.
12 Grid Computing in India AcademiaGovernmentCollaborationIndustry(majority focus on Grid integration)
13 Grid Computing in India: Academic and Industrial Activities Academic and Government Initiatives:TIFR, IITM, Anna University, IITD, UoH, etc.C-DAC’s Garuda – Ministry of ITSoftware Companies in India:Top 4 Indian IT Companies: Satyam, Infosys, TCS (Tata Consultancy Service), and Wipro.Oracle 10g, IBM, HP, Sun ertc. have a large Grid development centers in Bangalore, India.Satyam is leading the pack in Grid Business push:Grid Practice Centre with top management support.Singned MoU with Melbourne University and extensively using Gridbus in powering applications.Also contributing the development of Gridbus technologies (e.g., Alchemi) – SEI CMM Level 5 principles.Application Verticals: Manufacturing, Security, Life Sciences, Finance
15 Australian and Indian Grid Efforts Compared Korea: Is it like Australia or India?
16 On Demand Utility Computing The Gridbus Melbourne: Enable Leasing of ICT Services on DemandDistributed DataWWGGridbusWorld Wide Grid!On Demand Utility Computing
17 The Gridbus Project: http://www.gridbus.org A multi-institutional “Open Source” R&D Project with focus on:Architecture, Specification, and Open Source Reference Implementation.Service-Oriented Grid, Utility Computing & Distributed Data and Computation EconomyScaling from Desktops, Clusters, Cluster Federation, Enterprise Grids to Global Grids.Alchemi: Harnessing .NET/Windows-based ResourcesGrid Market Directory and Web ServicesGrid Bank: Accounting and Transaction ManagementVisual Tools for Creation of Distributed ApplicationsWorkflow Composition and Deployment ServicesData Grid Brokering and Grid Economy ServicesData Replication StrategiesGridSim Toolkit: Enhanced to support Data Grid, Reservation, etc.Libra: SLA-based Allocation of Cluster ResourcesCoupling of Clusters and Computational EconomyWWG: Global Data Intensive Grid TestbedApplication Enabler Projects:High-Energy Physics , Astronomy, Brain Activity Analysis – Osaka U., Natural Language Processing, Portfolio Analysis – Spain, BioGrid - WEHI (via APACGrid), SensorGrid (NICTA), Medical Imaging (HFI)Supported by:
18 Grid Economy: Methodology for Sustained Resourced Sharing and Managing Supply-and-Demand for ResourcesFor resource providers, they provide service with different goalsFor consumers, they have different requirmentsHow to manage grid service supply-and-demand is hard taskSo how can consumers find a resource depending on their requirements.And what type of resource that providers provide and how can they serve consumers within certain QoS.For example, when many consumers want to access same resource, who should be served firstHow to manage supply-demand balance is another challenge.
19 New challenges of Grid Economy Resource OwnersHow do I decide prices ? (economic models?)How do I specify them ?How do I translate price to resource allocation ?How do I enforce them ?How do I advertise & attract consumers ?How do I do accounting and handle payments?…..Resource ConsumersHow do I decide expenses ?How do I express QoS requirements ?How do I trade between timeframe & cost ?How do I map jobs to resources to meet my QoS needs?They need mechanisms and technologies for value expression, value translation, and value enforcement.
20 Grid Entities and Architecture Grid consumerGSP site schedulerResource ownersbrokerGSP global schedulerGSP site schedulerResource ownersMarketMakeraccountingEnd usersPrivate enterprisesNational providers
21 A Reference Service-Oriented Architecture for Utility Grids Data CatalogueGrid BankGrid Market ServicesInformation ServiceSign-onHealth MonitorInfo ?Grid ExplorerGrid Node N……Programming EnvironmentsSecureJob ControlAgentApplicationsSchedule AdvisorGrid Node1QoSPricing AlgorithmsTrade ServerTradingTrade ManagerAccountingResourceReservationMisc. services…Deployment AgentJobExecResource AllocationGrid Resource BrokerStorageR1R2…RmGrid MiddlewareServicesGrid ConsumerGrid Service Providers
23 Alchemi: .NET-based Enterprise Grid Platform & Web Services Alchemi ManagerWeb ServicesInternetAlchemi UsersInternetlike ModelGeneral PurposeDedicated/Non-dedicate workersRole-based Security.NET and Web ServicesC# ImplementationGridThread and Job Model ProgrammingEasy to setup and useWidely in use!Alchemi Worker Agents
24 Many users in Universities: See next for an example. Some Users of AlchemiTier Technologies, USALarge scale document processing using Alchemi frameworkSatyam Computers Applied Research Laboratory, IndiaMicro-array data processing using Alchemi frameworkCSIRO, AustraliaNatural Resource ModelingThe University of Sao Paulo, BrazilThe Alchemi Executor as a Windows Servicestochastix GmbH, GermanyAsynchronous Excel Tasks using ManagedXLL and Alchemi .Net Grid Computing framework.The Friedrich Miescher Institute (FMI) for Biomedical Research, SwitzerlandPatterns of transcription factors in mammalian genesMany users in Universities: See next for an example.
25 On Demand Assembly of Services: Putting Them All Together Visual Application ComposerApplication CodeExplore data1Data SourceDataResults+ Cost Info10Grid Resource Broker2Bill12(Instruments/distributed sources)Data Replicator(GDMP)Data Catalogue5ASP CatalogueGrid Info ServiceGrid Market Directory463Job8Results97GSP(e.g., UofM)PE(e.g., VPAC)(e.g., IBM)CPUor PEGrid Service (GS)(Globus)AlchemiGSGTSCluster SchedulerGSP(Accounting Service)Gridbus GridBank11Cluster SchedulerPEGrid Service Provider (GSP) (e.g., CERN)
26 The Gridbus Grid Service Broker for Data Grid Applications Builds on the Nimrod-G Computational Grid Broker and Computational Economy [Buyya, Abramson, Giddy, Monash University, ]AndExtends its notion for Data and Service Grids
28 Gridbus Broker and Remote Service Access Enablers Home Node/PortalGridbusBrokerfork()batch()-PBS-Condor-SGE-Alchemi-XGridCredential RepositoryMyProxyPortletsData CatalogAlchemi-PBS-Condor-SGEGlobusJob managerfork()batch()GridbusagentData StoreGatewayUnicore-PBS-Condor-SGE-XGridSSHfork()batch()GridbusagentAccess TechnologySRBGrid FTP
29 Gridbus Services for eScience applications Application Development Environment:XML-based language for composition of task farming (legacy) applications as parameter sweep applications.Task Farming APIs for new applications.Web APIs (e.g., Portlets) for Grid portal development.Threads-based Programming InterfaceWorkflow interface and Gridbus-enabled workflow engine.Resource Allocation and SchedulingDynamic discovery of optional computational and data nodes that meet user QoS requirements.Hide Low-Level Grid Middleware interfacesGlobus, Alchemi, Unicore, NorduGrid, XGrid, etc.
30 Figure 3 : Logging into the portal. Click Here for DemoFigure 3 : Logging into the portal.Drug DesignMade Easy!
31 Economy-based Data Grid Scheduling CLICK HERE TO SKIP IF RUNNING OUT of TIMEHigh Energy Physics as eScience Application Case Study
33 Case Study: Event Simulation and Analysis B0->D*+D*-KsSimulation and Analysis Package - Belle Analysis Software Framework (BASF)Experiment in 2 parts – Generation of Simulated Data and Analysis of the distributed dataAnalyzed 100 data files (30MB each) were distributed among the five nodes
34 Resources Used and their Service Price OrganizationNode detailsRoleCost (in G$/CPU-sec)CS,UniMelbbelle.cs.mu.oz.au 4 CPU, 2GB RAM, 40 GB HD, LinuxBroker host, Data host, NWS serverN.A. (Not used as a compute resource)Physics, UniMelbfleagle.ph.unimelb.edu.au 1 CPU, 512 MB RAM, 40 GB HD, LinuxReplica Catalog host, Data host, Compute resource, NWS sensor2CS, University of Adelaidebelle.cs.adelaide.edu.au4 CPU (only 1 available) , 2GB RAM, 40 GB HD, LinuxData host, NWS sensorANU, Canberrabelle.anu.edu.au 4 CPU, 2GB RAM, 40 GB HD, LinuxData host, Compute resource, NWS sensor4Dept of Physics, USydbelle.physics.usyd.edu.au 4 CPU (only 1 available), 2GB RAM, 40 GB HD, LinuxVPAC, Melbournebrecca-2.vpac.org180 node cluster (only head node used), LinuxCompute resource,NWS sensor6
36 Deploying Application Scenario A data grid scenario with 100 jobs and each accessing remote data of ~30MBDeadline: 3hrs.Budget: G$ 60KScheduling Optimisation Scenario:Minimise TimeMinimise CostResults:
37 Grid and Gridbus Technologies for Various Grid (Market) Types free tradingPublic computing(Alchemi)Private enterprises(Libra, Gridbus, Globus)Sharing ModelNational provider (Globus, Gridbus,..)regulationscientificcommercialApplication Category
38 (5) IT services as the fifth utility (water, electricity, gas, telephone, IT) eScienceeBusinesseGovernmenteHealthMultilingualeEducation…
39 Summary and Conclusion Grids exploit synergies that result from cooperation of autonomous entities:Resource sharing, dynamic provisioning, and aggregation at global level.Grid Economy provides incentive needed for sustained cooperation.Grid Network has potential to serve as Cyberinfrastructure for Utility ComputingGrids offer enormous opportunities for realizing eScience and eBusiness at global level.
40 Gridbus Project - http://www.gridbus.org Any Questions ?Gridbus Project -
41 Thanks for your attention! The Gridbus Cooperation!
43 Some Open Research Challenges Value expression, translation, and enforcement mechanisms and supporting Grid technologies for:different economic models for spot markets and futuresapplication modelsDynamic Pricing SchemesInteraction Protocols for Service NegotiationMicro payments and Digital CurrenciesScheduling AlgorithmsProgramming Environments for Building Information Utility ApplicationsLast, but not least:Dispute Managements and Legal IssuesTaxation (consult your National Taxation Office)State, national, and international boundariesTax returns!
44 This talk is designed to answer: How can Grid technologies support the emergence and operation of virtual enterprises?How can Grid shared resources be treated, brokered, and marketed as ICT ‘commodities’ or ‘futures’ among networked organisations?What kind of Grid architecture is needed for handling such market mechanisms in an automated fashion?How can Grid economies map the evolution of networked business models?
45 What do Grids aim for and how to support them. Grids aim at exploiting synergies that result from cooperation of autonomous distributed entities. Synergies include:Resource sharing“On-demand” Virtual Enterprises creationAggregation of resources on demand.For this cooperation to be sustainable, participants needs to have (economic) incentive.Therefore, “incentive” mechanisms should be considered as one of key design parameters of Grid computing.
47 Appropriate Market Model for different market types highVariable price auctionCommodity marketWillingness to PayPosted price oligopolylowweakstrongDemand elasticity
48 Realising Market-based Grid: Minimal New Components Grid Market Directory ServicesGrid Trading Services –for different economic modelsGrid Metering ServicesGrid Accounting and Payment ServicesGrid Service Broker
49 Deadline (D) and Budget (B) Constrained Scheduling Algorithms Execution Time (D)Execution Cost (B)Compute GridData GridCost OptLimited by DMinimizeYesCost-Time OptMinimize if possibleTime OptLimited by BConservative-Time OptLimited by B, jobs have guaranteed minimum budget