Presentation on theme: "Grid Monitoring Futures with Globus Jennifer M. Schopf Argonne National Lab April 2003."— Presentation transcript:
Grid Monitoring Futures with Globus Jennifer M. Schopf Argonne National Lab April 2003
2 My Definitions l Grid: –Shared resources –Coordinated problem solving –Multiple sites (multiple institutions) l Monitoring: –Discovery >Registry service >Contains descriptions of data that is available –Expression of data >Access to sensors, archives, etc.
3 What do *I* mean by Grid monitoring? l Different levels of monitoring needed: –Application specific –Node level –Cluster/site Level –Grid level l Grid level monitoring concerns data –Shared between administrative domains –For use by multiple people –(think scalability)
4 Grid Monitoring Does Not Include… l All the data about every node of every site l Years of utilization logs to use for planning next hardware purchase l Low-level application progress details for a single user l Application debugging data (except perhaps notification of a failure of a heartbeat) l Point-to-point sharing of all data over all sites
5 Overview of This Talk l Evaluation of information infrastructures –Globus Toolkit MDS2, R-GMA, Hawkeye –Insights into performance issues –(publication at HPDC 2003) l What monitoring and discovery could be l Next-generation information architecture –Open Grid Services Architecture mechanisms –Integrated monitoring & discovery arch for GT3
6 Performance and the Grid l Its not enough to use the Grid, it has to perform – otherwise, why bother? l First prototypes rarely consider performance (tradeoff with devt time) –MDS1–centralized LDAP –MDS2–decentralized LDAP –MDS3–decentralized Grid service l Often performance is simply not known
7 Globus Monitoring and Discovery Service (MDS2) l Part of Globus Toolkit, compatible with other elements l Used most often for resource selection –aid user/agent to identify host(s) on which to run an application l Standard mechanism for publishing and discovery l Decentralized, hierarchical structure l Soft-state protocols l Caching l Grid Security Infrastructure credentials
8 MDS2 Architecture
9 Relational Grid Monitoring Architecture (R-GMA) l Implementation of the Grid Monitoring Architecture (GMA) defined within the Global Grid Forum (GGF) l Three components –Consumers –Producers –Registry l GMA as defined currently does not specify the protocols or the underlying data model to be used.
10 GGF Grid Monitoring Architecture
11 R-GMA l Monitoring used in the EU Datagrid Project –Steve Fisher, RAL, and James Magowan, IBM-UK l Based on the relational data model l Used Java Servlet technologies l Focus on notification of events l User can subscribe to a flow of data with specific properties directly from a data source
12 R-GMA Architecture
13 Hawkeye l Developed by Condor Group l Focus – automatic problem detection l Underlying infrastructure builds on the Condor and ClassAd technologies –Condor ClassAd Language to identify resources in a pool –ClassAd Matchmaking to execute jobs based on attribute values of resources to identify problems in a pool
14 Hawkeye Architecture
15 Comparing Information Systems MDS2R-GMAHawkeye Info Collector Information Provider ProducerModule Info Server GRIS Producer Servlet Agent Aggregate Info Server GIIS(Archiver)Manager Directory Server GIISRegistryManager
16 Some Architecture Considerations l Similar functional components –Grid-wide for MDS2, R-GMA; Pool for Hawkeye –Global schema l Different use cases will lead to different strengths –GIIS for decentralized registry; no standard protocol to distribute multiple R-GMA registries –R-GMA meant for streaming data – currently used for NW data; Hawkeye and MDS2 for single queries l Push vs Pull –MDS2 is PULL only –R-GMA allows push and pull –Hawkeye allows triggers – push model
17 Experiments l How many users can query an information server at a time? l How many users can query a directory server? l How does an information server scale with the amount of data in it? l How does an aggregator scale with the number of information servers registered to it?
18 Testbed l Lucky cluster at Argonne –7 nodes, each has two 1133 MHz Intel PIII CPUs (with a 512 KB cache) and 512 MB main memory l Users simulated at the UC nodes –20 P3 Linux nodes, mostly 1.1 GHz –R-GMA has an issue with the shared file system, so we also simulated users on Lucky nodes l All figures are 10 minute averages l Queries happening with a one second wait between each query (think synchronous send with a 1 second wait)
19 Metrics l Throughput –Number of requests processed per second l Response time –Average amount of time (in sec) to handle a request l Load –percentage of CPU cycles spent in user mode and system mode, recorded by Ganglia –High when running small number compute intensive aps l Load1 –average number of processes in the ready queue waiting to run, 1 minute average, from Ganglia –High when large number of aps blocking on I/O
20 Performance of Information Servers vs. Number of Users
21 Experiment 1 Summary l Caching can significantly improve performance of the information server –Particularly desirable if one wishes the server to scale well with an increasing number of users l When setting up an information server, care should be taken to make sure the server is on a well-connected machine –Network behavior plays a larger role than expected –If this is not an option, thought should be given to duplicating the server if more than 200 users are expected to query it
22 Directory Server Scalability
23 Experiment 2 Summary l Because of the network contention issues, the placement of a directory server on a highly connected machine will play a large role in the scalability as the number of users grows l Significant loads are seen even with only a few users, it will be important that this service be run on a dedicated machine, or that it be duplicated as the number of users grows.
24 Information Service Throughput vs. Num. of Information Collectors
25 Experiment 3 Summary l Too many information collectors is a performance bottleneck l Caching data helps l Alternatively, register to more instances of information servers with each handling a subset of the collectors
26 Overall Results l Performance can be a matter of deployment –Effect of background load –Effect of network bandwidth l Performance can be affected by underlying infrastructure –LDAP/Java strengths and weaknesses l Performance can be improved using standard techniques –Caching; multi-threading; etc.
27 So what could monitoring be? l Basic functionality –Push and pull (subscription and notification) –Aggregation and Caching l More information available l More higher-level services –Triggers like Hawkeye –Viz of archive data like Ganglia l Plug and Play –Well defined protocols, interfaces and schemas l Performance considerations –Easy searching –Keep load off of clients
28 Topics l Evaluation of information infrastructures –Globus Toolkit MDS2, RGMA, Hawkeye –Throughput, response time, load –Insights into performance issues l What monitoring and discovery could be l Next-generation information architecture –Open Grid Services Architecture mechanisms –Integrated monitoring & discovery arch for GT3
29 Open Grid Services Architecture (OGSA) l Defines standard interfaces and behaviors for distributed system integration, especially: –Standard XML-based service information model –Standard interfaces for push and pull mode access to service data >Notification and subscription
30 Key OGSI concept - serviceData l Every service has its own service data –OGSA has common mechanism to expose a service instances state data to service requestors for query, update and change notification –Monitoring data is baked right in –Service-level concept, not host-level concept
31 serviceData l Every Grid Service can expose internal state as serviceData elements –An XML element of arbitrary complexity l Each service has a serviceData set –The collection of serviceData Elements (SDEs) l Example: state of a host is exposed as an SDE by GRAM. l Similar to MDS2 GRIS functionality, but in each service (rather than once per host)
32 Example: Reliable File Transfer Service Performance Policy Faults service data elements Pending File Transfer Internal State Grid Service Notfn Source Policy interfaces Query &/or subscribe to service data Fault Monitor Perf. Monitor Client Request and manage file transfer operations Data transfer operations
33 MDS3 Monitoring and Discovery System l Consists of a various components –Core functionality –Information providers –Higher level services –Clients
34 Core Functionality l Xpath support –XPath is a language that describes a way to locate and process items in XML docs by using an addressing syntax based on a path through the document's logical structure or hierarchy l Xindice support – native XML database l Registry support
35 Schema Issues l Need to keep track of service data schema –Avoid conflicts –Find the data easier l Should really have unified naming approach l All of the tool are schema-agnostic, but interoperability needs a well-understood common language
36 MDS3 Information Providers in June Release l All the data currently in core MDS2 l Full data in the GLUE schema for compute elements (CE) –Ganglia information provider for cluster data will also be available from Ganglia folks (with luck) l Service data from RFT, RLS, GRAM l GT2 to GT3 work –GridFTP server data –Software version and path data –Documentation for translating your GT2 information provider to a GT3 information provider
37 MDS3 Higher Level Products l Higher-level services can perform actions on service data collected from other services l Part of this functionality can be provided by a set of building blocks provided –Provider interface: GRIS-style API for writing information providers –Service Data Aggregator: set up subscriptions to data for other services, and publish it as a single data stream –Hierarchy Builder: allow for hierarchy of aggregators
38 MDS3 Index Server l Simplest higher-level service is the caching index service l Much like the GIIS in MDS2 l Will have configurablity like an GIIS hierarchy l Will also have PHP-style scripts, much as available today
40 Clients currently in GT3 l findServiceData command line client –Same functionality of grid-info-search l C bindings –Core C bindings provide findServiceData C function –findServiceData command line client gives an example of using it to parse out information (in this case, registry contents)
41 Service Data Browser l GUI client to display service data from any service l Extensible for data-specific visualization l A version was released with GT3 alpha l alpha/docs/infosvcs/sdbquickstart.html
42 Comparing Information Systems MDS2R-GMAHawkeyeMDS3 Info Collector Info. ProviderProducerModule Service (SDEs) Info Server GRIS Producer Servlet Agent Service (service data set) Agg. Info Server GIIS(Archiver)Manager Index Service Directory Server GIISRegistryManagerIndex Service
43 Is this enough? l No! l Many places where additional help developing MDS3 is needed
44 We Need More Basic Information l Interfaces to other sources of data –GPT data –Other monitoring systems –Others? l Service data from other components –Every service has service data –OGSA-DAI l Will need to interface on schema
45 We Will Need More GUIs and Clients l Additional GUI visualizers may be implemented to display service data specific to a particular port type (as part of service data browser) l Additional Client interfaces possibly –Integration into current portals, brokers
46 We Need More Higher Level Services l We have a couple planned –Archiving service –Trigger template
47 Post-3.0 release: Archiving Service l Will allow subscription to service data l Logging in a flexible way l Well defined interfaces for mining l Open questions: –Best way to store time-series of arbitrary XML? –Best way to query this archive? –Link to OGSA-DAI? –Link to other archivers?
48 Post-3.0 release: Trigger Template l Will provide a template to allow subscription to data, reasoning about that data, and a course of action to take place l Essentially, a gateway service between OGSA Notifications and some other notification framework, with filtering of notifications l Example: Subscribe to disk space information, send mail to sys admin when it reached 90% full l Needed: trigger template and several small examples of common triggers, and documentation for how users could extend them or write new ones.
49 Other Possible Higher Level Services l Site Validation Service l Job Tracking Service l Interfacing to Netlogger?
50 We Need Security l Need I say more?
51 Summary l Current monitoring systems –Insights into performance issues l What we really want for monitoring and discovery is a combination of all the current systems l Next-generation information architecture –Open Grid Services Architecture mechanisms –MDS3 plans l Additional work needed!
52 Thanks l Testbed/Experiment support and comments –John Mcgee, ISI; James Magowan, IBM-UK; Alain Roy and Nick LeRoy at University of Wisconsin, Madison;Scott Gose and Charles Bacon, ANL; Steve Fisher, RAL; Brian Tierney and Dan Gunter, LBNL. l This work was supported in part by the Mathematical, Information, and Computational Sciences Division subprogram of the Office of Advanced Scientific Computing Research, U.S. Department of Energy, under contract W Eng-38. This work also supported by DOESG SciDAC Grant, iVDGL from NSF, and others.
53 Additional Information l MDS3 technology coordinators –Ben Clifford –Jennifer Schopf l Zhang, Freschl and Schopf, A Performance Study of Monitoring and Information Services for Distributed Systems, to appear in HPDC 2003 –http://people.cs.uchicago.edu/~hai/hpdcv25.dochttp://people.cs.uchicago.edu/~hai/hpdcv25.doc l MDS-3 information –Soon at
55 Why Information Infrastructure? l Distributed, often complex, performance-critical nature of Grids & apps demands tools for –Discovering available resources –Discovering available sensors –Integrating information from multiple sources –Archiving and replaying historical information l These and other functions are provided by an information infrastructure l Many projects are concerned with design, deployment, evaluation, and application
56 Performance of GIS Information Servers vs. Number of Users
57 Performance of GIS Information Servers vs. Number of Users
58 Performance of GIS Information Servers vs. Number of Users
59 Performance of GIS Information Servers vs. Number of Users