Presentation is loading. Please wait.

Presentation is loading. Please wait.

Globus MDS for Developers Mike DArcy Laura Pearlman USC Information Sciences Institute.

Similar presentations


Presentation on theme: "Globus MDS for Developers Mike DArcy Laura Pearlman USC Information Sciences Institute."— Presentation transcript:

1 Globus MDS for Developers Mike DArcy Laura Pearlman USC Information Sciences Institute

2 September 11, 2006MDS for Developers2 Agenda l Review of MDS services and components l Publishing information u Publishing resource properties with the new Resource Property Provider component. u Using ExecutionAggregatorSource to monitor things other than WSRF services l Acting on Information u Using the Trigger Service to monitor both WSRF and non-WSRF services so that user programs may be invoked when specific conditions are detected. l Querying/displaying information u Querying the Index Service u Using the new TargetedXPath query dialect u Producing custom WebMDS views

3 September 11, 2006MDS for Developers3 MDS Services l Index Service u Collects information u Publishes its data as resource properties l Trigger Service u Collects information using the same mechanism as the Index Service u Executes trigger actions when conditions are met.

4 September 11, 2006MDS for Developers4 Data Flow in MDS Execution Aggregator Source Trigger Service Index Service Resource property requests Subscription / notiification Clients: wsrf-query, WebMDS, Java API, C API Trigger actions anything WSRF Service Query Aggregator Source WSRF Service With rpProvider Information Provider Resource property requestsexecution

5 September 11, 2006MDS for Developers5 Resource Property Provider l The new RPProvider component provides a modular plug-in API for creating Resource Properties from various user-defined input sources. l This mechanism allows for the easy inclusion of custom XML content, represented as a WS- ResourceProperty, into GT4 services with significantly reduced integration overhead. l In many cases, no new code may be required at all. For example, periodically reading a XML file and storing the contents in the IndexService requires no custom coding and minimal configuration.

6 September 11, 2006MDS for Developers6 RPProvider API Basics l The goal is the output of XML, bound to a QName that represents the name of the WS Resource Property that the data will be published as. l This task is handled by 3 sub-components: u Element Producers: perform the generation of the XML data to be used as the resource property value(s). u Element Transforms: optionally perform post-processing and validation on the output of element producers. u Execution Engine: reads configuration and performs the periodic execution of ResourcePropertyProviderTasks, which first executes a specified element producer, and then optionally executes any specified element transform.

7 September 11, 2006MDS for Developers7 RPProvider: Element Producers l Element producers are required only to implement the single method of a Java interface of the form: public Element getElement(String[] args) throws Exception; l RPProvider comes with a default set of element producers that can be used for a variety of common tasks. u ExternalProcessElementProducer: executes a external process, then attempts to parse the STDOUT stream of that process into a valid XML document. u FileElementProducer: reads data directly from a XML file. u URLElementProducer: reads data by using an HTTP Get request to fetch a XML file.

8 September 11, 2006MDS for Developers8 RPProvider: Element Transforms l Element transforms are required only to implement a single method of a Java interface of the form: public Object transformElement(Element source, String[] args) throws Exception; l The return type of the transformElement method is expected to be either a DOM Element, or a XML serializable Java bean object generated from WSDL or XSD with Axis WSDL2Java. l RPProvider includes a basic XSLT-based transform for general use, XSLTFileElementTransform, which reads in a XSLT file as specified by arg[0] and applies it to the result element generated by an element producers getElement method.

9 September 11, 2006MDS for Developers9 RPProvider: Execution Engine l The execution engine of the RPProvider component is a set of Java classes which handles the creation and periodic execution of Java TimerTasks that generate data by invoking configured Element Producers and Element Transforms. l The execution engine is implemented as a GT4 operation provider. The implementation for the operation provider is contained primarily within the ResourcePropertyProviderCollection.java class. l The execution engine supplies utility classes that implement the ReflectionResourceProperty interface. These classes provide default mechanisms for returning element producer output as either a single value or an array of values.

10 September 11, 2006MDS for Developers10 RPProvider: Execution Engine, continued l Pre-existing Factory and Singleton type services can include the operation provider without code modification by editing the service descriptor in the server-config.wsdd file. l Service resources created at runtime must have support for RPProvider compiled into the code by creating an instance of ResourcePropertyProviderCollection at resource initialization time.

11 September 11, 2006MDS for Developers11 RPProvider: Configuration l Adding RPProvider support to a Factory or Singleton service via server- config.wsdd: share/schema/mds/index/index_service.wsdl

12 September 11, 2006MDS for Developers12 Adding RPProvider Support to a Resource l Adding RPProvider support directly into a resource implementation class: ResourcePropertyProviderCollection rpCollection = new ResourcePropertyProviderCollection ( path/to/config/file/rpConfig.xml, // a full path to the configuration file this.getResourcePropertySet(), // the resource property set to add into true // initialize immediately after construction ); l Note: This code fragment assumes that the class referred to by the this variable implements the org.globus.wsrf.ResourceProperties interface.

13 September 11, 2006MDS for Developers13 RPProvider usage in GT4 l The 4.1.x GRAM ManagedJobFactoryResource uses the latter approach to integrate resource-level RPProvider support for the Fork, PBS, and other GRAM job scheduler factory resources. l A refactored version of the GLUEResourceProperty.java class, based on the RPProvider API, is now used by GT4.1.x GRAM to publish the GLUECE Resource Property. l This newer version of GLUEResourceProperty supports more flexible configuration of the underlying information providers that compose the GLUECE RP for a given scheduler type. l Scheduler providers can now be easily swapped out and replaced with developer versions.

14 September 11, 2006MDS for Developers14 RPProvider: Configuration l Sample RPProvider Configuration File Format as defined in rpprovider.xsd mds:GLUECE org.globus.mds.usefulrp.rpprovider.GLUEResourceProperty org.globus.mds.usefulrp.glue.GangliaElementProducer localhost org.globus.mds.usefulrp.rpprovider.transforms.GLUEComputeElementTransform org.globus.mds.usefulrp.rpprovider.producers.SchedulerInfoElementProducer libexec/globus-scheduler-provider-pbs org.globus.mds.usefulrp.rpprovider.transforms.GLUESchedulerElementTransform 300

15 September 11, 2006MDS for Developers15 RPProvider Example, Overview l Putting it all together: Adding a MOTD Resource Property to the DefaultIndexService u Step 1: Create the MOTD XML file that will contain the value(s) of the resource property. u Step 2: Create a RPProvider configuration file for the MOTD resource property. u Step 3: Configure the DefaultIndexService to use the ResourcePropertyProviderCollection operation provider. u Step 4: Start the container and query the MOTD resource property to verify that things are functional.

16 September 11, 2006MDS for Developers16 RPProvider Example, Step 1 l Step 1: Create MOTD XML file u Our sample XML file will publish a Message of the Day as a resource property named: {http://mds.globus.org/MOTD}MessageOfTheDay u The motd.xml sample file contains the following data: The system will be going down for maintenance today at 3:00AM EST for approximately 10 minutes. We apologize for any inconvenience. Thu Aug 10 15:05:00 PDT 2006 Ricky Bobby

17 September 11, 2006MDS for Developers17 RPProvider Example, Step 2 l Step 2: Create a configuration file for the MOTD resource property u Create the file motd-sample-config.xml in the $GLOBUS_LOCATION/etc/globus_wsrf_mds_index directory. u The MOTD RP configuration file should contain the following data: motd:MessageOfTheDay org.globus.mds.usefulrp.rpprovider.SingleValueResourcePropertyProvider org.globus.mds.usefulrp.rpprovider.producers.FileElementProducer /full/path/to/motd.xml 86400

18 September 11, 2006MDS for Developers18 RPProvider Example: Step 3 l Step 3: Configure the DefaultIndexService to use the ResourcePropertyProviderCollection operation provider. share/schema/mds/index/index_service.wsdl

19 September 11, 2006MDS for Developers19 RPProvider Example: Step 4 l Step 4: Start the container and query the MOTD resource property to verify that things are functional. u Start container: $GLOBUS_LOCATION/bin/globus-start-container [-nosec] u Issue a GetResourceProperty query: $GLOBUS_LOCATION/bin/wsrf-get-property –s {http://mds.globus.org/MOTD}MessageOfTheDay u The query command output should contain the following data: The system will be going down for maintenance today at 3:00AM EST for approximately 10 minutes. We apologize for any inconvenience. Thu Aug 10 15:05:00 PDT 2006 Ricky Bobby

20 September 11, 2006MDS for Developers20 ExecutionAggregatorSource l The ExecutionAggregatorSource provides a flexible data ingestion mechanism for WS-MDS Service Groups that are based on the Aggregator Framework. This includes: u Index Service u Trigger Service l ExecutionAggregatorSource functions by executing an external process and then parsing the process STDOUT stream into a XML file. l The result XML file is stored as the value of the Content ResourceProperty for the ServiceGroupEntry resource that is created when the Service Group Add operation is invoked against the service.

21 September 11, 2006MDS for Developers21 ExecutionAggregatorSource Example l Prerequisite: Install the GridFTP information provider u Ensure that the information provider file gridftp-banner.pl is located in the following directory: $GLOBUS_LOCATION/libexec/aggrexec l Step 1: Enable the usage of gridftp-banner.pl by editing the executableMappings field of the AggregatorConfiguration block of the jndi-config.xml file for the Index or Trigger. u $GLOBUS_LOCATION/etc/globus_wsrf_mds_index/jndi-config.xml u $GLOBUS_LOCATION/etc/globus_wsrf_mds_trigger/jndi-config.xml l Step 2: Create or modify a mds-servicegroup-add registration file to contain the instructions to run the GridFTP information provider periodically against a specifed GridFTP server address.

22 September 11, 2006MDS for Developers22 ExecutionAggregatorSource Example, continued l Configuring the Index and Trigger services to use ExecutionAggregatorSource to monitor GridFTP u Step 3: Start the container and run the mds-servicegroup- add program to register a new service group entry that will run the GridFTP information provider. u Step 4: Issue a wsrf-query to check the status of the provider: $GLOBUS_LOCATION/bin/wsrf-query –s

23 September 11, 2006MDS for Developers23 ExecutionAggregatorSource Example: Step 1 l Step 1: Enable execution of gridftp-banner.pl u For security reasons, all executable programs used by the ExecutionAggregatorSource must be enumerated by the adminstrator of the Globus installation prior to starting the container. u The administrator is required to map a logical probe name to the physical filename of the information provider. Furthermore, the information provider executable file must by located in $GLOBUS_LOCATION/libexec/aggrexec (or symlinked from that location). u In this case, edit the AggregatorConfiguration block of the service jndi- config.xml as follows: … executableMappings gridftp-info=gridftp-banner.pl, aggr-test=aggregator-exec-test.sh, pingexec=example-ping-exec …

24 September 11, 2006MDS for Developers24 ExecutionAggregatorSource Example: Step 2 l Step 2: Create or modify a mds-servicegroup-add registration file for the GridFTP information provider https://yourhost:8443/wsrf/services/DefaultIndexService ftp://yourhost: gridftp-info

25 September 11, 2006MDS for Developers25 ExecutionAggregatorSource Example: Step 3 l Step 3: Start the container and run the mds-servicegroup- add program to register a new service group entry that will run the GridFTP information provider. u Start container: $GLOBUS_LOCATION/bin/globus-start-container -nosec u Run mds-servicegroup-add: $GLOBUS_LOCATION/bin/mds-servicegroup-add my-gridftp- registration.xml

26 September 11, 2006MDS for Developers26 ExecutionAggregatorSource Example: step 4 l Step 4: Issue a wsrf-query to check the status of the information provider: u Query : l $GLOBUS_LOCATION/bin/wsrf-query –s "//*[ local-name()='Address' and contains(.,'ftp://wiggum.mcs.anl.gov:2811') ]/../..//*[ local- name()='AggregatorData' ] " u Query Output:./gridftp-banner.pl skynet-login GridFTP ftp://wiggum.mcs.anl.gov:2811 UP

27 September 11, 2006MDS for Developers27 ExecutionAggregatorSource: Conclusion u Useful for Non-WSRF Services u Developers can implement their own information providers using ExecutionAggregatorSource in a similar manner. u Benefits of this approach: l Information modeled as ServiceGroupEntry content. l WS-Lifetime interfaces used to manage information lifetime for each ServiceGroupEntry resource. l WSRF and Non-WSRF Service information can co-exist in the same Service Group, in this case, the DefaultIndexService.

28 September 11, 2006MDS for Developers28 Trigger Service l Using the Trigger Service to monitor other services so that user programs may be invoked when specific conditions are detected. u The Trigger Service may be used to monitor the results of data aggregation operations for both WSRF and Non-WSRF services. u The Trigger Service can be configured to probe information directly, or can also be configured to probe summary information from another service already aggregating information, such as the IndexService. u This example will show how to use the DefaultTriggerService to monitor the DefaultIndexService in order to detect and trigger a status notification when a GridFTP service (as described in the previous section) goes down.

29 September 11, 2006MDS for Developers29 Trigger Service: Configuration l Configuring the containers DefaultTriggerService: u Step 1: Enable the action script that will be used by the DefaultTriggerService. u Step 2: Edit the action script to specify the addressee and optional event description. u Step 3: Edit the registration file that will be used with mds- servicegroup-add that will be used to register the trigger configuration to the DefaultTriggerService. u Step 4: Run the mds-servicegroup-add command to register and activate the trigger and query the DefaultTriggerService for status about the trigger state.

30 September 11, 2006MDS for Developers30 Trigger Service Example: Step 1 l Step 1: Enable the action script that will be used by the DefaultTriggerService u Like the ExecutionAggregatorSource, for security reasons all trigger action scripts used by the TriggerService must be enumerated by the administrator of the Globus installation prior to starting the container. u The administrator is required to map a logical action name to the physical filename of the executable trigger action. Furthermore, the information provider executable file must by located in the following directory: $GLOBUS_LOCATION/libexec/trigger u In this case, edit the TriggerConfiguration block of the Trigger Service jndi- config.xml as follows: … executableMappings -trigger= -trigger.sh, test-trigger=test-trigger-action.sh, glue-trigger=glue-trigger- action.sh, echo-trigger=echo-trigger-action.sh …

31 September 11, 2006MDS for Developers31 Trigger Service Example, Step 2 l Step 2: Edit the action script to specify the addressee and optional event description. u Create a file called -trigger.sh file in $GLOBUS_LOCATION/libexec/trigger l NOTE: Make sure the file has permissions allowing it to be executed by the user account that the container process will run as. u The following file can be used as a template for those systems that have the mailx program installed. #!/bin/sh CC= SUBJECT="Your event has been triggered" DATE=`date '+20%y-%m-%d'` MSG="The GridFTP server at ftp://wiggum.mcs.anl.gov:2811 is down." echo "$MSG" | mailx -s "$SUBJECT" $CC $TO cat <

32 September 11, 2006MDS for Developers32 Trigger Service Example, Step 3 l Step 3: Edit the registration file that will be used with mds- servicegroup-add that will be used to register the trigger configuration to the DefaultTriggerService wssg:Entry //test[target/url='ftp://wiggum.mcs.anl.gov:2811' and result/status='DOWN'] -trigger true

33 September 11, 2006MDS for Developers33 Trigger Service Example, Step 4 l Step 4: Run the mds-servicegroup-add command to register and activate the trigger and query the DefaultTriggerService for status about the trigger state. u Run mds-servicegroup-add using the trigger registration file. u Query the DefaultTriggerService for the trigger state: l wsrf-query -s "//*[local-name()='TriggerStatusArray'] l Sample Query Output: 07b3f db-bf4d-89fc4ea14c8f //test[target/url='ftp://wiggum.mcs.anl.gov:2815' and result/status='DOWN'] T03:40:52.575Z T03:41:52.668Z T03:41:52.564Z Your event has been triggered DOWN The trigger action was executed successfully

34 September 11, 2006MDS for Developers34 Querying the Index Service l Standard QueryResourceProperties WSRF request u takes a dialect and a query expression l GT4 supports the standard xpath dialect: u Examples: l / - returns entire xml tree l //glue:Cluster returns all elements of that type u Limitations: cant specify namespace mappings (e.g., cant guarantee that glue will be recognized in second example) u Workaround: use xpath functions like local-name l //*[local-name()='Cluster']

35 September 11, 2006MDS for Developers35 QueryResourceProperties Overview 1. Set up a porttype object. u Server address u security options 2. Create query arguments. u dialect u query 3. Run query request

36 September 11, 2006MDS for Developers36 Set Up the Porttype Object String myURL = https://myhost:8443/wsrf/services/DefaultIndexService EndpointReferenceType endpoint = new EndpointReferenceType(); endpoint.setAddress(new AttributedURI(myURL)); WSResourcePropertiesServiceAddressingLocator locator = new WSResourcePropertiesServiceAddressingLocator(); QueryResourceProperties_PortType port; try { port = locator.getQueryResourcePropertiesPort(endpoint); } catch (ServiceException e) { // Do something. } ((Stub)port)._setProperty( Constants.GSI_ANONYMOUS, Boolean.TRUE);

37 September 11, 2006MDS for Developers37 Create Query Arguments (XPath) String queryString = /; QueryExpressionType query = new QueryExpressionType(); try { query.setDialect(WSRFConstants.XPATH_1_DIALECT); } catch (MalformedURIException e) { // Do something. } query.setValue(queryString); QueryResourceProperties_Element request = new QueryResourceProperties_Element(); request.setQueryExpression(query);

38 September 11, 2006MDS for Developers38 Run Query QueryResourcePropertiesResponse response; try { response = port.queryResourceProperties(request); } catch (RemoteException e) { // Do something. } // response is an array of MessageElement System.out.println( AnyHelper.toSingleString(response));

39 September 11, 2006MDS for Developers39 targetedXPath l GT4.1.x has a new dialect, targetedXPath u XPath dialect is still there l Allows the user to specify namespace mappings for use in queries l Usage is the same as with the XPath dialect, except that setting up the query args is different

40 September 11, 2006MDS for Developers40 Create Query Arguments (targetedXPath) TargetedXPathQueryType targetedQuery = new TargetedXPathQueryType(); NamespaceMappingType nsMap[] = new NamespaceMappingType[1]; nsMap[0] = new NamespaceMappingType(); nsMap[0].setMappedName(glue); nsMap[0].setNamespace( new URI(http://mds.globus.org/glue/ce/1.1)); targetedQuery.setNamespaceMappings(nsMap); targetedQuery.setQueryString(//glue:Cluster); QueryExpressionType query = new QueryExpressionType(); try { query.setDialect(TargetedXPathConstants.TARGETED_XPATH_DIALECT); } catch (MalformedURIException e) { // Do something. } query.setValue(targetedQuery);

41 September 11, 2006MDS for Developers41 WebMDS

42 September 11, 2006MDS for Developers42 WebMDS: How it Works Web Browser Web Form WebMDS plugin XSL Transform Index Service HTML Output Resource property query HTTP query with form arguments

43 September 11, 2006MDS for Developers43 WebMDS and HTML Forms For a URL like l info arg tells WebMDS where to get data u Looks for indexinfo file in WebMDS conf directory l info is the only mandatory argument l Optional xsl arg tells WebMDS where to get the XSL transform u Uses same config mechanism as info arg l Other arguments are available

44 September 11, 2006MDS for Developers44 WebMDS Plugins l Resource Property Plugin u Performs resource property queries u Can be configured to accept an EPR argument. l Implemented by openEndedQuery conf file u Can be configured to query a specific EPR l Implemented by indexinfo conf file l File Plugin u Reads a file u Useful for getting XML transforms u Also used for testing

45 September 11, 2006MDS for Developers45 WebMDS openEndedQuery Resource property query of the local default index service org.globus...NodeXmlSource class org.globus...ResourcePropertyQueryNodeSource allowUserEndpoints true

46 September 11, 2006MDS for Developers46 WebMDS Resource Property Plugin Resource property query of the local default index service org.globus...xmlDomNode.NodeXmlSource class org.globus.mds...ResourcePropertyQueryNodeSource endpoint https:// :8443/wsrf/services/DefaultIndexService

47 September 11, 2006MDS for Developers47 WebMDS File Plugin XSL file to show service group summary information org.globus...FileXmlSource true file xslfiles/servicegrouptable.xsl

48 September 11, 2006MDS for Developers48 Implementing a New XSL Transform 1. Write an XSL transform 2. Create a config file in $GLOBUS_LOCATION/lib/webmds/conf 3. Use a web browser to test the transform

49 September 11, 2006MDS for Developers49 WebMDS – Write an XSL Transform l XSL is documented at l There are many examples of XSL transforms in $GLOBUS_LOCATION/lib/webmds/xslfiles l Install the transform in $GLOBUS_LOCATION/lib/webmds/xslfiles/local u For purposes of this example, call the file mytransform.xsl

50 September 11, 2006MDS for Developers50 WebMDS – Configure an XSL Transform l Create a config file in $GLOBUS_LOCATION/lib/webmds/conf u For this example, call the file mytransform XSL file to implement my custom transform org.globus...FileXmlSource true file xslfiles/local/mytransform.xsl

51 September 11, 2006MDS for Developers51 WebMDS – Test Your XSL Transform l Use mytransform as the xsl argument in an HTML form or URL u =indexinfo&xsl=mytransform

52 September 11, 2006MDS for Developers52 WebMDS – Adding Summary Info to Service Group Summary l The Service Group summary view (xsl=servicegroupxsl) has a summary line for each entry in the Index. l The format of the summary data is based on the data type u GRAM data contains the number of queues, RFT data includes the number of files transferred, etc. u For unrecognized types, only the type name and EPR are reported

53 September 11, 2006MDS for Developers53 WebMDS – Adding Summary Data to Service Group Summary, continued l To tell the service group summary transform how to deal with a new data type, edit the file sgtypes.xsl in $GLOBUS_LOCATION/lib/webmds/xslfiles/local l Define a transform that will produce a suitable summary line for your type u See examples in the directory $GLOBUS_LOCATION/lib/webmds/xslfiles/sgtypes u The transform should act on a service group entry that contains content of your type u Fortunately, theres a template… l If you have several custom types, you may want to define them in separate files and have sgtypes.xsl include them.

54 September 11, 2006MDS for Developers54 WebMDS – sgtypes.xsl MyLabel Summary data, which probably has one or more select statements of the form

55 September 11, 2006MDS for Developers55 How to Contribute l Join the mailing lists and offer your thoughts! u u u l Offer to contribute your information providers, higher level services, or WebMDS XSL transforms l If youve got a complementary monitoring system – think about being an Incubator project (contact or come to the talk on Thursday)

56 September 11, 2006MDS for Developers56 Meet the Developers Session at Globus Alliance Booth (152A-P7) September 12 8:00am - 9:00am "Java WS Core and Security (C, Java)" -- Olle Mulmo, Jarek Gawor, Rachana Anantakrishnan 11:30am -12:30pm "RLS" -- Rob Schuler, Ann Chervenak 12:30pm -1:30pm "MDS" -- Mike DArcy, Laura Pearlman 3:00pm - 4:00pm Resource Management (GRAM, Virtual Workspaces and Dynamic Accounts)" – Stu Martin, Peter Lane, Tim Freeman, Kate Keahey 6:00pm - 7:00pm "C WS Core" -- Joe Bester 7:00pm - 8:00pm "Python WS Core" -- Joshua Boverhof September 13 8:00am - 9:00am "GridShib" -- Von Welch, Ton Scavo, Tim Freeman 11:30am - 12:30pm "GT Installation and Administration" -- Charles Bacon 12:30pm - 1:30pm "MyProxy" -- Jim Basney 3:00pm - 4:00pm "GridFTP, XIO, RFT" -- John Bresnahan, Ravi Madduri

57 COME CELEBRATE WITH US! In appreciation of your support of all things Globus over the past decade, you are cordially invited to the Globus 10th Birthday Party. When: Monday, September 11, :00pm, immediately following Ian Fosters Globus State of the Union Keynote. Where: The convention center concourse, in the center of the GlobusWORLD / GridWorld conference activity. What: Food, drinks, music, friends and lots of fun!


Download ppt "Globus MDS for Developers Mike DArcy Laura Pearlman USC Information Sciences Institute."

Similar presentations


Ads by Google