1. The Legendary Web & Grid Services Track Coordinators: Brian Wilson, Paul Davis Speakers: Mike Smorul, Gary Jackson, Richard Troy, Rudy Husar, Heather Weir, and others Gift Bags for first 20 participants!

2. Web & Grid Services Track Wilson, 01/04/2005 2 ESIP Federation Meeting, Jan. 4-6, 2005 Carbon Cycle n We’ll Answer These Provocative Questions : n Is SOAP the universal "glue" for distributed scientific data processing? n Will Web and Grid services ever converge (as in OGSI and WSRF)? n What exactly can one do with a Data Grid? n Can I really use the Globus Toolkit to steal compute cycles from somebody else's computer cluster? n How do I do distributed computing and still sleep at night? (i.e., with security, authentication, authorization, single sign-on, reliability, traceability, data provenance, load balancing, etc.) Promo

3. Web & Grid Services Track Wilson, 01/04/2005 3 ESIP Federation Meeting, Jan. 4-6, 2005 Carbon Cycle n Evaluate existing and emerging standards for Web and Grid services. n Web: SOAP, WSDL, UDDI, WSRF, WS-* n Data Grid: Storage Resource Broker (SRB) versus Globus Toolkit n Grid Computing: Globus versus others n Grid Workflow solutions n Investigate how we can implement them within the Federation. Goals

4. Web & Grid Services Track Wilson, 01/04/2005 4 ESIP Federation Meeting, Jan. 4-6, 2005 Carbon Cycle n Catalog existing Federation services n Federation Service Registry (Rudy Husar, W 9 AM) n GCMD SERF catalog (Heather Weir, W 9:30 AM) n How do we optimize service discovery? n Develop service chains or collaborations within the Federation n Interop between OpenDAP, WMS/WCS, & SOAP n SOAP Choreography / Grid Workflow n SOAP Service Advocacy (Wilson, Tu 1:30 PM) n SOAP as a Universal Interop Glue n Think (& design) in XML and ontologies n Develop a standard Federation architecture for Data Discovery & Data Query/Access as SOAP services Specific Proposals

5. Web & Grid Services Track Wilson, 01/04/2005 5 ESIP Federation Meeting, Jan. 4-6, 2005 Carbon Cycle n Data Grid Advocacy n Mike Smorul, Gary Jackson, Paul Davis (Tu 2:30 – 5 PM) n SRB versus Globus Toolkit n A prototype, SRB-based Federation Data Grid exists. How do we join? n Demonstrate Globus-style Grid Computing n Federation demo? n Grid Workflow Solutions n SciFlo Framework & Engine (Wilson, Tu 4 PM) n Producer-Archive Workflow Network (Smorul, Tu 4:30 PM) Specific Proposals

6. Web & Grid Services Track Wilson, 01/04/2005 6 ESIP Federation Meeting, Jan. 4-6, 2005 Carbon Cycle n [Enter your proposal here] n Get out of Iraq. Specific Proposals

7. Web & Grid Services Track Wilson, 01/04/2005 7 ESIP Federation Meeting, Jan. 4-6, 2005 Carbon Cycle Agenda – Tuesday PM

8. Web & Grid Services Track Wilson, 01/04/2005 8 ESIP Federation Meeting, Jan. 4-6, 2005 Carbon Cycle Agenda – Wednesday AM

9. Web & Grid Services Track Wilson, 01/04/2005 9 ESIP Federation Meeting, Jan. 4-6, 2005 Carbon Cycle Agenda – Wednesday PM

10. Web & Grid Services Track Wilson, 01/04/2005 10 ESIP Federation Meeting, Jan. 4-6, 2005 Carbon Cycle Agenda – Thursday AM

11. Web & Grid Services Track Wilson, 01/04/2005 11 ESIP Federation Meeting, Jan. 4-6, 2005 Carbon Cycle n 1st: Static HTML pages with Pictures! n Hyperlinks: click and jump! n Easy authoring of text with graphics (HTML layout). n Killer App: Having your own home page is hip. n Cons: Too static, One-way communication. n 2nd: Dynamic HTML with streaming audio & video n Browser as an all-purpose, ubiquitous user interface. n Fancy clients using embedded Java applets. n Killer App: Fill out your time card on-line. n Cons: Applets clunky, ease of authoring disappears, information is still HTML (semi-structured). n 3rd: SOAP-based Web Computing & Semantic Web n Exchange structured data in XML format (no fragile HTML); semantics (“meaning”) kept with data. n Programmatic interfaces rather than just GUI for a human. n Killer Apps: Grid Computing, automated data processing. Three Generations of the Web

12. Web & Grid Services Track Wilson, 01/04/2005 12 ESIP Federation Meeting, Jan. 4-6, 2005 Data Storage Algorithms CPU A Conceptual Grid Virtualization Portal “On-Demand” Scientist Networks

13. Web & Grid Services Track Wilson, 01/04/2005 13 ESIP Federation Meeting, Jan. 4-6, 2005 Carbon Cycle Evolving Grid Computing Standards [From Globus Toolkit “Ecosystem” presentation at GGF11 by Lee Liming] GT4

14. Web & Grid Services Track Wilson, 01/04/2005 14 ESIP Federation Meeting, Jan. 4-6, 2005 WS Architecture

15. Web & Grid Services Track Wilson, 01/04/2005 15 ESIP Federation Meeting, Jan. 4-6, 2005 Grid Apps Using Globus Middleware Web Browser Compute Server Globus MCS/RLS Data Viewer Tool Certificate Authority CHEF Chat Teamlet MyProxy CHEF Compute Server Resources implement standard access & management interfaces Collective services aggregate &/or virtualize resources Users work with client applications Application services organize VOs & enable access to other services Database service Database service Database service Simulation Tool Camera Telepresence Monitor Globus Index Service Globus GRAM Globus DAI Application Developer 2 Off the Shelf9 Globus Toolkit 4 Grid Community 4

16. Web & Grid Services Track Wilson, 01/04/2005 16 ESIP Federation Meeting, Jan. 4-6, 2005

17. Web & Grid Services Track Wilson, 01/04/2005 17 ESIP Federation Meeting, Jan. 4-6, 2005 Federation Data Grid Storage Resource Broker (SRB) –Query using Metadata Catalog (MCAT) –Transparent Replication –Archiving, Caching, Synchs, & Backups –Heterogeneous Storage –Container & Aggregated Data Movement –Bulk Data Ingestion –Version Control, Locking –Partitioned Data Managment Globus Toolkit –Monitoring & Discovery System (MDS) –Replica Location Service (RLS) –Other 3-letter words,...

18. Web & Grid Services Track Wilson, 01/04/2005 18 ESIP Federation Meeting, Jan. 4-6, 2005 REST versus SOAP REpresentational State Transfer (REST) –Roy Fielding’s Ph.D. Thesis –Everything is an object or document with a URI. –Problems: Hidden state: cookies, sessions, CGI db, etc. What about virtual datasets? Need to generate too many URI’s. Service Oriented Architecture (SOA) –Everything is a service. –What about stateful services?  WS-Resource Same Old Tradeoff –Like object-oriented programming versus functional programming (need both). –It’s both an object and a service.

19. Web & Grid Services Track Wilson, 01/04/2005 19 ESIP Federation Meeting, Jan. 4-6, 2005 Carbon Cycle n Open Your Databases! n Support Query on and Retrieval of ALL Metadata n Could support remote SQL queries on your entire database. n Distributed/Local Metadata - not centralized as in ECHO n Support XML formats with schemas & OWL/RDF semantics n Examples: Amazon & Google SOAP Web services n Open Your Architectures – Modularity, Reuse n Think in XML!! (design in XML first, not Java objects) n Expose algorithms & application modules via SOAP n Massive distributed computing via XML message exchange n Reuse modules from OpenDAP/DODS and WMS/WCS n Web Services Choreography / Grid Dataflow n Leverage WS-Coordination, -Context, -Security, etc. n Leverage WS-Resource Framework (WSRF, Globus v4) n Semantic Web Reasoning on top of WS substrate Vision

20. Web & Grid Services Track Wilson, 01/04/2005 20 ESIP Federation Meeting, Jan. 4-6, 2005 Thinking in XML  Design the XML metadata and data representations first!!! –Then create objects in your favorite programming language. –Then create objects in more languages for broader use.  4DRectilinearGrid datatype – -90, 90, 2 http://gsfc.nasa.gov/cgi-bin/dods/AIRS.hdf?Longitude http://gsfc.nasa.gov/cgi-bin/dods/AIRSgranule.hdf?TAirStd

21. Web & Grid Services Track Wilson, 01/04/2005 21 ESIP Federation Meeting, Jan. 4-6, 2005 New Paradigm for Scientific Computing  Use two or more channels –XML Messaging on one –Binary data protocol (like OpenDAP) on the other  Two channels can use different transports –XML Messaging over HTTP or dark P2P –OpenDAP over LambdaRail or scalable P2P caching network  XML Messaging Channel for: –Control, Service Chaining, Workflow, Metadata Exchange –Configuration of Binary channel  Binary Data Channel(s) –Out-of-band from XML point of view –Bulk data transfer, replication, or caching –Could be multi-protocol (OpenDAP, GridFtp, LambdaGrid)

22. Web & Grid Services Track Wilson, 01/04/2005 22 ESIP Federation Meeting, Jan. 4-6, 2005 Classifying Data Services - Dimensions Services Provided  Is it discovery, query, access, and/or subsetting? Degree of Metadata Support Richness of the Data Model Request Interface / Protocol Response Interface / Protocol Representation of Large Binary Data Objects Transport Layers (http, GridFtp, p2p, LambdaRail) Permanent Names & Version Support Service Chaining Semantic Web Support Interoperability with other services Maturity, Support, Open or Proprietary, Cost

23. Web & Grid Services Track Wilson, 01/04/2005 23 ESIP Federation Meeting, Jan. 4-6, 2005 Data Query/Access Services  Services Chain goes beyond simple data access –Time & Geolocation Query  yielding data inventory –Query on Quality Flags –General Metadata Query  catalog satisfying conditions –Data Access - by object or granule ID and variable name –Data Slicing - as in OpenDAP –Data Subsetting – by lat, lon, alt, & time ranges –Parameter Subsetting – select only desired physical variables –Data Reformatting – choose output format as in WMS/WCS –On-Demand Grid Computations – grid diff in GraDS/DODS –Variable Bundles – external metadata from scientist –Return Composite Data Objects 4DRectilinearGrid, 4DCurvilinearGrid (swath) XML & Native Binary representations –User Composites – add custom object to data model –Semantic Support Use generic variable names Reason about variables to be bundled

24. Web & Grid Services Track Wilson, 01/04/2005 24 ESIP Federation Meeting, Jan. 4-6, 2005 SciFlo Data Query/Access (SOAP) Services QueryByGeoLocation(startTime, endTime, lat, lon, timeTolerance, distanceTolerace, variable, metadataGroups) –Returns granule ID’s and geolocation info. for AIRS L2 swaths that intersect lat/lon point or are near enough. QueryByGeoRegion(startTime, endTime, lat/lon region,...) –Returns granule ID’s and geolocation info. for AIRS L2 swaths that intersect the lat/lon region or are near enough. QueryByMetadata(variable, ListOfConstraints, groups) –Returns granule ID’s and selected metadata for AIRS granules that satisfy the metadata constraint expression: (min1 <= field1 <= max1 and/or min2 <= field2 <= max2...). FindDataById(IdList, UrlOptions) –Given list of unique ID’s, returns list of ftp, http, or DODS URL’s pointing to the granules (files). Uses cache and redirection server. -- Other semantic interfaces possible

25. Web & Grid Services Track Wilson, 01/04/2005 25 ESIP Federation Meeting, Jan. 4-6, 2005 SciFlo Data Query/Access (SOAP) Services GetMetadata(type, groups) –Returns metadata describing scientific domain, dataset info, list of metadata fields, generic name translation table, location of XML schema documents, location of related ontologies (OWL/RDF), etc. GetHelp(type, groups) –A SOAP service that itself returns help documentation describing the available SOAP services. --These Services Combined Can Support: –Data Catalog, General Query, Data Access, Generic Names, –Data Discovery by Domain and Dataset Keywords –Type checking via XML schemas –Hooks to semantic web

26. Web & Grid Services Track Wilson, 01/04/2005 26 ESIP Federation Meeting, Jan. 4-6, 2005 Data Discovery to Inventory Level SOAP Services just described provide: –Domain description Metadata –Dataset Info. and Keywords –Generic VariableName Lookup Table –Query by Metadata or just Time & GeoRegion Layer more semantics on top of these services –Dataset Discovery by keyword search –Bind in additional metadata provided by scientist –Tie ontologies into keywords and genericVariableNames –Semantic Inference: AIRS and MODIS both provide atmosphericTemperature and cloudProperties. Hmmm. Compare them. –XML/SOAP provides substrate; possibilities only limited by ontology development.

27. Web & Grid Services Track Wilson, 01/04/2005 27 ESIP Federation Meeting, Jan. 4-6, 2005 Compare/Contrast Template Services Provided Implemented Standards / Protocols Inputs, Outputs, & Parameters Metadata Maturity –Number of Users, Data Volume Support –Level, Type, Availability Open / Proprietary –Open? Proprietary? Published? Costs? Activities –Interoperability? Future Plans –Services to be Implemented

28. Web & Grid Services Track Wilson, 01/04/2005 28 ESIP Federation Meeting, Jan. 4-6, 2005 Carbon Cycle n Automate large-scale, multi-instrument science processing by authoring a dataflow document that specifies a tree of executable operators. n iEarth Visual Authoring Tool n Distributed Dataflow Execution Engine n Move operators (executables) to the data. n Built-in reusable operators provided for many tasks such as subsetting, co-registration, regridding, data fusion, etc. n Custom operators easily plugged in by scientists. n Leverage convergence of Web Services (SOAP) with Grid Services (Globus v3.2). n Hierarchical namespace of objects, types, & operators. n sciflo.data.EOS.AIRS.L2.atmosphericParameters n sciflo.operator.EOS.coregistration.PointToSwath SciFlo Dataflow Engine

29. Web & Grid Services Track Wilson, 01/04/2005 29 ESIP Federation Meeting, Jan. 4-6, 2005 Carbon Cycle n We’ll Answer These Provocative Questions : n Is SOAP the universal "glue" for distributed scientific data processing? n Will Web and Grid services ever converge (as in OGSI and WSRF)? n What exactly can one do with a Data Grid? n Can I really use the Globus Toolkit to steal compute cycles from somebody else's computer cluster? n How do I do distributed computing and still sleep at night? (i.e., with security, authentication, authorization, single sign-on, reliability, traceability, data provenance, load balancing, etc.) Promo