Presentation on theme: "Robert M. Candey (Code 632) Bernard T. Harris (Code 583)"— Presentation transcript:
1 Connecting data and services with Web Services (SECAA/SPDF CDAWeb/SSCweb) Robert M. Candey (Code 632)Bernard T. Harris (Code 583)Reine A. Chimiak (Code 583)David B. Han (Code 586)Rita C. Johnson (QSS, Code 630)Colin A. Klipsch (QSS, Code 630)Tamara J. Kovalick (QSS, Code 630)Howard A. Leckner (QSS, Code 630)Michael H. Liu (Raytheon ITSS, Code 630)Robert E. McGuire (Code 632)NASA Goddard Space Flight Center, Greenbelt MD 20771aka SECAA
3 SECAA/SPDF providing services today Accessible via FTP, HTML browser, and now web services APIs and new clients that use the web services <http://spdf.gsfc.nasa.gov/>Space physics data access - Coordinated Data Analysis Web (CDAWeb) and Coordinated Data Analysis System (CDAS)World-wide data finding across public FTP sites and in CDAWeb (Public data file finder)Return URLs to files matching a range of time in many FTP and some HTTP data sites (filenaming format based on strptime % strings)Example XML (required tags in bold)<dataset ID="ace_mfi_h0_16sec_cdf_cdaweb" nssdc_ID="None” serviceprovider_ID="ac_h0_mfi"> <access filenaming="ac_h0_mfi_%Y%m%d_%Q.cdf" protocol="ftp" subdividedby="%Y" timerange_start=" :00:12" timerange_stop=" :59:46"> <URL>ftp://cdaweb.gsfc.nasa.gov/pub/istp/ace/mfi_h0</URL> </access> </dataset>Orbit locations - Satellite Situation Center Web (SSCweb)Data Format Translation web service (DTWS)Radbelts (OMNIweb) solar and magnetospheric indicesMore to come
4 What are web services?Services Oriented Architecture (vs. client-server)Distributed software to software communication, analogous to older technologies such as RPC, DCOM, CORBA, RMINo HTML or human interaction requiredCross-platform and language-independentEnables others to develop tools and services leveraging core logic and science data and orbit informationEveryone can use their own clients/tools
5 Lower level components of VOs as a set of interoperable services Strung together in many combinationsForm an integrated system that is much greater than the sum of its partsEasily extendableOpen to other systems and external applications by using standard distributed Application Program Interfaces (APIs)Based on XML and Simple Object Access Protocol [SOAP] standards and/or HTTP calling interfaceTie together existing data holdingsStandardize and simplify their useEnable much enhanced interoperability and data analysis.
6 How to use our web services: Example IDL program Uses IDL 6.0 IDL-Java bridge feature to call the SSCWeb services<http://sscweb.gsfc.nasa.gov/WebServices/RsiIdlExample.html>Get list of satellites on SSCweb and then get orbit locations for 1 day of Polar in GSE coordinatesConfigure IDL-Java bridge provided with the IDL softwareAdd JAXRPC jar files and sscWebServices-client.jar to CLASSPATHIDL> .run SSCexample.pro
8 How to setup your own web service: SECAA implementations Server SoftwareApache HTTPDApache Jakarta Tomcat 5 (triggers on “/WS/” in URL)Java Runtime Environment (JRE) 1.4.xSun JAX-RPC 1.1 runtime libraries (could replace with Java Axis)Apache Jakarta Commons/Net library (for FTP access)CDAS and SSC Web application implementing web servicesCDAS and SSC core (existing) softwareServices described in WSDL filesDevelopment SoftwareSun Java Web Services Developers Pack (JWSDP) 1.3Sun Java Developers Kit (JDK) 1.4.2Java Client SoftwareApplicationSun JAX-RPC 1.1 runtime librariesCDAS/SSC Web service client libraries
10 Lessons we are learning (and concerns) Our services require underlying standards, although still manual input for each dataset stored in master CDFsMany firewalls limit ports available and may impose timeout on sessions (had to modify Ethernet stack to send Keep-alive packets)Web service standards are evolving, subject to interpretation, and implementations may choose which options to supportDifferent data bind options (RPC-encoded, RPC-literal, document-literal)Different communication model options (synchronous RPC, asynchronous messaging)Handling of null Date values (pass by reference [Java] or value [.NET])Support for passing multi-dimension arrays (ASP .NET)Handling of empty arraysOur Web services were initially implemented before Web Services Interoperability (WS-I) Basic Profile 1.0 and use RPC-encoded binding style. We’ll be changing to document-literal to be compliant with WS-I and .NET.Need to be flexible and rewrite code but stay stable for clients (how to update API without breaking existing clients: add new methods?, add whole new API?)Difficult error handling (error codes) between applicationsNeed careful attention to how existing services are doneCreated duplicate functionality that we will have to merge software to remove (Perl interface to call web services)
11 Tie together with clients SECAA-provided or external clients (own customer of web services and dependent on service stability)HTML web page access always for simple accessTIPSOD (Tool for Interactive Plotting, Sonification and 3-D Orbit Display)CDAS Java client of getting CDAWeb data listings, plots, files (via CDAWlib) and URLs to files on other systems (via FTP) (functions to be merged soon):Original CDAS Client (no FTP support)<http://cdaweb/~rchimiak/cdasClient>CDAS FTP Client (no CDAWlib support)<http://cdaweb/~rchimiak/cdasFtpClient>Easy client install (WebStart)Translator pageVSPO also using CDAS Web services
12 Call to actionCooperative identification among major services and clients to aid in understanding usage and various reporting questions: Set user-agent string to identify calling routine/institution in all web services callsTry out our services and give us feedback
13 or contact Robert.M.Candey@nasa.gov Joint effort of the NASA GSFC Sun-Earth Connections Active Archive (SECAA) in the Space Physics Data Facility (SPDF) and National Space Science Data Center (NSSDC)See <http://spdf.gsfc.nasa.gov/> for links to these tools and servicesor contact
14 Distributed components of a space physics virtual observatory Work together via standard interfaces and metadata agreements to form a systemAppears as a single super-instrument providing geophysical measurements and models across time and spaceEnables researchers to easily and seamlessly analyze data from many more sources than possible before
15 Tie together services One scenario: Search bowshock event database Query indices database for times of no sun spotsCorrelate with spacecraft on same field linesSearch for data filesTranslate files into preferred formatPlot
17 Web Services Definition Language (WSDL) XML description of web service purpose, calling methods and locationWSDL file specifying web service can describe both SOAP and CGI interfacesCGI calls are lighter and less opaque but not as powerful as SOAP calls. See REST discussions by Mark Baker <http://webservices.devchannel.org/article.pl?sid=02/12/10/ >, Paul Prescod <http://www.prescod.net/> and <http://www.xml.com/lpt/a/2002/02/06/rest.html>Example CDAWeb CGI call for plots of the lat, long and altitude variables for NOAA5/6 datasets for 1 day:<http://cdaweb.gsfc.nasa.gov/cgi-bin/cdaweb/eval3.cgi?dataset=NOAA05_H0_SEM+NOAA06_H0_SEM&start=1980%2F11%2F17+03%3A00%3A00&stop=1980%2F11%2F18+03%3A00%3A00&auth=hello&index=radiation&action=plot&var=NOAA05_H0_SEM+Lat&var=NOAA05_H0_SEM+Lon&var=NOAA05_H0_SEM+Alt&var=NOAA06_H0_SEM+Lat&var=NOAA06_H0_SEM+Lon&var=NOAA06_H0_SEM+Alt
18 Coordinated Data Analysis Web (CDAWeb) HTML user interface supports graphics, digital listings, file retrieval and merged/subsetted CDF creation, and simultaneous multi-mission, multi-instrument selection and comparison of science data among a wide range of current and past space missions and ground-based facilitiesNow comprises >600 datasets and >1M files of science dataVery popular with 165k user sessions, 94k plots, 622k FTP requests, 67k ASCII listings, 586 CDF create requests, 2882 file retrieval requests in FY2004
19 Coordinated Data Analysis System (CDAS) Web services Simultaneous multi-mission, multi-instrument selection and comparison of science data among a wide range of current space missions from CDAWebDistributed programming interface so you can add access to many space science datasets, listings and plots to your science applicationHigher level web services planned for the near future
20 CDAWeb data available through FTP file access, including software and documentsWeb browserWeb services in additionLatter two provide data listings, plots, original data files and combined files
21 Data File FinderUsing just several pieces of metadata, we can return URLs to files matching a range of time in many FTP and some HTTP data sitesFilenaming format based on strptime % stringsExample XML (required tags in bold)<dataset ID="ace_mfi_h0_16sec_cdf_cdaweb" nssdc_ID="None” serviceprovider_ID="ac_h0_mfi"> <access filenaming="ac_h0_mfi_%Y%m%d_%Q.cdf" protocol="ftp" subdividedby="%Y" timerange_start=" :00:12" timerange_stop=" :59:46"> <URL>ftp://cdaweb.gsfc.nasa.gov/pub/istp/ace/mfi_h0</URL> </access> <mission_group ID="ACE" serviceprovider_ID="ACE"> <description short="Advanced Composition Explorer"/> </mission_group> <observatory ID="ACE" nssdc_ID="None" serviceprovider_ID="AC"> <description short="Advanced Composition Explorer"/> </observatory> <instrument ID="MAG" nssdc_ID="None" serviceprovider_ID="MAG"> <description short="ACE Magnetic Field Instrument"/> </instrument> <data_producer affiliation="Bartol Research Institute" name="N. Ness" title="None"/> <other_info> <link URL="http://www.srl.caltech.edu/ACE/ASC/level2/index.html" title="the ACE Science Center Level 2 Data website">Release notes and other info available at </link> </other_info> <spase:product_ID>ace_mfi_h0_16sec_cdf_spase</spase:product_ID> <instrument_type ID="Magnetic Fields (space)" serviceprovider_ID="Magnetic Fields (space)"/> <description short="H0 - ACE Magnetic Field 16-Second Level 2 Data - N. Ness (Bartol Research Institute)"/> </dataset>
22 What is Common Data Format (CDF)? Self-describing platform-independent scientific data format (object oriented like HDF5)Most widely used format in Space PhysicsISTP, Stereo, TIMASAvailable on all major computer platformsInterfaces supported: C, Fortran, Java, and PerlCDF FAQ <http://nssdc.gsfc.nasa.gov/cdf/html/FAQ.html>
23 Software for CDFIDL, MATLAB, IBM DX, FlexPro, Application Visualization SystemGoddard-developed Web-based systems leverage CDF powerCDAWlib <http://spdf.gsfc.nasa.gov/CDAWlib.html>Powerful set of IDL routines, including reading/writing CDFsAvailable at <ftp://cdaweb.gsfc.nasa.gov/pub/CDAWlib/>Standard CDF toolsCDFcompare, CDFconvert, CDFedit, CDFexport, CDFstats, SkeletonTable, SkeletonCDF distributed as part of the standard CDF distribution packageSKTeditor metadata editor applying the ISTP GuidelinesMakeCDF converts ASCII and binary custom formats into CDF
24 CDF Markup Language (CDFML) XML Markup Language describing Common Data Format (CDF) data and metadataCDFML file (XML file) is an ASCII representation of a CDF fileUsed as a mechanism for data interoperability with other science data formats (since translation between data formats via XSL is easy)
25 Data Translation Web Service (DTWS) Translation between one or more local or remote files into other data formatsFormats: CDF, netCDF, HDF4, HDF5, FITS,Interoperability with other data formats using XML: CDF Markup Language (CDFML), FITSML, XDF Version of CDFML (CDFX-ML), HDF in XML, Native Data Format (NDF) in XML, ASCII (data object mapping via Extensible Style Language (XSL))Input: Files to be translated, translation modeOutput: Direct or notification with URL links to translated files<http://nssdc.gsfc.nasa.gov/cdf/html/dtws.html>
26 Satellite Situation Center (SSCweb) Browser-based service for geocentric spacecraft locations with geophysical regions and mappings along magnetic field linesSupports multiple coordinate systemsExtensive business logic and comprehensive list of active and past space physics missionsSupports complex multi-satellite and ground conjunction queriesOutputs listings and graphsHowever, SSCweb currently limited:Static, 2D GIF images rendered by a browserNot easy to access from other applications
27 Tool for Interactive Plotting, Sonification and 3-D Orbit Display (TIPSOD) Shows spacecraft moving in time and 3D space as an interactive movie, while user changes orientation and scale (zoom and rotation)Interfaces with the new flexible SSC XML Web Services (expanding on operational system)Exposes relationships between spacecraft visually (in 3D) and temporallyEasy to download and use (Web Start downloads or updates TIPSOD client, Java and Java 3D with a few clicks from the browser)Runs on Windows, Unix, Linux, Mac OS X (requires Java3D)Additional functions planned, such as sonification and connections to CDAWeb science data and logic
28 Future and Improved Web Services Adding more I/O formats to CDAWeb (FITS, HDF, netCDF, IDL savesets)Smart client issues (WebStart)Other web servicesOMNI_near_earth field and plasma data and indicesATMOweb ionospheric dataModelWebOrbit_regions and cross-spacecraft queriesCOHO_heliospheric dataNMC_metadata (NSSDC Master Catalog)SPASE_dictionary (Space Physics Archive Search and Exchange)Community-wide client project? plugins? (NOAA plot tool)Grid support now that Globus 3 is OGSA/web services compatible
33 What’s important to you? What services and features do you want?What other web services are being developed? (VHO, VSO, EGSO, STARS)???
34 SECAA Goals Ease of use Open source, modular and reusable software Access to multi-spacecraft and multi-instrument dataPerformance (reasonable response times)Broad base of users???
35 Importance of Standards and Agreements Wide variety of science data file formats (CDF, netCDF, HDF, FITS, etc. and many custom formats) impedes science research across instruments and missions by greatly increasing learning and translating time for each datasetCustom binary and ASCII formats can be converted into CDF with the makeCDF tool for possible further conversion to other formatsOptional mapping templates handles dataset-specific translationsThese tools and service will increase interoperability and improve cross-mission dataset transparencyCooperative identification among major services and clients to aid in understanding usage and various reporting questions (set user-agent string)
36 Poster specific stuffFall 2004 AGU Special Session SF05 - Data and Services for Space and Earth Sciences(SF )
37 Importance of SECAA Web Services for the Integrated SEC Data Environment
38 Authors Robert M. Candey (Code 632) Reine A. Chimiak (Code 583) David B. Han (Code 586)Bernard T. Harris (Code 583)Rita C. Johnson (QSS, Code 630)Colin A. Klipsch (QSS, Code 630)Tamara J. Kovalick (QSS, Code 630)Howard A. Leckner (QSS, Code 630)Michael H. Liu (Raytheon ITSS, Code 630)Robert E. McGuire (Code 632)NASA Goddard Space Flight Center, Greenbelt MD 20771
39 AbstractThe Space Physics Data Facility (SPDF) at NASA GSFC has developed new and state-of-the-art, distributed programming (web services) interfaces to SPDF’s space science mission services and data. This work is an essential step in integrating these powerful and popular SPDF software systems (and the NASA mission data they now serve) with external science data and model software systems under a NASA Virtual Observatory paradigm for enhancing the scientific return of space physics research.The distributed components of a space physics virtual observatory work together via standard interfaces and metadata agreements to form a system that appears as a single super-instrument providing geophysical measurements and models across time and space, enabling researchers to easily and seamlessly analyze data from many more sources than possible before. We are providing a critical set of these lower level components, leveraging our data format expertise and our existing and very popular science and orbit data web-based services, Coordinated Data Analysis Web [CDAWeb] and Satellite Situation Center Web [SSCweb], by adding web services for orbit location, data finding across FTP sites and in CDAWeb, data file format translation, and display. These services tie together existing data holdings, standardize and simplify their use, and enable much enhanced interoperability and data analysis. A Java3D application, TIPSOD (Tool for Interactive Plotting, Sonification and 3-D Or!bit Display), developed in-house, makes use of these web services.CDAWeb, SSCWeb, CDF tools, and web services are a joint effort of the Sun-Earth Connections (SEC) Active Archive (SECAA) in the NASA GSFC Space Physics Data Facility (SPDF) and the National Space Science Data Center (NSSDC); see <http://spdf.gsfc.nasa.gov/>.
41 SECAA SOAP implementations SOAP uses HTTP calls (port 80) to Apache which sees "/WS/" in the path and calls Tomcat (or Tomcat use own port but lose logging and other features Apache provides). Other URLs are handled by Apache directlyTomcat <http://jakarta.apache.org/tomcat/> calls calls Java SOAP servlet to parse the incoming message, call the underlying tool, and martial the SOAP response <http://ws.apache.org/soap/docs/install/tomcat.html>Axis <http://ws.apache.org/axis/> called from Tomcat in place of Sun Jax-RPC