1. Copyright Jet Propulsion Laboratory1 Object Oriented Data Technology Dan Crichton Steve Hughes JPL/Caltech CCSDS XML Workshop Oxford, England April 3-5, 2002

2. Copyright Jet Propulsion Laboratory2 Overview  Funded in 1998 by the NASA Office of Space Science to research technology in middleware and distributed object computing architectures to enable interoperability of data management systems

3. Copyright Jet Propulsion Laboratory3 OODT Focus Areas  Information Architectures – Metadata Management – Data Languages (i.e. XML)  Distributed Architectures – Messaging – Distributed Object Computing – P2P Architectures  Development of an architecture for distributed information management and retrieval  Application to the Planetary Data System to support Mars Odyssey Data Distribution

4. Copyright Jet Propulsion Laboratory4 Middleware Web, GUI, console, 2D/3D visualization, immersion product data, real-time data Applications s/w, h/w, models Middleware Data User Interface N. Lamarra

5. Copyright Jet Propulsion Laboratory5 Middleware Review  History – Pre-web: IDE (dev), client/server (apps) – Post-web: n-tier c/s, thin/thick, frameworks, … – Communications: RPC, MOM, Web, … – Applications: Desktop and Remote, … – Architecture: Open Reference Model, HLA, TOGAF, …  Standards – O/S: POSIX, Unix, Windows – Communications: TCP/IP, OSI, DCE/ONC, HLA/RTI, HTTP – DOC: CORBA/COSS, DCOM/COM+, Java RMI – Data access: O/JDBC, EJB persistence, STEP, HTML, XML – Security: SSL, SSH, SHTTP, IPSec, PKI, CDSA, BioAPI, etc. – Directory: X.500, LDAP – Systems Management: CIM, SNMP, CMIS/CMIP – GUI: X11, CDE, Java-Swing

6. Copyright Jet Propulsion Laboratory6 OODT Principles  Location Independence  Information Hiding  Model driven architecture  Scalable, Extensible  Client APIs for locating and accessing distributed data products

7. Copyright Jet Propulsion Laboratory7 Model Driven Architecture  Decouple solutions from platform specific technology  Allows data architecture and technology architecture to evolve independently  The technology architecture supports the data architecture (i.e. it supports the model)  Apply architecture to different problem spaces

8. Copyright Jet Propulsion Laboratory8 OODT Architecture  Address two key architectural focuses – Technology Architecture Basic communication between geographically distributed data systems Framework for plugging in new data systems – Data Architecture Standard method for describing data (profile) Standard method for exchanging data (XMLQuery)

9. Copyright Jet Propulsion Laboratory9 OODT Standards  The architecture for accessing distributed resources is based on the following standards: – W3C Extensible Markup Language (XML) 1.0 (Second Edition) (Recommendation 6 October 2000) – W3C HTML 4.01 Specification (Recommendation 24 December 1999) – W3C XHTMLT 1.0: The Extensible Hypertext Markup Language (Recommendation 26 January 2000) – ISO/IEC 11179 - Specification and Standardization of Data Elements, Parts 1-6, ISO/IEC specification, http://www.iso.ch/iso. – Dublin Core Metadata Initiative. The Dublin Core Element Set Version 1.1, Dublin: DMCI, July 1999.

10. Copyright Jet Propulsion Laboratory10 OODT Technology Implementation  Implemented in Java  Data layer implemented with XML  Uses CORBA for remote object invocation – We use JacORB  SSL for data encryption  Uses a standardized XML DTD (XMLQuery) as the messaging format (See http://oodt.jpl.nasa.gov/dtd/xmlquery.dtd)  Supports a variety of client access methods – Java API – HTTP – C++ API

11. Copyright Jet Propulsion Laboratory11 Why XML?  XML doesn’t provide a “silver bullet,” but it does allow us to refocus the problem on metadata – Metadata is a key to interoperability  XML is language neutral  Allows the designer to separate the data and the transport (re: CORBA vs XML-over-CORBA) – Transport mechanism and data are not tied together Could be XML/HTTP Simpler deployments Simpler interfaces Allows technologies to grow and change independently  Real value of XML is providing a content language for documenting the information architecture

12. Copyright Jet Propulsion Laboratory12 XML and CORBA  XML over CORBA/IIOP module jpl { module user { interface UserManager { string do(string xml); }; }; }; Doe  CORBA method module jpl { module user { interface UserManager { User findUser(string name); }; interface User { String getName(); }; }; };

13. Copyright Jet Propulsion Laboratory13 OODT Data Architecture  Use Profiles to describe data resources – System Profiles – Dataset Profiles – Product Profiles  A profile is just a metadata description of the resource defined by an XML DTD (see oodt/dtd/prof.xml) – Uses Dublin Core – Uses ISO 11179

14. Copyright Jet Propulsion Laboratory14 What is a profile? Profiles describe and provide the location for anything of interest –Things of interest (data and system resources) Data Systems, Data Sets, Web Sites, etc – Written as XML documents –Grouped Data Product profiles grouped by data set All others grouped into single database –Provide sufficient information to determine if the resource can resolve a user query

15. Copyright Jet Propulsion Laboratory15 Profile Overview Implementation A profile includes a pointer to the resources A profile has three main sections.  The profAttributes (Profile Attributes) section describes the profile.  The resAttributes (Resource Attributes) section describes the resource by using a common set of keywords (Dublin Core)  The profElement (Profile Element) section describes the resource by using domain specific attributes (PDS keywords)

16. Copyright Jet Propulsion Laboratory16 Profile DTD <!ELEMENT profiles (profile+)> <!ELEMENT profile (profAttributes, resAttributes, profElement*)> <!ELEMENT profAttributes (profId, profVersion*, profTitle*, profDesc*, profType*, profStatusId*, profSecurityType*, profParentId*, profChildId*, profRegAuthority*, profRevisionNote*, profDataDictId*)> <!ELEMENT resAttributes (Identifier, Title*, Format*, Description*, Creator*, Subject*, Publisher*, Contributor*, Date*, Type*, Source*, Language*, Relation*, Coverage*, Rights*, resContext*, resAggregation*, resClass*, resLocation*)> <!ELEMENT profElement (elemId*, elemName, elemDesc*, elemType*, elemUnit*, elemEnumFlag*, (elemValue | (elemMinValue, elemMaxValue))*, elemSynonym*, elemObligation*, elemMaxOccurrence*, elemComment*)>

17. Copyright Jet Propulsion Laboratory17 ?xml version="1.0"?>.. 1.1.1.1.102 profile ACTIVE EDRN_MOFFITT_PRODUCT_SERVER EDRN H. Lee Moffitt Cancer Center Product Server text/html A product server providing access to a database of clinical sample information for the Early Detection Research Network (EDRN) at the H. Lee Moffitt Cancer Center. en NIH.NCI.EDRN.MOFFITT granule system.productServer iiop://oodt.jpl.nasa.gov:10000/NIH.NCI.EDRN.MOFFITT.PRODUCT_SERVER STUDY_PARTICIPANT_ID EDRN Participant ID CHARACTER F

18. Copyright Jet Propulsion Laboratory18 Adopt an XML Query Structure for Messaging  Support queries across distributed resources  Provide a standard for inter- communication between heterogeneous applications and computing environments  Describe both queries and results

19. Copyright Jet Propulsion Laboratory19 XMLQuery F Implemented in XML at http://oodt.jpl.nasa.gov/dtd/xmlquery.dtd F Defined as follows ä The query description ä The results F Result 1 ä Result Header ä Result Data F Result 2 ä Result Header ä Result Data... F Result N ä Result Header ä Result Data ä Final results are aggregated from all profile/product servers

20. Copyright Jet Propulsion Laboratory20 XML Query Example (1 of 2) 1.2.3.4… OODT_XML_QUERY Query Example Query for TARGET_NAME = MARS QUERY ACTIVE UNKNOWN 2000-05-12 JSH V1.2 Updated for new prof.dtd 1.2.3.5… ATTRIBUTE BROADCAST N/A 100 0 TARGET_NAME = MARS

21. Copyright Jet Propulsion Laboratory21 XML Query Example (2 of 2) elemName TARGET_NAME LITERAL MARS RELOP EQ

22. Copyright Jet Propulsion Laboratory22 OODT Components to Support the Architecture  Name Server  Query Server  Archive Server  Profile Server  Product Server  Server Manager

23. Copyright Jet Propulsion Laboratory23 OODT Software Components  Name Server – CORBA name server that maps an object to a location (i.e. ern:NIH.NCI.EDRN.MOFFITT.PRODUCT_SERVER can be communicated at edrn.moffitt.edu:7877) – All implementations of CORBA, DCE, DCOM, SOAP, etc use a name server – Only needs to be run once at a central site  Query Server – Implemented by OODT – Multi-threaded server for managing queries to distributed servers – Aggregates results from multiple servers – Only needs to be run once at a central site

24. Copyright Jet Propulsion Laboratory24 Archive Server  Provides a server framework for – Adding products remotely to a generic archive – Retrieving products remotely from a generic archive  A generic data model that is driven by the metadata model – Captures a catalog of archived products using well defined data elements  Uses XML to label data products  Extract Profiles of archived products

25. Copyright Jet Propulsion Laboratory25 Profile Server  Profile Server – Used to identify the existence and location of resources in a distributed data environment – Two types of profile servers Dynamic Profile Server –Front ends an existing database catalog to extract the resources and format them as profiles –Implemented as a Java Interface class Static Profile Server –Manages its own local resources – XMLQuery is the messaging structure to query and return profiles – As many profile servers can be run as needed to scale the system. Typically one per catalog.

26. Copyright Jet Propulsion Laboratory26 Profile Server Overview Profile Servers serve profiles –Allow search and retrieval of profiles –Retrieves from profile databases Static profiles stored as XML documents Dynamic profiles generated from information stored in databases –Can be distributed for local management and scalability at a site

27. Copyright Jet Propulsion Laboratory27 Profile Server Requirements 1)A profile server shall search and retrieve profiles from a profile database 2)For search, a profile server shall allow any profile attribute as a query constraint. Profile attributes include those from the profile element, resource attribute, and profile attribute sections of the profile document. 3)For retrieval, a profile server shall return matching profiles. The user can request the complete profile or any subset of the profile.

28. Copyright Jet Propulsion Laboratory28 Profile Server XML Interface 1)Input: XMLQUERY – A profile server accepts an XMLQUERY XML document as input. User query constraints have been encoded into the WHERE clause of the XMLQUERY. The requested profile attribute results have been encoded into the SELECT clause. 2)Output: XMLQUERY – A profile server inserts the set of matching profiles in the RESULTS clause of the XMLQUERY.

29. Copyright Jet Propulsion Laboratory29 Profile Server Servlet Interface User calls optional servlet using “DIS” query Servlet creates XMLQUERY and calls predetermined profile server Results returned in XMLQUERY Query Server passes parallel query to each product server that has a match htpp://ginger.fhcrc.org/servlet/jpl.oodt.servlets.Profile_Servlet? keywordQuery={CANCER_SPECIMEN_TYPE = BLOOD &object= NIH.NCI.EDRN.DMCC.PROFILE_SERVER &mimeType=XML &mimeType=*/*}

30. Copyright Jet Propulsion Laboratory30 Profile Server Distributed Architecture

31. Copyright Jet Propulsion Laboratory31 Product Server  Product Server – Used to return a product using the XMLQuery messaging structure – Implemented as a Java Server – Contains one or more query handlers Query handler implemented as a Java interface class Query handlers loaded at server start time Defined in a local configuration file – Can use any method to access and return data

32. Copyright Jet Propulsion Laboratory32 Product Server Implementation  The Product Server plugs into the OODT framework and manages the “handshake” between the data system and the OODT system.  Extensible by dynamically loading objects at runtime which are specific to the data system model  Queries and results are passed using an OODT XML Query structure Product Server Flat file accessor Unusual accessor Query Result Generic Server Implementation Class File Sys Database

33. Copyright Jet Propulsion Laboratory33 Profile Server Distributed Profile Servers Product Servers for Planetary Data System HTTP, IIOP, Java, C++ APIs ASU Themis Products jdbc:// Data Source Interface For Dynamically Loaded Query Handlers Java Server Framework PDS.ASU.THEMIS.PRODUCT_SERVER Mars Odyssey Imaging Products Java Server Framework PDS.IMAGING.PRODUCT_SERVER XMLQUERY Data Source Interface For Dynamically Loaded Query Handlers HTTP, IIOP, Java, C++ APIs XMLQUERY

34. Copyright Jet Propulsion Laboratory34 Layered OODT Product Server Architecture PC/Sun,etc Hardware Unix/Windows operating system Java OODT CORBA/SSL Local Product servers Provided by JPL Provided by hosting site Stability

35. Copyright Jet Propulsion Laboratory35 Server Manager  Server Manager – Distributes software – Manages distributed software – Remote Management and Debugging – Automatic process management – APIs to allow tools to inquire about server status (i.e PDS Themis Node at ASU is up or down)

36. Copyright Jet Propulsion Laboratory36 Challenges with Distributed System Integration  Off site development – Debugging by telephone  New databases and database drivers – Inexperience led to programming errors  Institutional Challenges – Finicky configurations – Mysterious network issues

37. Copyright Jet Propulsion Laboratory37 Possible Solutions  Remote PC control – Unacceptable security risk at some sites  Mandate Unix – Lack of IT experience at member sites – Retooling time too great  Put remote management into the product server itself

38. Copyright Jet Propulsion Laboratory38 Server Manager  Manages servers – Server manager runs at each site  Client/server implementation – XML-RPC with HTTP authentication  One-click installation – Well, almost: install Java first  Capture Log Files  Perform restarts

39. Copyright Jet Propulsion Laboratory39 Remote Control Application  Authenticates with a server manager  Enables remote administration of server processes – View log files and process output – Set Java runtime parameters – Update files – Define, start, stop processes – Execute upgrade scripts

40. Copyright Jet Propulsion Laboratory40 Remote Control User Interface

41. Copyright Jet Propulsion Laboratory41 Software Distribution Environment  Installation – First, you install Java – Windows users download and run one self- installer pds-setup.exe Displays typical Windows-style wizard installer – Unix/Mac users download a tarball and follow instructions in a README file

42. Copyright Jet Propulsion Laboratory42 OODT Configurations  oodtrc.xml – Local configuration file that contains Server names List of profile servers Logging methods Timeout methods Database configuration parameters …

43. Copyright Jet Propulsion Laboratory43 OODT Applied to PDS  Provides a common messaging infrastructure for PDS  Provides data abstraction and location independence for PDS Nodes – Common tools can look across the PDS sites – Heterogeneous implementations are standardized  Uses the PDS Data Dictionary  Uses existing PDS databases and catalogs

44. Copyright Jet Propulsion Laboratory44 Generic Architecture Query Server Profile Server Site 3 Product Server XMLQuery Profile Server QueryClient Web server Plugins Web Server XMLQuery Site 1 DB Site 2 DB Site 3 DB Web I/F Desktop I/F XMLQuery Name Server Client Environment Distributed Repositories Site 2 Product Server Site 1 Product Server

45. Copyright Jet Propulsion Laboratory45 Profile/Location Server Profile DB Default Data Set Browsers Custom Data Set Browsers Middleware PDS-Distribution Delivery - D01 Architecture Single Point of Entry Data Set View Data Set Selection Page Data Set Catalog Other Node Resources Product Server GRS RS Profile Server RS Product Server THEMIS Product Server SPICEMARIE Custom Data Set Browsers Custom Data Set Browsers

46. Copyright Jet Propulsion Laboratory46 OODT Projects  PDS D01 will use OODT to support Mars Odyssey Distribution (October 2002)  PDS D02 will rollout OODT to all of PDS  OODT building a data architecture for the National Cancer Institute’s (NCI) Early Detection Research Network integrating up to 40 sites by 2005  OODT Supporting development of a JPL Institutional Data Architecture with Data Management Services

47. Copyright Jet Propulsion Laboratory47 OODT References  OODT Overview – http://oodt.jpl.nasa.gov http://oodt.jpl.nasa.gov  OODT Papers – http://oodt.jpl.nasa.gov/doc/papers http://oodt.jpl.nasa.gov/doc/papers