A Software Architecture for Highly Data-Intensive Systems Chris A. Mattmann USC Center for Software Engineering Annual Research Review March 2004 Special thanks to Dan Crichton, Steve Hughes, and Sean Kelly for some of the slides!

Overview Motivation Problem Statement OODT: A Software Architecture and Middleware for Data-Intensive Systems Evaluation: Science Problems  Planetary Science  Cancer Research Conclusion

Motivation

Problem Statement Information Integration in Data-Intensive Systems  Needed to support data access, distribution, processing and retrieval across existing heterogeneous data sources NASA’s Planetary Data System NCI’s Early Detection Research Network Software and Techniques exist to perform Information Integration  But….. No Software Re-use No Design Methods to start from No mapping of integration techniques to software components, interaction mechanisms, or arrangements of components Lack of Re-use and software standards for information integration in data-intensive systems has forced systems to be “built from scratch”  Little or no interoperability with other software systems  Programmer almost always “in the loop”  New GDS proposal accompanies most new NASA mission proposals

Our Approach A Software Architecture for Data-Intensive Systems  Data Architecture Data Dictionary Resource Profiles  Software Architecture Components: Product Servers, Profile Servers, Query Servers Connector: Messaging Layer Configurations of Product/Profile/Query Servers..and a middleware implementation based on the software architecture  Middleware leverages existing distributed object middleware frameworks such as CORBA, RMI We’re currently working on a SOAP version Built and maintained at the Jet Propulsion Laboratory  Yes, the Mars folks Architecture+middleware = OODT (Object Oriented Data Technology) Middleware being developed at JPL Architecture being formalized at USC-CSE

Data Dictionary Common Data Model containing  Data Elements which the user is interested in querying for  Data Elements which the user would like to retrieve Challenge:  Integrate data sources linked in by exploiting the Data Dictionary structure Map common data model to data source models across data-intensive system  Use a common data element structure ISO Specification and Standardization of Data Elements  Handles the integration of data models across the system, but still need to integrate software interfaces

Resource Profiles Provides mechanisms for describing data systems, data products, etc including  Common data attributes using Dublin Core (I.e. Title, Author, Subject) data elements to describe electronic resources  Mechanisms for describing where the data is located and how to access it  Domain data elements that are useful for describing the product (i.e. TARGET_NAME, MISSION_NAME, INSTUMENT_NAME, etc) Enables “search and retrieval” of distributed data products  Searches to a Profile Server yields information regarding the characteristics of distributed resources (i.e. descriptive information about the product, access information, etc)

Resource Profiles Example “country = US and windspeed > 120” … … urn:eda:rmi:Western… country … US … state … WA CA … windspeed … … … … urn:eda:rmi:Southern… country … US … state … LA TX … windspeed … 1 89 … Matches!

Components Product Server  Responsible for abstracting heterogeneous data source interfaces Attach a Product Server to each data source that is integrated  Provides a common query interface across heterogeneous data sources Profile Server  Describe data resources using resource profiles Allow data resources to be discovered and located at query-time Query Server  Tie it all together  Uses Profile Servers to discover data resources which could potentially satisfy a query  Queries discovered data resources (such as Product Servers) and collects obtained data products to return to the user

Connectors Messaging Layer  Each OODT component registers itself with a Component Registry Allows Components to define and provide services Components defined by unique URNs  Transfers OODT Query Object containing OODT Style Query  (Keyword = Value) predicates joined by logical operators (AND, OR, etc) The result list to be populated

Configurations: Example

Configurations: Example (2)

Configurations: Example (3)

Planetary Science Planetary Data System  Official NASA “Active” Archive for all Planetary Data Data ingestion required as part of Announcement of Opportunity (AO) for a mission  9 Nodes with data located at discipline sites  Common Data Architecture  Different data systems located at the sites  Prior to October 2002, no ability to find and share data between PDS nodes Data distribution via CD ROM Limited electronic distribution

OODT PDS Deployment

Early Detection Research Network OODT’s success has lead to interagency agreements with both NIH and NCI OODT has provided the NCI with a bioinformatics infrastructure for sharing data across the nation  Currently deployed at 10 of 31 NCI Research Institutions for the Early Detection Research Network (EDRN)  Providing real-time access to distributed, heterogeneous databases  Created a national virtual repository for biospecimens (now a NCI Director Initiative)  Now integrating new datasets: validation studies, images, biomarkers, etc  Meet Federal security regulations  Operational September 2002 Same core software framework as deployed in planetary, earth and engineering

OODT EDRN Deployment

Conclusion OODT is…..  A novel software architecture to describe data intensive systems integration, search, retrieval and discovery of heterogeneous data stored in heterogeneous domain data sources  A reference implementation of above software architecture Java-based middleware C++. Perl, Python, PHP Client APIs  A process for annotating and creating standard metadata models to describe heterogeneous data based on data standards Dublin Core ISO-11179

Referred Papers Mattmann C, Ramirez P, Crichton D, and Hughes, J.S. Packaging Data Products using Data Grid Middleware for Deep Space Mission Systems. Accepted for Publication at the 8th International Conference on Space Operations, Montreal, Canada, Mattmann C, Freeborn D, Crichton D. Towards a Distributed Information Architecture for Avionics Data. In Proceedings of the 2nd International IADIS Conference on the World- Wide-Web and Internet, Volume II, pp Algarve, Portugal, Crichton D, Hughes, J.S., Kelly, S. A Science Data System Architecture for Information Retrieval. Clustering and Information Retrieval. Kluwer Academic Publishers. December Book Chapter on OODT Crichton D, Hughes, J.S., Kelly, S, Rameriz, P. A Component Framework Supporting Peer Services for Space Data Management IEEE Aerospace Conference. Big Sky, Montana. March Crichton D, Downing G, Hughes J. S, Kincaid H, Srivistava S. An Interoperable Data Architecture for Data Exchange in a Biomedical Research Network. 14th IEEE Symposium on Computer-Based Medical Systems. July Crichton, D., Hughes J. S, Hardman S, Kelly S. A Distributed Component Framework for Data Product Interoperability. 17th CODATA International Conference, Baveno, Italy. October Crichton, D., Hughes J. S, Kelly S, Hyon J. Science Search and Retrieval using XML. Second National Conference on Scientific and Technical Data, Washington D.C., National Academy of Sciences. March 2000.

Questions? Contacts  OODT Website:  Principal Investigator Dan Crichton  Co-Investigator Steve Hughes  Programmer/Research Grunt Me Thanks for your attention!

Backup Slides

Resource Profiles Example “country = US and windspeed > 120” … … urn:eda:rmi:Western… country … US … state … WA CA … windspeed … … … … urn:eda:rmi:Southern… country … US … state … LA TX … windspeed … 1 89 … Matches!

Object Oriented Data Technology Object-Oriented Data Technology (OODT)  Funded in 1998 by NASA’s Office of Space Science to develop a national software framework for sharing data across heterogeneous, distributed data repositories  Develop… a common data and software framework to enable data sharing across multiple science and engineering disciplines A reusable software architecture across data management projects  Reusable software components with common interfaces  Interfaces to enable new components to be plugged in  Mechanism to wrap legacy data system components with minimal impact  OODT should provide.. Science domain independence (use in engineering, science and biomedicine) Data location independence (describe what you want, not how/where to get it