Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Federated Data System DataFed: Experiences in Data Homogenization and Networking R.B. Husar, K. Hoijarvi, S. R. Falke, E. M. Robinson, Washington University,

Published byChristiana Joseph Modified over 8 years ago

Similar presentations

Presentation on theme: "The Federated Data System DataFed: Experiences in Data Homogenization and Networking R.B. Husar, K. Hoijarvi, S. R. Falke, E. M. Robinson, Washington University,"— Presentation transcript:

1 The Federated Data System DataFed: Experiences in Data Homogenization and Networking R.B. Husar, K. Hoijarvi, S. R. Falke, E. M. Robinson, Washington University, St. Louis G. Leptoukh, NASA GSFC Spring AGU, May 29, 2008, Ft. Lauderdale

2 DataFed in a Nutshell: A Federation of autonomous, distributed data providers Performs non-intrusive wrapping of data into web services Provides service-based analysis services and tools General Experience with DataFed: It is an agile virtual data system can deliver info products to diverse users Third-party mediation can homogenize distributed data on the fly Since 2005, DataFed is used by EPA and in research DataFed Motivated by GEOSS DataFed development is guided by the meme of GEOSS

3 Five practices for agile, seamless data federation: 1.Space-Time Query for standardized access to all data (WCS) 2.Data Wrappers for turning heterogeneous data into web services 3.Data Mediators for transforming data into ‘Views’ 4.Mashups for connecting autonomous application 5.DataSpaces for shared metadata by the users, for the users

4 Parameter-Space-Time Query Using OGC WCS Data Access Protocol Regardless of the data location, data type and format, the parameter-space-time query is the same the return is in user selectable format from the offerings Coverage=THEEDDS.T& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=NetCDF Coverage=SEAW.Refl& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=GeoTIFF Coverage=SURF.Bext& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=NetCDF-table GridImageStation Data ParameterBounding BoxTime RangeOut Format

5 DataFed wrappers are non-intrusive, third party Third Party Data Wrappers Heterogeneous input data >>> Homogeneous (WCS) Query

6 Mediated User-Data Interface Mediator turns data into Views Mediated Integration is a flexible design pattern for System of Systems Client-Server design is demanding: User carries the burden of integration Query Data Views

7 SOAP RDF Mashup Workflow Mashups: Loose Coupling of Autonomous Applications DataFed – Wiki -- GoogleEarth

8 DataSpaces for Datasets GEOSS Comp. Registry Community AQ Portal extracts Service Offeror registers GEOSS Clearinghouse Catalog list Searches, harvests invokes references publishes provides Standards; SIF Registry Adopted from Percivall, Feb 2008 by R. Husar, March 2008Percivall, Feb 2008 Community AQ Catalog Catalog User Service Workflow composes Data Analyst visualizes Reports to Decision Maker Policy Analyst Informs Services find Community DataSpaces links to GEOSS Core Service Offerors and Users Shared Metadata by the Users, for the Users

9 GEOSS Comp. Registry Community AQ Portal extracts Service Offeror registers GEOSS Clearinghouse Catalog list Searches, harvests invokes references publishes provides Standards; SIF Registry Adopted from Percivall, Feb 2008 by R. Husar, March 2008Percivall, Feb 2008 Community AQ Catalog Service Workflow composes Data Analyst visualizes Reports to Decision Maker Policy Analyst Informs Services find Community DataSpaces links to GEOSS Core Service Offerors and Users Shared Metadata by the Users, for the Users views report

10 GEOSS Comp. Registry Community AQ Portal extracts Service Offeror registers GEOSS Clearinghouse Catalog list Searches, harvests invokes references publishes provides Standards; SIF Registry Adopted from Percivall, Feb 2008 by R. Husar, March 2008Percivall, Feb 2008 Community AQ Catalog Service Workflow composes Data Analyst visualizes Decision Maker Policy Analyst Informs Services find Community DataSpaces links to GEOSS Core Service Offerors and Users Shared Metadata by the Users, for the Users -+ views report

11 Wiki ‘DataSpaces’ Creating and Sharing Metadata Community Catalog - Find Dataset Describe Dataset Discuss Dataset ESIP Communal Wiki Semantic Wiki: Structured (RDF and Unstructured Content Open, Standard Matadata - RDF Ready for Export/Harvesting by Registries, Catalogs

12 Sharing Best Practices: GEO Best Practice Wiki

13 Developments and Challenges: Favorable Engineering Developments: A Core network for Air Quality data sharing is emerging. Standards are available for sharing previously unstructured data Third-party mediation can homogenize the distributed data Agile SOA-based systems can deliver info products to diverse users Since 2005, one such IS, DataFed is used by EPA and in research However: Service interfaces are still uneven; networks are still fragile The utility of social networking in science is not understood Users can not provide feedback to upstream providers Many cultural, legal and other barriers hamper progress

14 ESIP Coordination Application

15

16

Download ppt "The Federated Data System DataFed: Experiences in Data Homogenization and Networking R.B. Husar, K. Hoijarvi, S. R. Falke, E. M. Robinson, Washington University,"

Similar presentations

GEOSS ADC Architecture Workshop Clearinghouse, Catalogues, Registries Doug Nebert U.S. Geological Survey February 5, 2008.

GEOSS ADC Architecture Workshop Clearinghouse, Catalogues, Registries Doug Nebert U.S. Geological Survey February 5, 2008.
1 GEOSS Architecture Implementation Pilot (AIP) Phase 2 Air Quality and Health Working Group Stefan Falke, Rudy Husar, Frank Lindsay, David McCabe, Erin.

1 GEOSS Architecture Implementation Pilot (AIP) Phase 2 Air Quality and Health Working Group Stefan Falke, Rudy Husar, Frank Lindsay, David McCabe, Erin.
Web Services Implementation Case Study: DataFed Air Quality Data & Services Project Coordinators: Software Architecture: R. Husar Software Implementation:

Web Services Implementation Case Study: DataFed Air Quality Data & Services Project Coordinators: Software Architecture: R. Husar Software Implementation:
REASoN REASoN Project to link NASA's data, modeling and systems to users in research, education and applications Application of NASA ESE Data and Tools.

REASoN REASoN Project to link NASA's data, modeling and systems to users in research, education and applications Application of NASA ESE Data and Tools.
Enabling Tools and Methods for International, Inter- disciplinary and Educational Collaboration E. M. Robinson, K. Hoijarvi, S. Falke, E. Fialowski, M.

Enabling Tools and Methods for International, Inter- disciplinary and Educational Collaboration E. M. Robinson, K. Hoijarvi, S. Falke, E. Fialowski, M.
Web 2.0 in Air Quality Analysis and Management Rudolf B. Husar and Erin M. Robinson Washington University, St. Louis, MO 2007 National Environmental Information.

Web 2.0 in Air Quality Analysis and Management Rudolf B. Husar and Erin M. Robinson Washington University, St. Louis, MO 2007 National Environmental Information.
Enhancing Data Discovery, Understanding and Usage through an Air Quality Metadata System ISRSE, 6 May 2009 The Palazzo dei Congressi di Stresa Stresa,

Enhancing Data Discovery, Understanding and Usage through an Air Quality Metadata System ISRSE, 6 May 2009 The Palazzo dei Congressi di Stresa Stresa,
GEOSS Architecture Describes how components fit together for providing data and information that will be better …than the individual components or systems.

GEOSS Architecture Describes how components fit together for providing data and information that will be better …than the individual components or systems.
ESIP Air Quality Workgroup and the GEO Air Quality Community of Practice collaboratively building an air quality community network for finding, accessing,

ESIP Air Quality Workgroup and the GEO Air Quality Community of Practice collaboratively building an air quality community network for finding, accessing,
Emissions Information Infrastructures NEISGEI and GEOSS Stefan Falke Northrop Grumman and Washington University in St. Louis

Emissions Information Infrastructures NEISGEI and GEOSS Stefan Falke Northrop Grumman and Washington University in St. Louis
Global Earth Observation System of Systems (GEOSS) Architecture Implementation Pilot (AIP) Air Quality and Health Working Group Stefan Falke, Rudy Husar,

Global Earth Observation System of Systems (GEOSS) Architecture Implementation Pilot (AIP) Air Quality and Health Working Group Stefan Falke, Rudy Husar,
Air Quality Cluster Earth Science Information Partners Goals 1. Serve as facilitator and advisor.

Air Quality Cluster Earth Science Information Partners Goals 1. Serve as facilitator and advisor.
DRAFT June 6, 2005 ESIP AQ Cluster, Air Quality Cluster Air Quality Cluster TechTrack Earth Science Information Partners Partners NASA.

DRAFT June 6, 2005 ESIP AQ Cluster, Air Quality Cluster Air Quality Cluster TechTrack Earth Science Information Partners Partners NASA.
2009 Progress Report to UIC: GEOSS Air Quality Community of Practice (AQ CoP link to cop site ) Rudolf Husar (Point of Contact, Washington.

2009 Progress Report to UIC: GEOSS Air Quality Community of Practice (AQ CoP link to cop site ) Rudolf Husar (Point of Contact, Washington.
Instrument Builders Information Specialists (ESIP) Scientists Curriculum Developers Teachers Decision Analysts Decision Makers Reports From Kim Kastens.

Instrument Builders Information Specialists (ESIP) Scientists Curriculum Developers Teachers Decision Analysts Decision Makers Reports From Kim Kastens.
REASoN REASoN Project to link NASA's data, modeling and systems to users in research, applications and education Application of NASA ESE Data and Tools.

REASoN REASoN Project to link NASA's data, modeling and systems to users in research, applications and education Application of NASA ESE Data and Tools.
Architecture and Technologies for an Agile, User-Oriented Air Quality Data System Rudolf B. Husar Washington University, St. Louis Presented at the workshop.

Architecture and Technologies for an Agile, User-Oriented Air Quality Data System Rudolf B. Husar Washington University, St. Louis Presented at the workshop.
GEO Architecture and Data Committee Task AR-07-02 Architecture Implementation Pilot George Percivall Open Geospatial Consortium GEO Task AR-07-02 Point.

GEO Architecture and Data Committee Task AR Architecture Implementation Pilot George Percivall Open Geospatial Consortium GEO Task AR Point.
Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,

Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,
GEOSS AIP-2 Overview AIP-2 Societal Benefit Area Working Groups: Air Quality & Health Disaster Response Biodiversity and Climate Renewable Energy Air Quality.

GEOSS AIP-2 Overview AIP-2 Societal Benefit Area Working Groups: Air Quality & Health Disaster Response Biodiversity and Climate Renewable Energy Air Quality.

Similar presentations

Ads by Google