Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,

Slides:

Advertisements

Similar presentations

WCS Data Exchange at the DataFed Server/Client Center for Air Pollution Impact and Trend Analysis (CAPITA) Washington University, St. Louis, MO OGC TC.

Advertisements

1 NASA CEOP Status & Demo CEOS WGISS-25 Sanya, China February 27, 2008 Yonsook Enloe.

1 GEOSS Architecture Implementation Pilot (AIP) Phase 2 Air Quality and Health Working Group Stefan Falke, Rudy Husar, Frank Lindsay, David McCabe, Erin.

Web Services Implementation Case Study: DataFed Air Quality Data & Services Project Coordinators: Software Architecture: R. Husar Software Implementation:

Talk Outline Air Quality Information System Challenges (5min) –Real-time monitoring and data delivery (1 slide) –Characterization of pollutant in.

ESIP Vision: “Achieve a sustainable world” by Serving as facilitator and advisor for the Earth science information community Promoting efficient flow of.

ESIP Air Quality Workgroup and the GEO Air Quality Community of Practice collaboratively building an air quality community network for finding, accessing,

Stefan Falke Center for Air Pollution Impact and Trend Analysis Washington University in St. Louis Networked Data and Tools for Environmental Management.

Distributed Data Analysis & Dissemination System (D-DADS) Prepared by Stefan Falke Rudolf Husar Bret Schichtel June 2000.

Distributed Voyager (DVoy) Web Services

DRAFT June 6, 2005 ESIP AQ Cluster, Air Quality Cluster Air Quality Cluster TechTrack Earth Science Information Partners Partners NASA.

DRAFT April 28, 2005 ESIP AQ Cluster, The data life cycle consists of the acquisition and the usage parts Usage ActivitiesData Acquisition.

2009 Progress Report to UIC: GEOSS Air Quality Community of Practice (AQ CoP link to cop site ) Rudolf Husar (Point of Contact, Washington.

Instrument Builders Information Specialists (ESIP) Scientists Curriculum Developers Teachers Decision Analysts Decision Makers Reports From Kim Kastens.

REASoN REASoN Project to link NASA's data, modeling and systems to users in research, applications and education Application of NASA ESE Data and Tools.

Architecture and Technologies for an Agile, User-Oriented Air Quality Data System Rudolf B. Husar Washington University, St. Louis Presented at the workshop.

WCS Data Exchange at the DataFed Server/Client Center for Air Pollution Impact and Trend Analysis (CAPITA) Washington University, St. Louis, MO OGC TC.

Federated Network for Sharing Air Quality Data and Processing Services Center for Air Pollution Impact and Trend Analysis (CAPITA) Washington University,

1 International Standards for Data Interoperability GALEON Geo-interface for Air, Environment, Land, Ocean NetCDF Ben Domenico Unidata Program Center*

REASoN REASoN Project to link NASA's data, modeling and systems to users in research, education and applications Application of NASA ESE Data and Tools.

Spatio-Temporal Data Sharing using XML Web Services Presented at the Workgroup Meeting on Web-based Environmental Information System for Global Emission.

Current Air Quality Information ‘Ecosystem’ (Draft for Feedback) AQ information includes emissions, ambient & satellite data and model outputs The distributed.

Stefan Falke Center for Air Pollution Impact and Trend Analysis Washington University in St. Louis Brooke Hemming US EPA – Office of Research and Development.

Application of ESE Data and Tools to Particulate Air Quality Management The CAPITA REASoN Project August 15, 2003 Stefan Falke and Rudolf Husar Center.

Air Quality Cluster Air Quality Cluster TechTrack Earth Science Information Partners Partners(?) NASA NOAA EPA USGS DOE NSF Industry… Data Flow Technologies.

Adoption of RDA-DFT Terminology and Data Model to the Description and Structuring of Atmospheric Data Aaron Addison, Rudolf Husar, Cynthia Hudson-Vitale.

Accessing and Using Fire-Related Data with the CAPITA DataFed.net* Services Framework Stefan Falke Rudolf Husar Kari Hoijarvi Washington University in.

Air Quality Data Services: Application of OGC specifications Air Quality Data: Multi-dimensional, multi-source, multi-format Point observations are collected.

NASA Air Quality Applications Program and the ESIP Air Quality Cluster The goal of the NASA Air Quality Management program is to: Enable partners’ beneficial.

Interoperable Information System of Systems for HTAP Rudolf B. Husar and Rich Scheffe With Erin Robinson Presented at HTAP Workshop, WMO, Geneva, January.

Why so many data systems? Dickerson – ppt. Information as a Resource Shared not exchanged …

1 Using the GEOSS Common Infrastructure in the Air Quality & Health SBA: Wildfire & Smoke Assessment Prepared by the GEOSS AIP-2 Air Quality & Health Working.

Select, Overlay, Explore; Integration of diverse data Distributed Data Heterogeneous coding, access Connects providers to users; Homogenize data access.

Stefan Falke and Rudolf Husar Center for Air Pollution Impact and Trend Analysis Washington University in St. Louis A NSF Digital Government Pilot Project.

ESIP Vision: “Achieve a sustainable world” by Serving as facilitator and advisor for the Earth science information community Promoting efficient flow of.

Architectures and Technologies Enabling the Diffusion of Atmospheric Science Information Rudolf B. Husar and Erin Robinson Washington University, St. Louis.

Interoperability of Data and Processing Services: Experience Using the DataFed System Rudolf B. Husar, Stefan R. Falke and Kari Hoijarvi Washington University,

Interoperability of Data and Processing Services: Experience Using the DataFed System Rudolf B. Husar, Stefan R. Falke and Kari Hoijarvi Washington University,

Part I DataFed An Agile Distributed Air Quality Data System Rudolf B. Husar Washington University, St. Louis Seminar Presented at University of Alabama,

Distributed Data Analysis & Dissemination System (D-DADS ) Special Interest Group on Data Integration June 2000.

Fire Emissions Network Sept. 4, 2002 A white paper for the development of a NSF Digital Government Program proposal Stefan Falke Washington University.

NASA REASoN Project SHAirED: S ervices for H elping the Air -quality Community use E SE D ata Stefan Falke, Kari Höijärvi and Rudolf Husar, Washington.

NASA REASoN Project SHAirED: S ervices for H elping the Air -quality Community use E SE D ata Stefan Falke, Kari Höijärvi and Rudolf Husar, Washington.

2009 Progress Report to UIC: GEOSS Air Quality Community of Practice (AQ CoP link to cop site ) Rudolf Husar (Point of Contact, Washington.

Part I DataFed An Agile Distributed Air Quality Data System Rudolf B. Husar Washington University, St. Louis Seminar Presented at University of Alabama,

Architecture and Technologies for an Agile, User-Oriented Air Quality Data System Rudolf B. Husar Washington University, St. Louis Presented at the workshop.

Architecture and Technologies for an Agile, User-Oriented Air Quality Data System Rudolf B. Husar Washington University, St. Louis Presented at the workshop.

Web Services-Based Mediator of Distributed Data Flow and Processing Project Coordinators: Software Architecture: R. Husar Software Implementation: K. Höijärvi.

Application of NASA ESE Data and Tools to Air Quality Management Stefan Falke and Rudolf Husar (Co-PIs) Washington University in St. Louis Project Period:

1 Using the GEOSS Common Infrastructure in the Air Quality & Health SBA: Wildfire & Smoke Assessment Prepared by the GEOSS AIP-2 Air Quality & Health Working.

ESIP Air Quality Jan Air Quality Cluster Air Quality Cluster Technology Track Earth Science Information Partners Partners NASA NOAA EPA (?) USGS.

: Data Sharing/Processing Infrastructure Data Catalog and Access Dozens of datasets on aerosols, emissions, fire, meteorology,

The Federated Data System DataFed: Experiences in Data Homogenization and Networking R.B. Husar, K. Hoijarvi, S. R. Falke, E. M. Robinson, Washington University,

Architecture and Technologies for an Agile, User-Oriented Air Quality Data System Rudolf B. Husar Washington University, St. Louis, MO, USA Outline Highlight.

1 SEEDS IT Vision Scenario: Smoke Impact REASoN Project: Application of NASA ESE Data and Tools to Particulate Air Quality Management (PPT/PDF)Application.

1 Key Results from GALEON John Caron Ben Dominico UCAR/Unidata.

DRAFT June 6, 2005 ESIP AQ Cluster, Contact R. Husar Air Quality Cluster Air Quality Cluster TechTrack Earth Science Information Partners.

Application of NASA ESE Data and Tools to Particulate Air Quality Management A proposal to NASA Earth Science REASoN Solicitation CAN-02-OES-01 REASoN:

The Federated Data System DataFed R. Husar, K. Hoijarvi, S. Falke, DaFed Community EPA Data Summit, Feb. 12, 2008, RTP Non-intrusive data integration infrastructure.

Standards-based Access to Air Quality Data: Application of OGC WMS and WCS Protocols Client Server Std. Interface GetCapabilities GetData Capabilities,

Federated Network for Sharing Air Quality Data and Processing Services Center for Air Pollution Impact and Trend Analysis (CAPITA) Washington University,

HTAP Data Network GEO Task DA-09-02d: Atm. Model Evaluation Network Project Officer: Terry Keating, EPA ESIP Meeting, Air Quality Workgroup January 5,

Interoperability Day Introduction Standards-based Web Services Interfaces to Existing Atmospheric/Oceanographic Data Systems Ben Domenico Unidata Program.

Fire, Smoke & Air Quality: Tools for Data Exploration & Analysis : Data Sharing/Processing Infrastructure This project integrates.

There is increasing evidence that intercontinental transport of air pollutants is substantial Currently, chemical transport models are the main tools for.

NATIONAL AERONAUTICS AND SPACE ADMINISTRATION ESDS Reuse Working Group Earth Science Data Systems Reuse Working Group Case Study: SHAirED Services for.

ESIP Vision: “Achieve a sustainable world” by Serving as facilitator and advisor for the Earth science information community Promoting efficient flow of.

ESIP Air Quality Jan Air Quality Cluster Air Quality Cluster Technology Track Earth Science Information Partners Partners NASA NOAA EPA (?) USGS.

GEOSS Air Quality Community Infrastructure

4/5 May 2009 The Palazzo dei Congressi di Stresa Stresa, Italy

Presentation transcript:

Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University, St. Louis, MO Ispra, JRC, March Data handling approach Software tools Participant involvement Data dissemination Integration problems Summary

Policy Guidance for HTAP IT Terry Keating, HTAP Task Force Co-Chair: Transparency of the HTAP Process –Acceptance of the Tech Report will depend on openness Inclusiveness and Ease of Participation –Facilitate participation by smaller contributors

Current Air Quality Information ‘Ecosystem’ AQ info is distributed over many ‘dimensions’: Geography, Content, Agency, Program… AQ info content includes: emissions, ambient & satellite data and model outputs Info is provided and consumed by different agencies, (NASA, NOAA, EPA…) Providers have different access protocols, formats, and information usage conditions Lack of Interoperability Poor data & model utilization Less informed decision making

GEOSS Info Flow Architecture A General Framework Accepted by Members Model Data

Federated Network for Air Quality Data and Processing Services Project Team: Software Architecture: R. Husar Software Implementation: K. Höijärvi Data and Applications: S. Falke, R. Husar Data Handling Approach

The challenge is to design a general supportive infrastructure Simply connecting the relevant provides and users for each info product is messy Federated Data System for Air Quality The info system infrastructure needs to facilitate the creation of info products AQ Compliance Nowcast/Forecast Status & Trends Find Data Gaps ID New Problems ……… Info Needs Reports Providers supply the ‘raw material’ (data and models) for ‘refined’ info products Emission Surface Satellite Model Single Datasets Providers Wrappers Where? What? When? Federate Data Structuring Structuring the heterogeneous data into where-when-what ‘cubes’ simplifies the mess Slice & Dice Explore Data Viewers The ‘cubed’ data can be accessed and explored by slicing-dicing tools Programs Integrate Understand More elaborate data integration and fusion can be done by web service chaining Non-intrusive Linking & MediationData UsersData Providers

Data Handling Approach: DataFed Approach: Mediation Between Users and Data Providers DataFed assumes autonomous data management (a la Internet) Non-intrusive third-party data wrapping for unified web service (WS) access End-user programming of applications through chaining of WS components Applications Building browsers and analysis tools for distributed monitoring data Serve as data and service resources for user programs (science, GIS tools) Support application projects, e.g. FASTNET, Exceptional Event Rule

Typical DataFed AQ Analysis Tools ConsolesConsoles: Data from diverse sources are displayed to create a rich context for exploration and analysis CATTCATT: Combined Aerosol Trajectory Tool for the browsing backtrajectories for specified chemical conditions Tool Viewer: Viewer: General purpose spatio-temporal data browser and view editor applicable for all DataFed datasets

Quebec Smoke July, 7, 2002 SeaWiFS Satellite Aerosol Chemical Air Trajectory Map Boarder Web Service Composition

Software Tools: Demo: Networked Data Access, Processing and Fusion AeroCOM – VIEWS SO Networking

Visualization Goal: 4D, User-selectable layers Image below completed in 1998 Satellite Data Layers: Land Reflectance (SeaWiFS) Fire Pixels (ASTR Night) High Cloud (GOES, Meteosat, GMS2) Aerosol AOT (AVHRR)

4D Dynamic Visualization Demonstrate interactions, allow exploration ‘Google Earth’ for Earth Science is now possible

Partners N ASA NOAA EPA USGS DOE NSF Industry… Earth Science Information Partners Air Quality Cluster 1. Serve as facilitator to Earth Science information community. 2. Promote efficient flow of ES data from collection to end-use. 3. Improve quality and usability of ES data and info systems 4. Expand the use of Earth science information Participant Involvement

Data Dissemination & Use – Service Based Provide Catalog Services (Publish, Find data) Allow Data Access (‘Bind’) – Use International Standards Add tools for exploration and analysis Near Real Time Data Integration Delayed Data Integration Surface Air Quality AIRNOWO3, PM25 ASOS_STIVisibility, 300 sites METARVisibility, 1200 sites VIEWS_OL40+ Aerosol Parameters Satellite MODIS_AOTAOT, Idea Project GASPReflectance, AOT TOMSAbsorption Indx, Refl. SEAW_USReflectance, AOT Model Output NAAPSDust, Smoke, Sulfate, AOT WRFSulfate Fire Data HMS_FireFire Pixels MODIS_FireFire Pixels Surface Meteorology RADARNEXTRAD SURF_METTemp, Dewp, Humidity… SURF_WINDWind vectors ATADTrajectory, VIEWS locs.

WCS - Interoperable Data Access Service netCDF– Machine independent encoding ncML– XML data description CF– Naming, structure convention OGC Web Coverage Service - Interoperable Data Access Query Data Syntax + Semantics Coverage (parameter) BBOX Time Range netCDF Example Profile SERVICE=wcs‘service REQUEST=GetCoverage,VER=1.0‘service method COVERAGE=AIRNOW.pmfine‘what CRS=EPSG:4326‘projection BBOX=-125,22,-61,51,0,0‘where TIME= T15:00:00Z‘when FORMAT=netCDF‘return format

GALEON Interoperability Experiment Geo-interface for Atmosphere, Land, Earth, and Ocean netCDF Unify Earth Science & GIS Data Flows Strong European Participation IT – S. Nativi, L. Bigagli UK – Andrew Wolf DE – Peter Bauman) B. Domenico GALEON UNIDATA U Florence/CNR-IMMA WCS Server

OGC WCS Demonstration: THREDDS_GFS 4Dim DatasetTHREDDS_GFS Lat/Lon Box Elev Range Time Range Map: BBOX=-180,-90,180,90, 1350,1350& TIME= / /PT3H Time: BBOX=-34,49.05,-34,49.05, 1350,1350& TIME= / /PT3H Elev: BBOX=-34,49.05,-34,49.05, 0,18000 & TIME= / /PT3H The form of the WCS query is the same for all slices through the data cube (views) The only difference in the views is the thickness of the slices in each dimension Return grid is in multiple formats (NetCDF, CSV, GML, PNG, … ) Map View Services WCS Query Time View Services WCS Query Elevation View Services WCS Query WCS Query

Summary Current systems data & model analysis are heterogeneous Standardization is a key need for agile IT systems Non-intrusive mediators can achieve virtual standardization Technologies are currently available for dynamic NETWORKING We eager to share our networked data, tools and methods

OGC WCS Demonstration: Grid, Image, Station Data Types Coverage=THEEDDS.T& BBOX=-126,24,-65,52,0,0 &TIME= / &FORMAT=NetCDF Coverage=SURF.Bext& BBOX=-126,24,-65,52,0,0 &TIME= / &FORMAT=NetCDF-table Coverage=SEAW.Refl& BBOX=-126,24,-65,52,0,0 &TIME= / &FORMAT=GeoTIFF COVERAGE=sst& BBOX=-126,24,-65,52,0,0 &TIME= , &FORMAT=NetCDF UNIDATA – THREDDS/GALEON WCS THREDDS/GALEON WCS DataFed GALEON WCS U Florence, It GALEON WCS DataFed GALEON WCS Grid Image Station Services WCS Query Services WCS Query Services WCS Query Services WCS Query

Single Data Model for All AQ Data Most Views are slices through a cube of data organized by lat, lon, altitude, and time (X,Y,Z,T) Multidimensional Data Cube

OGC Web Coverage Service (WCS) Specification HTTP GET/POST based interfaces Services have XML service descriptions (“Capabilities”, “Description”) Filter parameters allow selection of subsets of source data Output formats advertised by each service instance OGC WCS getCoverage Schema Suitable for wrapping with SOAP envelope, WSDL access, loose coupling WCS is for "coverages" – information representing space-time-varying phenomena WCS describes, requests and delivers coverages in spatio-temporal domain WCS version 1.1 is limited to grids/"simple” coverages with homogeneous range sets

AQ Monitoring Network Data Storage and Delivery through OGC Protocols Relational Data Model Star Schema WMS WCSSOS SensorML WFS Observations Station Info. Param/Sensor/Method Data View Services WMS StationsPar-Meth Observations SOS WCS Data Access

Technology Support for ‘Integrated Solutions’ Air Quality Information System

through Data Access through Adapters DataFed SOAP,HTTP Get OGC WCS HTTP Get, Post OGC WMS HTTP Get Station-Point SQL Server, Files… Sequence Image, file nDim Grid OpenDAP NetCDF, … Other Traject., Event, Pic Sources Diverse formats Many data models Data Wrapper Data into geo-cubes Queries to views Virtual Data Cube Global geo-cube data model Makes queries data-neutral Others? e.g. OpenDAP Output Protocol dependent User specified GeoTable CSV,XLS,GML GeoGrid GML,NetCDF.. GeoImage GeoTIFF, PNG.. Other MS Dataset.. Query Adapter Maps query to protocol User selects protocols