Web Services-Based Mediator of Distributed Data Flow and Processing Project Coordinators: Software Architecture: R. Husar Software Implementation: K. Höijärvi.

Slides:

Advertisements

Similar presentations

Web Services Implementation Case Study: DataFed Air Quality Data & Services Project Coordinators: Software Architecture: R. Husar Software Implementation:

Advertisements

Proposal Outline: Extensions to the VIEWS System: Analysis Tools and Auxiliary Data R. Husar, CAPITA March, 2003 Presentation and Analysis Tools CATT for.

OGC Demo at IGARSS06 July 30 - Denver, CO Telecon 11 July 2006 Liping Di, George Mason University Rudolf Husar, Washington University.

Federated PM and Haze Data Warehouse Project a sub- project of (enter your sticker & logo here ) Nov 20, 2001, RBH St. Louis Midwest Supersite Project.

1 The FASTNET Project Presented by: Sean Raffuse 1 Rudy Husar 2 Rich Poirot and Gary Kleiman 3 1 Sonoma Technology, Inc. 2 Center for Air Pollution Impact.

Proposal Outline: Extensions to the VIEWS: General CATT Analysis Tool R. Husar, CAPITA Revised, June 26, 2003 Proposed Sub-Projects CATT for VIEWS$20k.

Stefan Falke Center for Air Pollution Impact and Trend Analysis Washington University in St. Louis Networked Data and Tools for Environmental Management.

Distributed Data Analysis & Dissemination System (D-DADS) Prepared by Stefan Falke Rudolf Husar Bret Schichtel June 2000.

Select, Overlay, Explore; Multidimensional data Maintain Distributed Data; Heterogeneous coding, access Connect providers to users; Homogenize data access.

CAPITA Projects NSF ToolsCollaboration Tools for Virtual Workgroups EPA WebVis Internet Visibility System NOAAASOS Data Evaluation EPAICAP Intercontinental.

Distributed Voyager (DVoy) Web Services

DRAFT June 6, 2005 ESIP AQ Cluster, Air Quality Cluster Air Quality Cluster TechTrack Earth Science Information Partners Partners NASA.

REASoN REASoN Project to link NASA's data, modeling and systems to users in research, applications and education Application of NASA ESE Data and Tools.

Federated Network for Sharing Air Quality Data and Processing Services Center for Air Pollution Impact and Trend Analysis (CAPITA) Washington University,

Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,

1 The FASTNET Project Presented by: Sean Raffuse 1 Rudy Husar 2 Rich Poirot and Gary Kleiman 3 1 Sonoma Technology, Inc. 2 Center for Air Pollution Impact.

Combined Aerosol & Trajectory Tool (CATT) Development R. Husar, K. Hoijarvi, J. Colson, S. Falke Center for Air Pollution Impact and Trend Analysis (CAPITA)

REASoN REASoN Project to link NASA's data, modeling and systems to users in research, education and applications Application of NASA ESE Data and Tools.

Integration of Multi-Sensory Earth Observations for Characterization of Air Quality Events E. M. Robinson Advisor, R. B. Husar 2010 Masters of Science.

Spatio-Temporal Data Sharing using XML Web Services Presented at the Workgroup Meeting on Web-based Environmental Information System for Global Emission.

Current Air Quality Information ‘Ecosystem’ (Draft for Feedback) AQ information includes emissions, ambient & satellite data and model outputs The distributed.

Stefan Falke Center for Air Pollution Impact and Trend Analysis Washington University in St. Louis Brooke Hemming US EPA – Office of Research and Development.

Application of ESE Data and Tools to Particulate Air Quality Management The CAPITA REASoN Project August 15, 2003 Stefan Falke and Rudolf Husar Center.

Air Quality Cluster Air Quality Cluster TechTrack Earth Science Information Partners Partners(?) NASA NOAA EPA USGS DOE NSF Industry… Data Flow Technologies.

Supersite Relational Database Project: (Data Portal?) a sub- project of St. Louis Midwest Supersite Project Draft of the November 16, 2001 Presentation.

Accessing and Using Fire-Related Data with the CAPITA DataFed.net* Services Framework Stefan Falke Rudolf Husar Kari Hoijarvi Washington University in.

1 Application Scenario: Smoke Impact REASoN Project: Application of NASA ESE Data and Tools to Particulate Air Quality Management (PPT/PDF)Application.

NASA Air Quality Applications Program and the ESIP Air Quality Cluster The goal of the NASA Air Quality Management program is to: Enable partners’ beneficial.

Why so many data systems? Dickerson – ppt. Information as a Resource Shared not exchanged …

OGC Demo at IGARSS06 July 30 - Denver, CO Telecon 11 July 2006 Liping Di, George Mason University Rudolf Husar, Washington University.

Select, Overlay, Explore; Integration of diverse data Distributed Data Heterogeneous coding, access Connects providers to users; Homogenize data access.

Stefan Falke and Rudolf Husar Center for Air Pollution Impact and Trend Analysis Washington University in St. Louis A NSF Digital Government Pilot Project.

VOYAGER Data Explorer: Architecture and Technologies See also the the Voyager Developer Website and early ApplicationsDeveloper WebsiteApplications Layered.

Federated Network for Sharing Air Quality Data and Processing Services Center for Air Pollution Impact and Trend Analysis (CAPITA) Washington University,

Architectures and Technologies Enabling the Diffusion of Atmospheric Science Information Rudolf B. Husar and Erin Robinson Washington University, St. Louis.

Part I DataFed An Agile Distributed Air Quality Data System Rudolf B. Husar Washington University, St. Louis Seminar Presented at University of Alabama,

The Federated Data System, DataFed ESIP Winter MeetingESIP Winter Meeting, Jan 10, 2013, Washington DC Rudolf Husar, Washington University, St. Louis Presented.

COMMUNITY. Data Acquisition and Usage Value Chain.

Distributed Data Analysis & Dissemination System (D-DADS ) Special Interest Group on Data Integration June 2000.

NASA REASoN Project SHAirED: S ervices for H elping the Air -quality Community use E SE D ata Stefan Falke, Kari Höijärvi and Rudolf Husar, Washington.

NASA REASoN Project SHAirED: S ervices for H elping the Air -quality Community use E SE D ata Stefan Falke, Kari Höijärvi and Rudolf Husar, Washington.

Processes of the Information Value Chain Informing Knowledge ActionProductive Knowledge Information Organizing Grouping Classifying Formatting Geo-referencing.

Part I DataFed An Agile Distributed Air Quality Data System Rudolf B. Husar Washington University, St. Louis Seminar Presented at University of Alabama,

An Integrated Fire, Smoke and Air Quality Data & Tools Network Stefan Falke and Rudolf Husar Center for Air Pollution Impact and Trend Analysis Washington.

Integration of Multi-Sensory Earth Observations for Characterization of Air Quality Events using Service Oriented Architecture E. M. Robinson Advisor,

ESIP Air Quality Jan Air Quality Cluster Air Quality Cluster Technology Track Earth Science Information Partners Partners NASA NOAA EPA (?) USGS.

: Data Sharing/Processing Infrastructure Data Catalog and Access Dozens of datasets on aerosols, emissions, fire, meteorology,

The Federated Data System DataFed: Experiences in Data Homogenization and Networking R.B. Husar, K. Hoijarvi, S. R. Falke, E. M. Robinson, Washington University,

Anatomy of a Wrapper Service: TOMS Satellite Image Data Given the URL template and the image description, the wrapper service can access the image for.

1 Integrated System Solutions Value & benefits to citizens and society Data Policy Decisions Management Decisions Predictions Observations High Performance.

1 SEEDS IT Vision Scenario: Smoke Impact REASoN Project: Application of NASA ESE Data and Tools to Particulate Air Quality Management (PPT/PDF)Application.

MEDIATORS. Mediation Typical file-sharing systems have a single global schema for describing their data P2P networks have to consider heterogeneous schemas.

Concepts on Aerosol Characterization R.B. Husar Washington University in St. Louis Presented at EPA – OAQPS Seminar Research Triangle Park, NC, April 4,

DRAFT June 6, 2005 ESIP AQ Cluster, Contact R. Husar Air Quality Cluster Air Quality Cluster TechTrack Earth Science Information Partners.

Application of NASA ESE Data and Tools to Particulate Air Quality Management A proposal to NASA Earth Science REASoN Solicitation CAN-02-OES-01 REASoN:

Harmonization and Integration of Semi- Structured Data Through Wikis and Controlled Tagging E. M. Robinson, R. B. Husar Washington University, St. Louis,

Proposal to MANE_VU: Extensions to the VIEWS: CATT Analysis Tool Full Proposal Text Full Proposal Text R. Husar, PI, CAPITA Revised, October 8, 2003 The.

Combined Aerosol Trajectory Tool, CATT Illustrated Instruction Manual Supported by: MARAMA contract on behalf of Mid-Atlantic/Northeast Visibility Union.

Standards-based Access to Air Quality Data: Application of OGC WMS and WCS Protocols Client Server Std. Interface GetCapabilities GetData Capabilities,

Topic Suggestions Scheffe GEOSS Support to Regional Air Quality (see next slide) –Data. Services –Sharing/Harvesting Infrastructure –Intellectual Resources.

Concepts on Aerosol Characterization R.B. Husar Washington University in St. Louis Presented at EPA – OAQPS Seminar Research Triangle Park, NC, April 4,

Voyager Data Services Services for Finding, Exploring and Presenting Distributed Environmental Data Outline Prepared by Voyager Interest Group on Environmental.

Federated Network for Sharing Air Quality Data and Processing Services Center for Air Pollution Impact and Trend Analysis (CAPITA) Washington University,

Fire, Smoke & Air Quality: Tools for Data Exploration & Analysis : Data Sharing/Processing Infrastructure This project integrates.

There is increasing evidence that intercontinental transport of air pollutants is substantial Currently, chemical transport models are the main tools for.

NATIONAL AERONAUTICS AND SPACE ADMINISTRATION ESDS Reuse Working Group Earth Science Data Systems Reuse Working Group Case Study: SHAirED Services for.

DATAFED Application Programs. Dvoy Data Flow and Processes DataView 1 View Data Abstract Portrayal Device Portrayal Render Device View Portrayal Device.

ESIP Air Quality Jan Air Quality Cluster Air Quality Cluster Technology Track Earth Science Information Partners Partners NASA NOAA EPA (?) USGS.

Intermountain West Data Warehouse - Western Air Quality Study

4/5 May 2009 The Palazzo dei Congressi di Stresa Stresa, Italy

Presentation transcript:

Web Services-Based Mediator of Distributed Data Flow and Processing Project Coordinators: Software Architecture: R. Husar Software Implementation: K. Höijärvi Data and Applications: S. Falke, R. Husar Center for Air Pollution Impact and Trend Analysis (CAPITA) Washington University, St. Louis, MO 63130

DataFed Description DataFed Vision Better air quality management and science through by effective use of relevant data DataFed Goals Facilitate the access and flow of atmospheric data from provider to users Support the development of user-driven data processing value chains Participate in specific application projects Approach: Mediation Between Users and Data Providers DataFed assumes spontaneous, autonomous emergence of AQ data (a la Internet) Non-intrusively wraps datasets for access by web services WS-based mediators provide homogeneous data views e.g. geo-spatial, time... End-user programming of data access and processing through WS composition (limited) Applications Building browsers and analysis tools for distributed monitoring data Serve as data gateway for user programs; web pages, GIS, science tools DataFed is currently focused on the mediation of air quality data

DataFed Multidimensional Data Model 4 D Geo-Environmental Data Cube (X, Y, Z, T) Environmental data represent measurements in the physical world which has space (X, Y, Z) and time (T) as its dimensions. The specific inherent dimensions for geo-environmental data are: Longitude X, Latitude Y, Elevation Z and DateTime T. The needs for finding, sharing and integration of geo- environmental data requires that data are ‘coded’ in this 4D data space – at the minimum.

Data Flow & Processing in Air Quality Management AQ DATA EPA Networks IMPROVE Visibility Satellite-PM Pattern METEOROLOGY Met. Data Satellite-Transport Forecast model EMISSIONS National Emissions Local Inventory Satellite Fire Locs Status and Trends AQ Compliance Exposure Assess. Network Assess. Tracking Progress AQ Management Reports ‘Knowledge’ Derived from Data Primary Data Diverse Providers Data ‘Refining’ Processes Filtering, Aggregation, Fusion

Mediator-Based Integration Architecture (Wiederhold, 1992) The job of the mediator is to provide an answer to a user query (Ullman, 1997)Ullman, 1997 In database theory sense, a mediator is a view of the data found in one or more sources Heterogeneous sources are wrapped by translation software local to global language Mediators (web services) obtain data from wrappers or other mediators and process it … Wrapper Service User QueryViews Heterogeneous Data

Generic Data Flow and Processing in DataFed DataView 1 DataProcessed Data Portrayed Data Process Data Portrayal/ Render Abstract Data Access View Wrapper Physical Data Abstract Data Physical Data Resides in autonomous servers; accessed by view- specific wrappers which yield abstract data ‘slices’ Abstract Data Abstract data slices are requested by viewers; uniform data are delivered by wrapper services DataView 2 DataView 3 View Data Processed data are delivered to the user as multi-layer views by portrayal and overlay web services Processed Data Data passed through filtering, aggregation, fusion and other web services

Anatomy of a Wrapper Service: TOMS Satellite Image Data Given the URL template and the image description, the wrapper service can access the image for any day, any spatial subset using a HTTP URL or SOAP protocol: Wrapper classes are available for geo-spatial (incl. satellite) images, SQL servers, text files,etc. The mediator classes are implemented as web services for uniform data access, transformation and portrayal. src_img_width src_img_height src_margin_rightsrc_margin_left src_margin_top src_margin_bottom src_lon_min src_lat_max src_lat_min src_lon_max Image Description for Data Access: src_image_width=502 src_image_height=329 src_margin_bottom=105 src_margin_left=69 src_margin_right=69 src_margin_top=46 src_lat_min=-70 src_lat_max=70 src_lon_min=-180 src_lon_max=180 The daily TOMS images reside on the FTP archive, e.g. ftp://toms.gsfc.nasa.gov/pub/eptoms/images/aerosol/y2000/ea gif ftp://toms.gsfc.nasa.gov/pub/eptoms/images/aerosol/y2000/ea gif URL template: ftp://toms.gsfc.nasa.gov/pub/eptoms/images/aerosol/y[yyyy]/ea[yy][mm][dd].gif Transparent colors for overlays RGB(89,140,255) RGB(41,117,41) RGB(23,23,23) RGB(0,0,0)

An Application Program: Voyager Data Browser The web-program consists of a stable core and adoptive input/output layers The core maintains the state and executes the data selection, access and render services The adoptive, abstract I/O layers connects the core to evolving web data, flexible displays and to the a configurable user interface: –Wrappers encapsulate the heterogeneous external data sources and homogenize the access –Device Drivers translate generic, abstract graphic objects to specific devices and formats –Ports connect the internal parameters of the program to external controls –WDSL web service description documents Data Sources Controls Displays I/O Layer Device Drivers Wrappers App State Data Flow Interpreter Core Web Services WSDL Ports

SeaWiFS Satellite Aerosol Chemical Air Trajectory Map Boarder VIEW by Web Service Composition

Air Quality DatasetsDatasets Data are accessed from autonomous, distributed providers DataFed ‘wrappers’ provide uniform geo-time referencing Tools allow space/time overlay, comparisons and fusion Near Real Time Data Integration Delayed Data Integration Surface Air Quality AIRNOWO3, PM25 ASOS_STIVisibility, 300 sites METARVisibility, 1200 sites VIEWS_OL40+ Aerosol Parameters Satellite MODIS_AOTAOT, Idea Project GASPReflectance, AOT TOMSAbsorption Indx, Refl. SEAW_USReflectance, AOT Model Output NAAPSDust, Smoke, Sulfate, AOT WRFSulfate Fire Data HMS_FireFire Pixels MODIS_FireFire Pixels Surface Meteorology RADARNEXTRAD SURF_METTemp, Dewp, Humidity… SURF_WINDWind vectors ATADTrajectory, VIEWS locs.

Some of the Tools of DataFed Consoles: Data from diverse sources are displayed to create a rich context for exploration and analysis CATT: Combined Aerosol Trajectory Tool for the browsing backtrajectories for specified chemical conditions Viewer: General purpose spatio-temporal data browser and view editor applicable for all DataFed datasets

Sulfate in the Northeast Sahara Dust in the Gulf Fires in the Southeast Time Series Console: Southeast Analyst Console Applications: Sulfate Episode: 8/27/04