1. Federated Network for Sharing Air Quality Data and Processing Services Center for Air Pollution Impact and Trend Analysis (CAPITA) Washington University, St. Louis, MO 63130 April 2005, rhusar@me.wustl.edu DRAFT Project Coordinators: Software Architecture: R. Husar Software Implementation: K. Höijärvi Data and Applications: S. Falke, R. Husar

2. AQ Data and Analysis: Challenges and Opportunities Inform Public AQ Compliance Forecast AQ Status & Trends Satellite Devel. Network Asses. Manage Hazards ……… The adoptive AQ management paradigm requires and agile supporting info system AQ Compliance Status & Trends Inform Public Manage Hazards Forecast AQ Satellite Devel. Network Asses. AQ Compliance Status & Trends Inform Public Manage Hazards Forecast AQ

3. Data Flow & Processing in AQ Management AQ DATA EPA Networks IMPROVE Visibility Satellite-PM Pattern METEOROLOGY Met. Data Satellite-Transport Forecast model EMISSIONS National Emissions Local Inventory Satellite Fire Locs Status and Trends AQ Compliance Exposure Assess. Network Assess. Tracking Progress AQ Management Reports ‘Knowledge’ Derived from Data Primary Data Diverse Providers Data ‘Refining’ Processes Filtering, Aggregation, Fusion

4. Loosely Coupled InfoSystems: Flow of Data and Flow of Control Provider Push User Pull Each management task can has a need for ‘actionable’ information that can be used for decision making. Thus, ideally, the consumers/managers should specify their information needs and other features of the supporting Infosystem However, consumers my not be able to satisfy their information needs for several reasons: they may not be aware of all the available info resources, particularly in fast-changing conditions the available info resources may require further processing, un-available to the customer ---- The information resources and tools are supplied by the data providers, custodians or integrator-mediators Providers and custodians can help ‘pushing’ the information toward the consumers by making it accessible and attractive to the users Providers are not aware and can not predict (at generation time) of all the future consumers of their information However, since in loosely coupled systems, the the choice of which information is actually used is made by the by the consumer Data as resource - Thus, data consumers, providers and mediators together form the info system Flow of Data Flow of Control AQ DATA METEOROLOGY EMISSIONS DATA Informing Public AQ Compliance Status and Trends Network Assess. Tracking Progress Data to Knowledge Transformation

5. Data Flow and Flow Control in AQ Management: User selects data and mediator ‘views’ Provider Push User Pull Data are supplied by the provider and exposed on the ‘smorgasbord’ However, the choice of which data is used is made by the user Thus, data consumers, providers and mediators together form the info system Flow of Data Flow of Control AQ DATA METEOROLOGY EMISSIONS DATA Informing Public AQ Compliance Status and Trends Network Assess. Tracking Progress Data to Knowledge Transformation

6. DataFed Description DataFed Vision Better air quality management and science through by effective use of relevant data DataFed Goals Facilitate the access and flow of atmospheric data from provider to users Support the development of user-driven data processing value chains P articipate in specific application projects Approach: Mediation Between Users and Data Providers DataFed assumes spontaneous, autonomous emergence of AQ data (a la Internet) Non-intrusively wraps datasets for access by web services WS-based mediators provide homogeneous data views e.g. geo-spatial, time... End-user programming of data access and processing through WS composition (limited) Applications Building browsers and analysis tools for distributed monitoring data Serve as data gateway for user programs; web pages, GIS, science tools DataFed is currently focused on the mediation of air quality data

7. Mediator-Based Integration Architecture (Wiederhold, 1992) The job of the mediator is to provide an answer to a user query (Ullman, 1997)Ullman, 1997 In database theory sense, a mediator is a view of the data found in one or more sources Heterogeneous sources are wrapped by translation software local to global language Mediators (web services) obtain data from wrappers or other mediators and process it … Wrapper Service User QueryViews Heterogeneous Data

8. DataFed Multidimensional Data Model 4 D Geo-Environmental Data Cube (X, Y, Z, T) Environmental data represent measurements in the physical world which has space (X, Y, Z) and time (T) as its dimensions. The specific inherent dimensions for geo-environmental data are: Longitude X, Latitude Y, Elevation Z and DateTime T. The needs for finding, sharing and integration of geo- environmental data requires that data are ‘coded’ in this 4D data space – at the minimum.

9. DataFed Software Software for the User Data Catalog for finding and browsing the metadata of registered datasets Dataset Viewer/Editor for browsing specific datasets, linked to the Catalog Data Views - geo-spatial, time, trajectory etc. views prepared by the user Consoles, collections of views on a web page for monitoring multiple datasets Mini-Apps, small web-programs using chained web services (e.g. CATT, PLUME) Software for the Developer Registration software for adding distributed datasets to the data federation Web services for executing data access, processing and rendering tasks Web service chaining facility for composing custom-designed data views DataFed Technologies and Architecture Form-based, semi-automatic, third-party wrapping of distributed data Web services (based web standards) for the execution of specific tasks Service Oriented Architecture for building loosely coupled application programs Software Issues Reliability: Distributed computing issues: network reliability, bandwidth, etc Chaining: Orchestrating distributed web services to act as a single application Links: Linking users to providers and other federations (e.g. OGC, OPenDAP)

10. Anatomy of a Wrapper Service: TOMS Satellite Image Data Given the URL template and the image description, the wrapper service can access the image for any day, any spatial subset using a HTTP URL or SOAP protocol: Wrapper classes are available for geo-spatial (incl. satellite) images, SQL servers, text files,etc. The mediator classes are implemented as web services for uniform data access, transformation and portrayal. src_img_width src_img_height src_margin_rightsrc_margin_left src_margin_top src_margin_bottom src_lon_min src_lat_max src_lat_min src_lon_max Image Description for Data Access: src_image_width=502 src_image_height=329 src_margin_bottom=105 src_margin_left=69 src_margin_right=69 src_margin_top=46 src_lat_min=-70 src_lat_max=70 src_lon_min=-180 src_lon_max=180 The daily TOMS images reside on the FTP archive, e.g. ftp://toms.gsfc.nasa.gov/pub/eptoms/images/aerosol/y2000/ea000820.gif ftp://toms.gsfc.nasa.gov/pub/eptoms/images/aerosol/y2000/ea000820.gif URL template: ftp://toms.gsfc.nasa.gov/pub/eptoms/images/aerosol/y[yyyy]/ea[yy][mm][dd].gif Transparent colors for overlays RGB(89,140,255) RGB(41,117,41) RGB(23,23,23) RGB(0,0,0)

11. Value-Added Processing in Service Oriented Architecture Control Data Chain 1 Chain 2 Chain 3 Peer-to-peer network representation Data Service Catalog User Data, services and users are distributed throughout the network Users compose data processing chains form reusable services Intermediate data are also exposed for possible further use Chains can be linked to form compound value-adding processes Service chain representation User Tasks: Fi nd data and services Compose service chains Expose output Chain 2 Chain 1 Chain 3 Data Service User Carries less Burden In service-oriented peer-to peer architecture, the user is aided by software ‘agents’

12. Generic Data Flow and Processing in DataFed DataView 1 DataProcessed Data Portrayed Data Process Data Portrayal/ Render Abstract Data Access View Wrapper Physical Data Abstract Data Physical Data Resides in autonomous servers; accessed by view- specific wrappers which yield abstract data ‘slices’ Abstract Data Abstract data slices are requested by viewers; uniform data are delivered by wrapper services DataView 2 DataView 3 View Data Processed data are delivered to the user as multi-layer views by portrayal and overlay web services Processed Data Data passed through filtering, aggregation, fusion and other web services

13. SeaWiFS Satellite Aerosol Chemical Air Trajectory Map Boarder VIEW by Web Service Composition

14. Service Flow Program for a VIEW Layer View

15. An Application Program: Voyager Data Browser The web-program consists of a stable core and adoptive input/output layers The core maintains the state and executes the data selection, access and render services The adoptive, abstract I/O layers connects the core to evolving web data, flexible displays and to the a configurable user interface: –Wrappers encapsulate the heterogeneous external data sources and homogenize the access –Device Drivers translate generic, abstract graphic objects to specific devices and formats –Ports connect the internal parameters of the program to external controls –WDSL web service description documents Data Sources Controls Displays I/O Layer Device Drivers Wrappers App State Data Flow Interpreter Core Web Services WSDL Ports

16. Datasets Used in FASTNET Data are accessed from autonomous, distributed providers DataFed ‘wrappers’ provide uniform geo-time referencing Tools allow space/time overlay, comparisons and fusion Near Real Time Data Integration Delayed Data Integration Surface Air Quality AIRNOWO3, PM25 ASOS_STIVisibility, 300 sites METARVisibility, 1200 sites VIEWS_OL40+ Aerosol Parameters Satellite MODIS_AOTAOT, Idea Project GASPReflectance, AOT TOMSAbsorption Indx, Refl. SEAW_USReflectance, AOT Model Output NAAPSDust, Smoke, Sulfate, AOT WRFSulfate Fire Data HMS_FireFire Pixels MODIS_FireFire Pixels Surface Meteorology RADARNEXTRAD SURF_METTemp, Dewp, Humidity… SURF_WINDWind vectors ATADTrajectory, VIEWS locs.

17. A Sample of Datasets Accessible through ESIP Mediation Near Real Time (~ day) It has been demonstrated (project FASTNET) that these and other datasets can be accessed, repackaged and delivered by AIRNow through ‘Consoles’ MODIS Reflectance MODIS AOT TOMS Index GOES AOT GOES 1km Reflec NEXTRAD Radar MODIS Fire Pix NRL MODEL NWS Surf Wind, Bext

18. FASTNET: Inter-RPO pilot project, through NESCAUM, 2004 Web-based data, tools for community use Built on DataFed infra- structure, NSF, NASA Project fate depends on sponsor, user evaluationFASTNET:

19. Some of the Tools Used in FASTNET –Data Catalog –Data Browser –PlumeSim, Animator –Combined Aerosol Trajectory Tool (CATT) Consoles: Data from diverse sources are displayed to create a rich context for exploration and analysis CATT: Combined Aerosol Trajectory Tool for the browsing backtrajectories for specified chemical conditions Viewer: General purpose spatio-temporal data browser and view editor applicable for all DataFed datasets

20. Midwest HazeCam Image Console Image Archive and Browser Image Archive and Browser Hourly Midwest HazeCam Images are archived by DataFed data access system Archived images for all cameras can be browsed through this console HazeCam URL for a day: http://www.datafed.net/consoles/MWH_WebCams.asp?image_width=400&image_height=300&datetime=2005-01-31T13:00:00 http://www.datafed.net/consoles/MWH_WebCams.asp?image_width=400&image_height=300&datetime=2005-01-31T13:00:00 URL for a site and day: http://webapps.datafed.net/datasets/webcam/cincinnati/20050131-13mwhcincinnati.jpg http://webapps.datafed.net/datasets/webcam/cincinnati/20050131-13mwhcincinnati.jpg URLs can be embedded as links into emails, bookmarks, web pages, PPT and PDF files. Midwest HazeCam Image Browser Select date and timeSet image size and time MW HazeCam Console Other FASTNET Consoles

21. Aerosol Event Catalog: Web pages Catalog of generic ‘web objects’ – pages, images, animations that relate to aerosol events Each ‘web object’ is cataloged by location, time and aerosol type.

22. Distribution of Responsibility Distributed Responsibility in OpenDAP Responsibility The data lies with the data providers The data access protocol lies with OPeNDAP Application programs with the developers (Matlab,.. Excel…) Data discovery with the GCMD and NVODS Distributed Responsibility DataFed(??) The data lies with the data providers The wrappers and mediators with DataFed community Application programs with end user Data discovery with data & service registries