Architectures and Technologies Enabling the Diffusion of Atmospheric Science Information Rudolf B. Husar and Erin Robinson Washington University, St. Louis Presented at EGU, Vienna April 18, 2007 Data Application Think Networking!

The Transformational Effect of Networking “Networking has led to an unprecedented surge of productivity” Time Magazine, Person of the Year 2006, YOU These are opportunities to enable Earth Science through more networking But many resistances to networking exist that need to be overcome Information has become the main driver of progress Time and place are no longer barriers to participation and interaction The Web has become a medium participation - ‘Web 2.0’ phenomenon

Networking Multiplies Value Creation Application Data 1 User Stovepipe Value = 1 1 Data x 1 Program = 1 Enclosed Value-Creating Process - ‘Stovepipe’

ApplicationData Application Stovepipe 1 User Stovepipe Value = 1 1 Data x 1 Program = 1 5 Uses of Data Value = 5 1 Data x 5 Program = 5 Networking Multiplies Value Creation

Merging data may creates new, unexpected opportunities Not all data are equally valuable to all programs 1 User Stovepipe Value = 1 1 Data x 1 Program = 1 5 Uses of Data Value = 5 1 Data x 5 Program = 5 Open Network Value = 25 5 Data x 5 Program = 25 Data Stovepipe Application Networking Multiplies Value Creation

Agile Information System: Data Access, Processing and Products Public Manager Scientist Users other Provider NASA DAACs EPA Model EPA AIRNow other s Data Organizing Document Structure/Format Interfacing Value Adding Processes

Agile Information System: Data Access, Processing and Products Uniform Access Public Manager Scientist Users other Provider NASA DAACs EPA Model EPA AIRNow other s Data Organizing Document Structure/Format Interfacing Value Adding Processes Homogenizing Format profile Standard access Data as Service

Agile Information System: Data Access, Processing and Products Uniform Access Data Processing Web Service Chain Custom Processing SciFlo DataF ed Public Manager Scientist Users other Provider NASA DAACs EPA Model EPA AIRNow other s Data Organizing Document Structure/Format Interfacing Characterizing Display/Browse Compare/Fuse Characterize Value Adding Processes Homogenizing Format profile Standard access Data as Service

Agile Information System: Data Access, Processing and Products Uniform Access Data Processing Web Service Chain Custom Processing SciFlo DataF ed Products Reports Forecast Compli. Other Science Public Manager Scientist Users other Provider NASA DAACs EPA Model EPA AIRNow other s Data Analyzing Filter/Integrate Aggregate/Fuse Custom Analysis Organizing Document Structure/Format Interfacing Characterizing Display/Browse Compare/Fuse Characterize Value Adding Processes Homogenizing Format profile Standard access Data as Service

Value-Adding Processes Uniform Access Data Processing Web Service Chain Custom Processing SciFlo DataF ed Products Reports Forecast Compli. Other Science Public Manager Scientist Users other Provider NASA DAACs EPA Model EPA AIRNow other s Data Analyzing Filter/Integrate Aggregate/Fuse Custom Analysis Organizing Document Structure/Format Interfacing Characterizing Display/Browse Compare/Fuse Characterize Reporting Inclusiveness Iterative/Agile Dynamic Report Value Adding Processes Homogenizing Format profile Standard access Data as Service

Agile Information System: Data Access, Processing and Products Uniform Access Data Processing Web Service Chain Custom Processing SciFlo DataF ed Products Reports Forecast Compli. Other Science Public Manager Scientist Users other Control Provider NASA DAACs EPA Model EPA AIRNow other s Data Control Seeking Information Providing Information Negotiating & Market Space

System of Systems Global Earth Observing System of Systems - GEOSS Characteristics of System of Systems (SoS) Autonomous constituents managed/operated independently Independent evolution of each constituent SoS displays emergent behavior Must recognize, manage, exploit the characteristics: No stakeholder has complete SoS insight Central control is limited; distributed control is essential Users, must be involved throughout the life of a SoS

Lets agree on Space-Time-Parameter Data Access Query Protocol

Interoperability Stack: Key concept of the Web Connecting Machines and People IP – Internet Protocol Service Orientation Open Architecture Data Standards Amplify Individuals Connect Minds System components have to be interoperable at each layer

Uniform Access Data Processing Web Service Chain Custom Processing SciFlo DataF ed Products Reports Forecast Compli. Other Science Public Manager Scientist Users other Control Data Acquisition Provider NASA DAACs EPA Model EPA AIRNow other s Data Standard Data Query Language: Where? When? What? (Space-time query - WMS, WCS) GetCapabilities GetData Capabilities, ‘Profile’ Data Where? When? What? Which Format? Server Back End Std. Interface Client Front End Std. Interface QueryGetData Standards Where?BBOXOGC, ISO When?TimeOGC, ISO What?TemperatureCF FormatnetCDF, HDF..CF, EOS, OGC T2T1 Loosely Coupled Data Access through Standard Protocols Standard Messaging What data you have? Give me this data

Uniform Access Data Processing Web Service Chain Custom Processing SciFlo DataF ed Products Reports Forecast Compli. Other Science Public Manager Scientist Users other Control Data Acquisition Provider NASA DAACs EPA Model EPA AIRNow other s Data Web Services and Workflow for Loose Coupling Service Chaining & Workflow Workflow Software: Dynamic Linking Software Mashups Software Mashup: Coarse-grain Linking

Uniform Access Data Processing Web Service Chain Custom Processing SciFlo DataF ed Products Reports Forecast Compli. Other Science Public Manager Scientist Users other Control Data Acquisition Provider NASA DAACs EPA Model EPA AIRNow other s Data Collaborative Reporting and Dynamic Delivery Co Writing - Wiki ScreenCast Collaborative Analysis and Writing Wiki, Blogs, Group Annotations Dynamic Content Delivery: GoogleEarth, Screencasting…

Scientist Science DAACs Current info systems are project/program oriented and provide end-to-end solutions Info UsersData ProvidersInfo System AIRNow Public AIRNow Model Compliance Manager ‘Stovepipe’ and Federated Usage Architectures Landscape Part of the data resources of any project can be shared for re-use through DataFed Through the Federation, the data are homogenized into multi-dimensional cubes Data processing and rendering can then be performed through web services Each project/program can be augmented by Federation data and services

DataFed: 100+ Datasets Non-intrusively Federated Data are accessed from autonomous, distributed providers DataFed ‘wrappers’ provide uniform geo-time referencing Tools allow space/time overlay, comparisons and fusion Near Real Time Data Integration Delayed Data Integration Surface Air Quality AIRNOWO3, PM25 ASOS_STIVisibility, 300 sites METARVisibility, 1200 sites VIEWS_OL40+ Aerosol Parameters Satellite MODIS_AOTAOT, Idea Project GASPReflectance, AOT TOMSAbsorption Indx, Refl. SEAW_USReflectance, AOT Model Output NAAPSDust, Smoke, Sulfate, AOT WRFSulfate Fire Data HMS_FireFire Pixels MODIS_FireFire Pixels Surface Meteorology RADARNEXTRAD SURF_METTemp, Dewp, Humidity… SURF_WINDWind vectors ATADTrajectory, VIEWS locs.

Sample of Federated Datasets

A Sample of Datasets Accessible through ESIP Mediation Near Real Time (~ day) It has been demonstrated (project FASTNET) that these and other datasets can be accessed, repackaged and delivered by AIRNow through ‘Consoles’ MODIS Reflectance MODIS AOT TOMS Index GOES AOT GOES 1km Reflec NEXTRAD Radar MODIS Fire Pix NRL MODEL NWS Surf Wind, Bext

Summary Grand Convergence Will we make use of it? Third-party mediation can homogenize distributed ES data Agile SOA-based IS can deliver diverse info products to users Since 2005, one such IS, DataFed is used by EPA and in research However, more data need to be federated by the community Parting thoughts Think outside the stovepipe – Think networking Divide and Conquer, NO! Connect and Enable, YES! Thank you