1. Why so many data systems? Dickerson – ppt

2. Information as a Resource Shared not exchanged …

3. The Transformational Effect of Networking “Networking has led to an unprecedented surge of productivity” Time Magazine, Person of the Year 2006, YOU These are opportunities to enable Earth Science through more networking But many resistances to networking exist that need to be overcome Information has become the main driver of progress Time and place are no longer barriers to participation and interaction The Web has become a medium participation - ‘Web 2.0’ phenomenon

4. Networking Multiplies Value Creation Application Data 1 User Stovepipe Value = 1 1 Data x 1 Program = 1 Enclosed Value-Creating Process - ‘Stovepipe’

5. ApplicationData Application Stovepipe 1 User Stovepipe Value = 1 1 Data x 1 Program = 1 5 Uses of Data Value = 5 1 Data x 5 Program = 5 Networking Multiplies Value Creation

6. Merging data may creates new, unexpected opportunities Not all data are equally valuable to all programs 1 User Stovepipe Value = 1 1 Data x 1 Program = 1 5 Uses of Data Value = 5 1 Data x 5 Program = 5 Open Network Value = 25 5 Data x 5 Program = 25 Data Stovepipe Application Networking Multiplies Value Creation

7. The Network Effect: Less Cost, More Benefits through Data Multi-Use Program Public Data Organization Data Program Organization Data Program Data Orgs Develop Programs Programs ask/get Data Public sets up Orgs Pay only once Richer content Less Prog. Cost More Knowledge Less Soc. Cost More Soc. Benefit Data Re-Use Network Effect Data are costly resource – should be reused (recycled) for multiple applications Data reuse saves $$ to programs and allows richer knowledge creation Data reuse, like recycling takes some effort: labeling, organizing, distributing Data repositories/Systems

8. Data are costly resource – should be reused (recycled) for multiple applications Data reuse saves $$ to programs and allows richer knowledge creation Data reuse, like recycling takes some effort: labeling, organizing, distributing

9. Increasing the Size of the Pie Data are costly resource – should be reused (recycled) for multiple applications Data reuse saves $$ to programs and allows richer knowledge creation Data reuse, like recycling takes some effort: labeling, organizing, distributing Cost = 1 for single use Cost = 1.5 for 5 uses Benefit = 5 for 5 uses Benefit = 1 for single use

10. Data Re-Use and Synergy Data producers maintain their own workspace and resources (data, reports, comments). Part of the resources are shared by creating a common virtual resources. Web-based integration of the resources can be across several dimensions: Spatial scale:Local – global data sharing Data content:Combination of data generated internally and externally The main benefits of sharing are data re-use, data complementing and synergy. The goal of the system is to have the benefits of sharing outweigh the costs. Content User Local Global Virtual Shared Resources Data, Knowledge Tools, Methods User Shared part of resources

11. Federated Information System Data producers maintain their own workspace and resources (data, reports, comments). However, part of the resources are shared through a Federated Information System. Web-based integration of the shared resources can be across several dimensions: Data sharing federations: Open GIS Consortium (GIS data layers) NASA SEEDS network (Satellite data) NSF Digital Government EPA’s National Env. Info Exch. Network. VIEWSRPO NASA NAAPS RPO Federated Data System Data, Tools, Methods SharedPrivate RPO Other Federations Applications PM Policy Regulation Mitigation

12. Federated Information System Data producers maintain their own workspace and resources (data, reports, comments). However, part of the resources are shared through a Federated Information System. Web-based integration of the shared resources can be across several dimensions: Data sharing federations: Open GIS Consortium (GIS data layers) NASA SEEDS network (Satellite data) NSF Digital Government EPA’s National Env. Info Exch. Network. VIEWS RPO RPO Federated Data System Data, Tools, Methods SharedPrivate RPO Other Federations Applications PM Policy Regulation Mitigation Unidata Portal ESIP Portal Portal Data to be “dispersed” to multiple “portals” This brings data closer to the user Each portal can serve different clientele Conditions is open architecture so that the resources can be reconfigured into many different “views” through the different portals User communities

13. Smoke Event Public EPA 1. 2. 3. NAAQS Exc. Events States: AQ Warning NOAA Travel Advisories AQ Forecasting FAA Flight Advisories NASA Earth Obs: Public SatModis Mod Vis PM25 SatTOMS SatGOES Chem

14. Scientist Science DAACs Current info systems are project/program oriented and provide end-to-end solutions Info UsersData ProvidersInfo System AIRNow Public AIRNow Model Compliance Manager ‘Stovepipe’ and Federated Usage Architectures Landscape Part of the data resources of any project can be shared for re-use through DataFed Through the Federation, the data are homogenized into multi-dimensional cubes Data processing and rendering can then be performed through web services Each project/program can be augmented by Federation data and services

15. Applicable to: –Model Validation –Deliver Information to the Public –Track Trends – Accountability GEOSS

16. Data Acquisition and Usage Activities Need similar generic pic for analysis

17. Staged Data Integration? Staged portal Monitor Store Data 1 Monitor Store Data 2 Monitor Store Data n Monitor Store Data m Integrated Data1 Virtual Int. Data Integrated Data2 Integrated Data3 System integrates foreword from provider to the users So that user can find/monitor content User can navigate backwards toward the provider PoP – harvester Oodle! CNet …

18. Agile Information System: Data Access, Processing and Products Public Manager Scientist Users other Provider NASA DAACs EPA Model EPA AIRNow other s Data Organizing Document Structure/Format Interfacing Value Adding Processes

19. Agile Information System: Data Access, Processing and Products Uniform Access Public Manager Scientist Users other Provider NASA DAACs EPA Model EPA AIRNow other s Data Organizing Document Structure/Format Interfacing Value Adding Processes Homogenizing Format profile Standard access Data as Service

20. Agile Information System: Data Access, Processing and Products Uniform Access Data Processing Web Service Chain Custom Processing SciFlo DataF ed Public Manager Scientist Users other Provider NASA DAACs EPA Model EPA AIRNow other s Data Organizing Document Structure/Format Interfacing Characterizing Display/Browse Compare/Fuse Characterize Value Adding Processes Homogenizing Format profile Standard access Data as Service

21. Agile Information System: Data Access, Processing and Products Uniform Access Data Processing Web Service Chain Custom Processing SciFlo DataF ed Products Reports Forecast Compli. Other Science Public Manager Scientist Users other Provider NASA DAACs EPA Model EPA AIRNow other s Data Analyzing Filter/Integrate Aggregate/Fuse Custom Analysis Organizing Document Structure/Format Interfacing Characterizing Display/Browse Compare/Fuse Characterize Value Adding Processes Homogenizing Format profile Standard access Data as Service

22. Value-Adding Processes Uniform Access Data Processing Web Service Chain Custom Processing SciFlo DataFed Products Reports Forecast Compli. Other Science Public Manager Scientist Users other Provider NASA DAACs EPA Model EPA AIRNow other s Data Analyzing Filter/Integrate Aggregate/Fuse Custom Analysis Organizing Document Structure/Format Interfacing Characterizing Display/Browse Compare/Fuse Characterize Reporting Inclusiveness Iterative/Agile Dynamic Report Homogenizing Format profile Standard access Data as Service Information Value Chain

23. Agile Information System: Data Access, Processing and Products Uniform Access Data Processing Web Service Chain Custom Processing SciFlo DataF ed Products Reports Forecast Compli. Other Science Public Manager Scientist Users other Control Provider NASA DAACs EPA Model EPA AIRNow other s Data Control Seeking Information Providing Information Negotiating & Market Space

24. System of Systems Global Earth Observing System of Systems - GEOSS Characteristics of System of Systems (SoS) Autonomous constituents managed/operated independently Independent evolution of each constituent SoS displays emergent behavior Must recognize, manage, exploit the characteristics: No stakeholder has complete SoS insight Central control is limited; distributed control is essential Users, must be involved throughout the life of a SoS

25. Lets agree on Space-Time-Parameter Data Access Query Protocol

26. Interoperability Stack: Key concept of the Web Connecting Machines and People IP – Internet Protocol Service Orientation Open Architecture Data Standards Amplify Individuals Connect Minds System components have to be interoperable at each layer

27. Uniform Access Data Processing Web Service Chain Custom Processing SciFlo DataF ed Products Reports Forecast Compli. Other Science Public Manager Scientist Users other Control Data Acquisition Provider NASA DAACs EPA Model EPA AIRNow other s Data Standard Data Query Language: Where? When? What? (Space-time query - WMS, WCS) GetCapabilities GetData Capabilities, ‘Profile’ Data Where? When? What? Which Format? Server Back End Std. Interface Client Front End Std. Interface QueryGetData Standards Where?BBOXOGC, ISO When?TimeOGC, ISO What?TemperatureCF FormatnetCDF, HDF..CF, EOS, OGC T2T1 Loosely Coupled Data Access through Standard Protocols Standard Messaging What data you have? Give me this data

28. Uniform Access Data Processing Web Service Chain Custom Processing SciFlo DataF ed Products Reports Forecast Compli. Other Science Public Manager Scientist Users other Control Data Acquisition Provider NASA DAACs EPA Model EPA AIRNow other s Data Web Services and Workflow for Loose Coupling Service Chaining & Workflow Workflow Software: Dynamic Linking Software Mashups Software Mashup: Coarse-grain Linking

30. Uniform Access Data Processing Web Service Chain Custom Processing SciFlo DataF ed Products Reports Forecast Compli. Other Science Public Manager Scientist Users other Control Data Acquisition Provider NASA DAACs EPA Model EPA AIRNow other s Data Collaborative Reporting and Dynamic Delivery Co Writing - Wiki ScreenCast Collaborative Analysis and Writing Wiki, Blogs, Group Annotations Dynamic Content Delivery: GoogleEarth, Screencasting…

31. DataFed: 100+ Datasets Non-intrusively Federated Data are accessed from autonomous, distributed providers DataFed ‘wrappers’ provide uniform geo-time referencing Tools allow space/time overlay, comparisons and fusion Near Real Time Data Integration Delayed Data Integration Surface Air Quality AIRNOWO3, PM25 ASOS_STIVisibility, 300 sites METARVisibility, 1200 sites VIEWS_OL40+ Aerosol Parameters Satellite MODIS_AOTAOT, Idea Project GASPReflectance, AOT TOMSAbsorption Indx, Refl. SEAW_USReflectance, AOT Model Output NAAPSDust, Smoke, Sulfate, AOT WRFSulfate Fire Data HMS_FireFire Pixels MODIS_FireFire Pixels Surface Meteorology RADARNEXTRAD SURF_METTemp, Dewp, Humidity… SURF_WINDWind vectors ATADTrajectory, VIEWS locs.

32. Sample of Federated Datasets

34. A Sample of Datasets Accessible through ESIP Mediation Near Real Time (~ day) It has been demonstrated (project FASTNET) that these and other datasets can be accessed, repackaged and delivered by AIRNow through ‘Consoles’ MODIS Reflectance MODIS AOT TOMS Index GOES AOT GOES 1km Reflec NEXTRAD Radar MODIS Fire Pix NRL MODEL NWS Surf Wind, Bext

35. Summary Grand Convergence Will we make use of it? Third-party mediation can homogenize distributed ES data Agile SOA-based IS can deliver diverse info products to users Since 2005, one such IS, DataFed is used by EPA and in research However, more data need to be federated by the community Parting thoughts Think outside the stovepipe – Think networking Divide and Conquer, NO! Connect and Enable, YES! Thank you