A PROPOSED EARTH SCIENCE COLLABORATORY K-S Kuo 1,2, Chris Lynnes 1, Rahul Ramachandran 3 1 NASA Goddard Space Flight Center, USA 2 Caelum Research Corporation, USA 3 University of Alabama-Huntsville, USA 7/27/111IGARSS 2011, Vancouver, Canada

Why ESC? 7/27/112IGARSS 2011, Vancouver, Canada Data Intensive Science Many forms and sources of data In situ measurements Remote sensing observations Model simulations Large volumes of data Effectiveness as a scientist Increasing proportion of effort in data management Threatening:ReproducibilityCorrectnessProductivity

What is an ESC? Vision of a rich model development/simulation and data analysis environment that: Provides access to various Earth Science models Facilitates model and analysis software development Provides access across a wide spectrum of Earth Science data Provides a diverse set of science analysis services and tools Supports the application of services and tools to data Supports collaboration, i.e. sharing of data, tools and results Supports discovery and publication of all science artifacts 7/27/113IGARSS 2011, Vancouver, Canada Basically, a new and natural place for Earth scientists to conduct their work and collaborate with others!

7/27/114IGARSS 2011, Vancouver, Canada The Situation Today Islands of data and services with selective connectivity Data Center A Data Center C Data Center B

High-Level View 7/27/115IGARSS 2011, Vancouver, Canada Cyberinfrastructure Tool Library Data Library Laboratory Notebook Workflow Mediator Data Centers

7/27/116IGARSS 2011, Vancouver, Canada Tool Library Discovery Social o Sharing o Tagging o Discussion Configuration Management o Testing o Versioning Packager autoconf RPM Web wrapper Packager autoconf RPM Web wrapper P ROVISIONED GrADS IDL MatLab ncl nco cdat C OMMUNITY Quality filter Coincidence Feature detection Event service Visualization C ONTRIBUTED [Tool 1] [Tool 2] [Tool 3] [Tool 4] [Tool 5] … P ERSONAL [Tool 1] [Tool 2] [Tool 3] [Tool 4] [Tool 5] …

7/27/117IGARSS 2011, Vancouver, Canada Data Library Cache Discovery Social o Sharing o Tagging o Discussion Configuration Management o Testing o Versioning Packager data probe format check metadata wizard Packager data probe format check metadata wizard P ROVISIONED EOSDIS C OMMUNITY Field campaigns MEaSUREs ACCESS Validation C ONTRIBUTED [Dataset 1] [Dataset 2] [Dataset 3] [Dataset 4] [Dataset 5] … P ERSONAL [Dataset 1] [Dataset 2] [Dataset 3] …

7/27/118IGARSS 2011, Vancouver, Canada Workflow Library Discovery Social o Sharing o Tagging o Discussion Configuration Management o Testing o Versioning Workflow Library Discovery Social o Sharing o Tagging o Discussion Configuration Management o Testing o Versioning Packager Workflow editor Packager Workflow editor P ROVISIONED Processing Algorithms C OMMUNITY GeoBrain SciFlo Data Mining Giovanni C ONTRIBUTED [Workflow 1] [Workflow 2] [Workflow 3] [Workflow 4] [Workflow 5] … P ERSONAL [Workflow 1] [Workflow 2] [Workflow 3] …

7/27/119IGARSS 2011, Vancouver, Canada Laboratory Notebook Discovery Social o Sharing o Tagging o Discussion Configuration Management o Versioning Packager Project Manager Experiment manager Notebook editor Packager Project Manager Experiment manager Notebook editor P ROVISIONED Tutorials User guides Example uses Educational packages C OMMUNITY Project results Publications Example cases Educational packages P ROJECT [Project 1] [Project 2] [Project 3] [Project 4] [Project 5] … P ERSONAL Notes Journals …

7/27/1110IGARSS 2011, Vancouver, Canada Mediator Mediates tool interaction with data OPeNDAP – a common data model (accessible by most tools) Custom modules reformat data for the rest of the tools Ontology matches tools with data, and vice versa.

Cyberinfrastructure Services used by all other components Securityauthenticationauthorization code audit/padded cell integrity checking SocialtaggingsharingdiscussionsgroupsCloud elastic provisioned storage and computing Discovery data, tools, workflows, experiments search by keyword, variable, time, author Information Management provenance identifiers archive Semantic Web data ontology tools ontology 7/27/1111IGARSS 2011, Vancouver, Canada

Key Advantages of ESC Tool availability will be a force multiplier More tools will be usable with more datasets More tools will be more available to more users Knowledge sharing evolves from text on paper to a rich mixture of data, tools, workflows and articles A “wikihow” for Earth Science data analysis Incorporating live data, services and workflows ESC maintains a record of the analysis process Share, repeat, build upon analysis techniques Transparency of the process is built in 7/27/1112IGARSS 2011, Vancouver, Canada

Prior Art Talkoot, myExperiment.org – workflow sharing, virtual notebooks Earth System Grid – provisioned tools, format standards/checkers NASA Earth Exchange (NEX) Land Information System – OPeNDAP as access infrastructure Earth Science Modeling Framework – programmatic approach to integration Giovanni, LAS – community services/tools Canadian Space Science Data Portal (EOS, Feb. 22, 2011) Nebula – cloud provisioning 7/27/1113IGARSS 2011, Vancouver, Canada

A Use Case GPM Precipitation Retrieval Algorithm Development 7/27/1114IGARSS 2011, Vancouver, Canada GPM Core Satellite: Dual-Frequency Precipitation Radar (JAXA) and GPM Microwave Imager (NASA) GPM Constellation: International partner satellites with mostly microwave radiometers Retrieval algorithms – 3 types Radar-onlyRadiometer-onlyRadar-radiometer-combined Participants in algorithm development are distributed in Japan, NASA centers (GSFC, MSFC, JPL), NCAR, and universities (FSU, Uwisc, etc.)

A Use Case GPM Algorithm Development – Current Situation 7/27/1115IGARSS 2011, Vancouver, Canada Interdependence among 3 types of algorithms Communication/Coordination – Narrow bandwidth Periodic workshop meetings and teleconferences Data access – Duplicative Each location/group has a copy or subset of required data Sharing of data/tools – Individual, not concerted through ftp/ Knowledge sharing – Delayed

A Use Case GPM Algorithm Development – with ESC 7/27/1116IGARSS 2011, Vancouver, Canada Tools Data ESC Client mySci Cat. A Tools Data ESC Client mySci Cat. Z Cloud VM Image ToolsData A VM Image ToolsData B Community Catalog ESC

A Use Case GPM Algorithm Development – Multi-level Membership D C B A K J I H G F E GPM Radar- Only Radiometer- Only Combined Algorithm M L

A Use Case GPM Algorithm Development – in ESC 7/27/1118IGARSS 2011, Vancouver, Canada Enhanced communication/coordination – wide bandwidth Efficient data access – less duplication Improved sharing – more pervasive Effective knowledge sharing – immediate

Thank you! 7/27/1119IGARSS 2011, Vancouver, Canada

Why now? Because we can do it (finally)! Advances in standards acceptance and implementation (OPeNDAP, autoconf) A consistent, loosely coupled architecture encapsulates complexity and maximizes flexibility Social networking has reached the mainstream Key lessons can be learned from prior efforts The need is growing Interest in working with multiple datasets is growing Calls for transparency and reproducibility are growing 7/27/1120IGARSS 2011, Vancouver, Canada

What’s New? Macro View (forest-level) Systematic approach to making data available to services and vice versa Integration of all major analysis components Consistent view of all architectural components Cyberinfrastructure services for all architectural components Micro View (tree-level): Nothing! 7/27/1121IGARSS 2011, Vancouver, Canada

How to move forward? Option 1 RFC to community on feasibility, challenges, approach Followed by RFPs for component and integration Option 2 Narrow end-to-end prototype Followed by refactoring and broadening 7/27/1122IGARSS 2011, Vancouver, Canada