Earth System CoG and the Earth System Grid Federation: A Partnership for Improved Data Management and Project Coordination BESSIG March 18, 2014 Boulder, CO Sylvia Murphy (NOAA/CIRES) Luca Cinquini (JPL/NOAA), Cecelia DeLuca (NOAA/CIRES), Allyn Treshansky (NOAA/CIRES)

Presentation Outline Overview of ESGF ESGF Usage Overview of CoG CoG Capabilities Future Work

Overview of ESGF The Earth System Grid Federation (ESGF) is a multi-agency, international collaboration of people and institutions working together to build an open source software infrastructure for the management and analysis of Earth Science data on a global scale. ESGF is a system of distributed and federated Nodes that interact dynamically through a Peer-To-Peer (P2P) paradigm. A client (browser or program) can start from any Node in the federation and discover, download and analyze data from multiple locations as if they were stored in a single central archive.

ESGF Usage CMIP5 (Phase 5 of the Climate Modeling Intercomparison Project)… possibly the largest coordinated scientific modeling effort of all time. » 40+ models, 25+ modeling centers, 17 countries » global, distributed archive comprising 2.5 PB of data obs4MIPs: NASA and DoE observations packaged as CMIP5 output for easy comparison ana4MIPs: reanalysis data CORDEX: regional climate models, 2PB data TAMIP: atmospheric models intercomparison GeoMIP: geo-engineering models intercomparison DCMIP: dynamic core models intercomparison

Overview of CoG CoG is a collaboration environment and hub to connect projects in the Earth sciences. It hosts software development projects, model intercomparison projects (MIPS), and university short-courses or workshops. It includes a configurable search to data on ANY ESGF data node. It provides projects with a wiki and customizable navigation to wiki content. It contains an ontology for the description and management of projects and provides a consolidated look at this content across a projects network. It contains a file server for documents and images. It provides services for Earth system model metadata collection and display. Some of the 74 projects hosted on CoG include: Ana4MIPs Obs4MIPs National Climate Predictions and Projections Platform (NCPP) Climate Informatics (University of Michigan) Earth System Documentation (ES- DOC) NOAAs High Impact Weather Prediction Project (HIWPP) Earth System Prediction Capability (ESPC) Dynamical Core Model Intercomparison Project (DCMIP)

Customizable Data Services…Interfacing with ESGF Search widget can be turned on/off. Search can be narrowed to any ESGF node and to any project (e.g. CMIP). Search facets can be created, deleted, and grouped. Help text can be added to the top of the search page. Search results can be saved to a Data Cart associated with a user. Items in the Data Cart persist. Search results can be: – Forwarded to the Live Access Server (LAS) for simple visualization. – Downloaded directly via a WGET script. – Associated with model metadata if it exists.

ESGF Search Customization

Data Cart Items in the Data Cart can be sent individually or collectively to LAS or WGET. The Data Cart is associated with a user and not a project.

Show Metadata

Wiki and Collaboration Tools The CoG layout is color- coded: The right-hand side (dark yellow) is where services (data, news, project connectivity) are located. The Upper Navigation bar (dark teal) contains links to project-level metadata. On the left (light teal) is an auto-generated navigation system created when projects develop freeform content. The central portion of the site is a wiki that allows projects to create their own content. Screenshot of the CoG project workspace for the 2012 Dynamical Core Model Intercomparison (DCMIP) Workshop.

Project Networks and the Project Browser Projects in CoG are arranged in a hierarchy of Parents, Peers, and Children. The Project Browser displays the network and allows for inter-project navigation. Projects can be tagged with keywords and projects can be searched for using keywords.

Project-level Metadata Roll-up Management of information is a major problem in projects that involve many sub-projects, partners, multiple leads, and many resources. CoG acts as an index into project information that is necessary for coordination and collaboration and enables people responsible for overall coordination to quickly get consolidated views of information. This example shows the Partners feature that allows projects to list their project partners and include a logo for each. Below the list for ED-DOC is a consolidated view of the partners for ES-DOCs peer projects.

CoG Schema The CoG schema contains classes to describe software development projects, short- courses or meetings, and overall project coordination. Projects select which metadata to display via a simple web form. Project-level metadata is linked in standardized locations via the upper navigation bar.

UML Diagram

Resources Resources are pointers to data, files, and URLs. Resources folders can be created, moved, and deleted. Projects can turn on a set of standardized Resources folders (e.g. Presentations, Minutes). Saved data searches can be saved as a Resource. Each Resource can have a private wiki-based notes page to facilitate discussions.

News News is a way to send announcements across a project network. News is visible in the news widget on any targeted project. News will be added to social media (Google+, Facebook, Twitter, RSS) in a future release.

Model Metadata Services The CoG Team is partnering with the international Earth System Documentation (ES- DOC) project to develop and use an Earth System Model metadata entry and view capability. The ES-DOC Viewer is a lightweight JavaScript plugin that will display any Common Information Model (CIM) record. The ES-Questionnaire collects standardized CIM metadata through a high-customizable web form. The output is saved to a community CIM repository.

Future Work CoG – ESGF Integration (through summer 2014): – CoG is going to replace the ESGF web front end. – CoG will be federated so that projects hosted on one CoG-ESGF instance will be visible on others. – OpenID access added. – Look and feel will be more customizable to meet institutional branding requirements. Possible other features: – Be able to export the CoG ontology. – Be able to list non-hosted projects the Project Browser. – Be able to version the wiki. – Enable RSS and social networking for news. – Enable non-wiki links in the left navigation bar.

Questions? Questions and contacts: CoG: ESRL ESGF data node: PCMDI ESGF data node: JPL ESGF data node: