The Earth System Grid: A Visualisation Solution Gary Strand

Introduction

Acknowledgments PI’s –Ian Foster (ANL) –Don Middleton (NCAR) –Dean Williams (LLNL) ESG Development Team –Veronika Nefedova (ANL) –Ann Chervenak (ISI/USC) –Carl Kesselman (ISI/USC) –David Bernholdt (ORNL) –Kasidit Chanchio (ORNL) –Line Pouchard (ORNL) –Alex Sim (LBNL) –Arie Shoshani (LBNL) –Bob Drach (LLNL) –Dave Brown (NCAR) –Gary Strand (NCAR) –Jose Garcia (NCAR) –Luca Cinquini (NCAR) –Peter Fox (NCAR)

Current Practices oScientist (or others) wants a visualisation oVisualisation person gets appropriate data after verifying with data manager as to the name, location, total size, etc. oData moved to “local” machine that has visualisation tools oVisualization created on “local” machine oHopefully, someone remembers to archive the visualisation

Simple Vis Example

Problems in the process oWhat if the data cannot be found (e.g. we have 1.2 million files, 73 TB of data), or the data manager is unavailable? oWhat if there isn’t enough disk space or sufficient other resources? oWhat if a better visualisation tool is located elsewhere? oWhat if the visualisation should be shared? oWhat if the visualisation is lost? o ESG is part of the answers to these questions

What is ESG? ORNL: Climate storage & computational resources ORNL: Climate storage & computational resources LANL: Next generation coupled models & computing LANL: Next generation coupled models & computing ANL: Computational grids, & grid-based applications ANL: Computational grids, & grid-based applications USC/ISI: Computational grids, & grid-based applications USC/ISI: Computational grids, & grid-based applications NCAR: Climate change predication and scenarios NCAR: Climate change predication and scenarios LBNL: Climate storage facility LBNL: Climate storage facility LLNL: Model diagnostics & inter-comparison LLNL: Model diagnostics & inter-comparison

ESG Architecture Tomcat servlet engine Tomcat servlet engine MCS Metadata Cataloguing Services MCS Metadata Cataloguing Services RLS Replica Location Services RLS Replica Location Services SOAP RMI MyProxy server MyProxy server MCS client RLS client MyProxy client GRAM gatekeeper GRAM gatekeeper CAS Community Authorization Services CAS Community Authorization Services CAS client disk MSS Mass Storage System HPSS High Performance Storage System disk HPSS High Performance Storage System disk SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server openDAPg server openDAPg server gridFTP Striped server gridFTP Striped server LBNL LLNL ISI NCAR ORNL ANL

Solutions oWhat happens when data cannot be found, or the data manager is unavailable? Metadata catalogue service (MCS) Replica location service (RLS)

MCS and RLS and Metadata Services METADATA EXTRACTION METADATA EXTRACTION METADATA DISPLAY METADATA DISPLAY METADATA BROWSING METADATA BROWSING METADATA QUERY METADATA QUERY ESG CLIENTS API & USER INTERFACES Data & Metadata Catalog Dublin Core Database COARDS Database mirror Dublin Core XML Files COMMENTS XML Files METADATA HOLDINGS METADATA ANNOTATION METADATA ANNOTATION METADATA VALIDATION METADATA VALIDATION METADATA ACCESS (update, insert, delete, query) METADATA ACCESS (update, insert, delete, query) SERVICE TRANSLATION LIBRARY SERVICE TRANSLATION LIBRARY CORE METADATA SERVICES METADATA AGGREGATION METADATA AGGREGATION METADATA DISCOVERY METADATA DISCOVERY METADATA & DATA REGISTRATION METADATA & DATA REGISTRATION PUBLISHING HIGH LEVEL METADATA SERVICES SEARCH & DISCOVERY ADMINISTRATION BROWSING & DISPLAY ANALYSIS & VISUALIZATION

Solutions (contd.) oWhat if there isn’t enough disk space or sufficient other resources? Hierarchical Resource Manager (HRM)

HRM

Solutions (contd.) oWhat What if a better visualization tool is located elsewhere? Distributed visualization

CDAT Example of an ESG Script Access  The next-generation language, Python, is used to access the Earth System Grid (ESG) at LLNL Import cdms, vcs db = cdms.open(“ldap://localhost:389/database=demo,ou=PCMDI,o=LLNL,c=US”) f = db.open( “ncep_reanalysis_mo”) ds = f(‘ts’) x=vcs.init( ) x.plot(ds)

CDAT: Example of an ESG GUI Client Access

Solutions (contd.) oWhat if the visualization should be shared? Access Grid plus Visualisation Tool

Collaborative Environments Science Portals + AccessGrid: University of Michigan (Knoop, Hardin) Vegetation & Ecosystem Mapping Program (VEMAP) NCAR/SCD VETS/KEG Argonne National Labs

Conclusions Visualisation can require as many services and resources as the initial computation Many sites do not offer sufficient resources for the visualisations earth sciences require ESG provides, and will provide, the tools that enable visualisation on a grander scale

Conclusions (contd.) ESG tools enable better data access, better data knowledge, and the processes of collaboration for the needs of investigating, visualising, and learning