ESG The Earth System Grid (ESG) Presented by Don Middleton & Luca Cinquini NCAR Scientific Computing Division On Behalf of the ESG Team SCD Executive Committee February 25, 2003 Presented by Don Middleton & Luca Cinquini NCAR Scientific Computing Division On Behalf of the ESG Team SCD Executive Committee February 25, 2003

ESG The Earth System Grid l U.S. DOE Scidac funded R&D effort l Build an “Earth System Grid” that enables management, discovery, distributed access, processing, & analysis of distributed terascale climate research data A “Collaboratory Pilot Project” A “Collaboratory Pilot Project” l Build upon ESG-I, Globus Toolkit , DataGrid technologies, and deploy l Potential broad application to other areas

ESG The Earth System Grid ESG Team l ANL –Ian Foster (PI) –Veronika Nefedova –(John Bresenhan) –(Bill Allcock) l LBNL –Arie Shoshani –Alex Sim l ORNL –David Bernholdte –Kasidit Chanchio –Line Pouchard l LLNL/PCMDI –Bob Drach –Dean Williams (PI) l USC/ISI –Anne Chervenak –Carl Kesselman l NCAR –David Brown –Luca Cinquini –Peter Fox –Jose Garcia –Don Middleton (PI) –Gary Strand

ESG The Earth System Grid Basic Numbers l T42 CCSM (current, 280km) –7.5GB/yr, 100 years ->.75TB l T85 CCSM (140km) –29GB/yr, 100 years -> 2.9TB l T170 CCSM (70km) –110GB/yr, 100 years -> 11TB

ESG The Earth System Grid Capacity-related Improvements Increased turnaround, model development, ensemble of runs Increase by a factor of 10, linear data l Current T42 CCSM –7.5GB/yr, 100 years ->.75TB * 10 = 7.5TB

ESG The Earth System Grid Capability-related Improvements Spatial Resolution: T42 -> T85 -> T170 Increase by factor of ~ 10-20, linear data Temporal Resolution: Study diurnal cycle, 3 hour data Increase by factor of ~ 4, linear data CCM at T170 (70km)

ESG The Earth System Grid Capability-related Improvements Quality: Improved boundary layer, clouds, convection, ocean physics, Improved land model, river runoff, new sea ice Increase by another factor of 2-3, data flat Scope: Atmospheric chemistry (sulfates, ozone…) Biogeochemistry (carbon cycle, ecosystem dynamics) Middle Atmosphere Model Increase by another factor of 10+, linear data

ESG The Earth System Grid Approaching Mesoscale (i.e. “weather”) Resolution Courtesy of John Taylor, ANL

ESG The Earth System Grid Model Improvements cont. Grand Total: Increase compute by a Factor O( )

ESG The Earth System Grid Longer-term Missions - Observation of Key Earth System Interactions Terra Aura Aqua Landsat 7 Exploratory - Explore Specific Earth System Processes and Parameters and Demonstrate Technologies GRACE PICASSO Cloudsat QuikScat EO-1 ICEsatJason-1 SRTM VCL We Will Examine Practically Every Aspect of the Earth System from Space in This Decade Triana

ESG The Earth System Grid

ESG The Earth System Grid What Is “The Grid”? l Analogous to the “power grid” l A “megatrend”… l Foundations for a meta- OS? Central Concept: “Coordinated resource sharing and problem-solving in dynamic multi-institutional virtual organizations”

ESG The Earth System Grid The Globus Toolkit™ l Security (!) l Directory, Metadata, and Replica Services l Resource Management l Data Access and Management l Distributed Computation l Coming Soon – Open Grid Services Architecture (OGSA) –Reliable, persistent web services An Open Source Project

ESG The Earth System Grid Corporate Commitments… l Compaq l Cray l Sun l SGI l Veridian l Entropia l Microsoft l IBM l NEC l Fujitsu l Hitachi l Platform Computing l Cisco

ESG The Earth System Grid ESG: Challenges l Enabling the simulation and data management team l Enabling the research community in analyzing and visualizing results l Enabling broad multidisciplinary communities to access simulation results We need integrated “cyberinfrastructure” to enable smooth WORKFLOW for knowledge development: compute platforms, collaboration & collaboratories, data management, access, distribution, and analysis.

ESG The Earth System Grid ESG: Strategies l Move data a minimal amount, keep it close to computational point of origin when possible –Data access protocols, distributed analysis l When we must move data, do it fast and with a minimum amount of human intervention –Storage Resource Management, fast networks l Keep track of what we have, particularly what’s on deep storage –Metadata and Replica Catalogs l Harness a federation of sites –Globus Toolkit -> The Earth System Grid -> The UltraDataGrid

ESG The Earth System Grid Server Tera/Peta-scale Archive HRM Storage Resource Management tools for reliable staging, replication, transport Server Tera/Peta-scale Archive HRM Client Selection Control Monitoring HRM

ESG The Earth System Grid Grid + OpenDAP -Transparency -Performance -Security -Resource Mgmt -Analysis functions Typical Application Data (local) netCDF lib Application Data (remote) OpenDAP Client Application OpenDAP Via http Big Data (remote) ESG client Application ESG Grid + DODS OpenDAP Server ESG Server Distributed Application data Distributed Data Access Protocols OpenDAP Via Grid

ESG The Earth System Grid ESG: Metadata Services METADATA EXTRACTION METADATA EXTRACTION METADATA DISPLAY METADATA DISPLAY METADATA BROWSING METADATA BROWSING METADATA QUERY METADATA QUERY ESG CLIENTS API & USER INTERFACES Data & Metadata Catalog Dublin Core Database COARDS Database mirror Dublin Core XML Files COMMENTS XML Files METADATA HOLDINGS METADATA ANNOTATION METADATA ANNOTATION METADATA VALIDATION METADATA VALIDATION METADATA ACCESS (update, insert, delete, query) METADATA ACCESS (update, insert, delete, query) SERVICE TRANSLATION LIBRARY SERVICE TRANSLATION LIBRARY CORE METADATA SERVICES METADATA AGGREGATION METADATA AGGREGATION METADATA DISCOVERY METADATA DISCOVERY METADATA & DATA REGISTRATION METADATA & DATA REGISTRATION PUBLISHING HIGH LEVEL METADATA SERVICES SEARCH & DISCOVERY ADMINISTRATION BROWSING & DISPLAY ANALYSIS & VISUALIZATION

ESG The Earth System Grid Metadata l Co-developed NcML with Unidata l Finalizing a specific schema for PCM/CCSM l Addressing interoperability via the generation of DIF/FGDC l Addressing interoperability with digital libraries via the creation of Dublin Core l Experimenting with relational and native XML databases l Exploratory work for first-generation ontology l Catalog population begins in the next 30 days

ESG The Earth System Grid l For XML encoding of metadata (and data) of any generic netCDF file l Objects: netCDF, dimension, variable, attribute l Beta version reference implementation as Java Library ( ESG: NcML Core Schema netCDF nc:netCDFType nc:dimension nc:variable nc: attribute nc:values nc:VariableType

ESG Technology Demonstration

ESG The Earth System Grid TOMCAT Servlet engine TOMCAT Servlet engine MCS Metadata Cataloguing Services MCS Metadata Cataloguing Services RLS Replica Location Services RLS Replica Location Services SOAP RMI MyProxy server MyProxy server MCS client RLS client MyProxy client GRAM gatekeeper GRAM gatekeeper CAS Community Authorization Services CAS Community Authorization Services CAS client disk MSS Mass Storage System HPSS High Performance Storage System disk HPSS High Performance Storage System disk SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server openDAPg server openDAPg server CAS-enabled Striped-gridFTP server CAS-enabled Striped-gridFTP server LBNL LLNL ISI NCAR ORNL ANL The Earth System Grid Striped gridFTP client Striped gridFTP client gridFTP openDAPg server openDAPg server CAS-enabled Striped-gridFTP server CAS-enabled Striped-gridFTP server gridFTP openDAPg server openDAPg server CAS-enabled Striped-gridFTP server CAS-enabled Striped-gridFTP server gridFTP LAS Live Access Server LAS Live Access Server

ESG The Earth System Grid Collaborations & Relationships l CCSM Data Management Group l The Globus Project l Other SciDAC Projects: Climate, Security & Policy for Group Collaboration, Scientific Data Management ISIC, & High-performance DataGrid Toolkit l OPeNDAP/DODS (multi-agency) l NSF National Science Digital Libraries Program (UCAR & Unidata THREDDS Project) l U.K. e-Science and British Atmospheric Data Center l NOAA NOMADS and CEOS-grid l Earth Science Portal group (multi-agency, intnl.)

ESG Questions?

END