Presentation is loading. Please wait.

Presentation is loading. Please wait.

Fox 2 AISRP April 4-6, 2005  Earth System Grid  Grid-enabled OPeNDAP  Architecture - Server and Application access  Framework experience.

Published byLambert Clarke Modified over 8 years ago

Similar presentations

Presentation on theme: "Fox 2 AISRP April 4-6, 2005  Earth System Grid  Grid-enabled OPeNDAP  Architecture - Server and Application access  Framework experience."— Presentation transcript:

1

2 Fox (pfox@ucar.edu) 2 AISRP April 4-6, 2005  Earth System Grid  Grid-enabled OPeNDAP  Architecture - Server and Application access  Framework experience  Summary  Plans for the coming year Overview

3 Fox (pfox@ucar.edu) 3 AISRP April 4-6, 2005  The goal of ESG is to make climate data – particularly climate model data – an easily accessible community resource. The project is funded by the SciDAC program: Scientific Discovery through Advanced Computing.  Enabling researchers to understand and make effective use of very large, distributed climate datasets is critical. The broad strategy is to develop a collection of server-side capabilities – minimize the amount of data movement.  Multiple interfaces to ESG will allow researchers to focus on science rather than issues of data transfer, format, and data set manipulation.  Foundation is Globus Grid technology Earth System Grid Overview

4 Fox (pfox@ucar.edu) 4 AISRP April 4-6, 2005 Earth System Grid Portal

5 Fox (pfox@ucar.edu) 5 AISRP April 4-6, 2005 ESG: U.S. Collaborations & Development ORNL: Climate storage & computational resources ORNL: Climate storage & computational resources LANL: Next generation coupled models & computing LANL: Next generation coupled models & computing ANL: Computational grids, & grid-based applications ANL: Computational grids, & grid-based applications USC/ISI: Computational grids, & grid-based applications USC/ISI: Computational grids, & grid-based applications NCAR: Climate change predication and scenarios NCAR: Climate change predication and scenarios LBNL: Climate storage facility LBNL: Climate storage facility LLNL: Model diagnostics & inter-comparison LLNL: Model diagnostics & inter-comparison

6 Fox (pfox@ucar.edu) 6 AISRP April 4-6, 2005 ESG areas of development âAuthentication and Authorization services : application of Globus technologies for secure data management and access (PKI certificates, proxy delegation, Community Authentication Services, web interfaces) âData Transport Services: based on gridFTP protocol and implementation (high speed, tunable, multi-stream, reliable), extensions for multi-file management and connection to offline storage systems (Hierarchical Storage Management), and for transparent data access and operations (grid-enabled OPeNDAP) âMetadata services (for data management, access, search & discovery, annotation, analysis, etc.) âOther services: Data Analysis and Visualization, Task Management, Monitoring and Control, Replication of data, etc.

7 Fox (pfox@ucar.edu) 7 AISRP April 4-6, 2005 ESG: ESG-II Architecture

8 Fox (pfox@ucar.edu) 8 AISRP April 4-6, 2005 TOMCAT Servlet engine TOMCAT Servlet engine MCS Metadata Cataloguing Services MCS Metadata Cataloguing Services RLS Replica Location Services RLS Replica Location Services SOAP RMI MyProxy server MyProxy server MCS client RLS client MyProxy client GRAM gatekeeper GRAM gatekeeper CAS Community Authorization Services CAS Community Authorization Services CAS client disk MSS Mass Storage System HPSS High Performance Storage System disk HPSS High Performance Storage System disk SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server openDAPg server openDAPg server CAS-enabled Striped-gridFTP server CAS-enabled Striped-gridFTP server LBNL LLNL ISI NCAR ORNL ANL Striped gridFTP client Striped gridFTP client gridFTP openDAPg server openDAPg server CAS-enabled Striped-gridFTP server CAS-enabled Striped-gridFTP server gridFTP openDAPg server openDAPg server CAS-enabled Striped-gridFTP server CAS-enabled Striped-gridFTP server gridFTP LAS Live Access Server LAS Live Access Server

9 Fox (pfox@ucar.edu) 9 AISRP April 4-6, 2005 NCAR LBNL LLNL ISI ANL ORNL GSI CAS server CAS client MyProxy clientMyProxy server TOMCAT SECURITY services GRAM METADATA services FRAMEWORK services Auth metadata RLS MySQL RLS MySQL RLS MySQL RLS MySQL NERSC HPSS NCAR MSS DISK ORNL HPSS DATA storage The Earth System Grid THREDDS catalogs Xindice MySQL OGSA-DAISMCS TRANSPORT services gridFTP server/client HRM openDAPg server ANALYSIS & VIZ services NCL openDAPg clientLAS server CDAT openDAPg client MONITORING services SLAMON daemon TOMCAT AXIS

10 Fox (pfox@ucar.edu) 10 AISRP April 4-6, 2005 Metadata-centric view of ESG services METADATA SERVICES METADATA SERVICES USER AUTHENTICATION AND AUTHORIZATION USER AUTHENTICATION AND AUTHORIZATION ACCESS AND AUTHORIZATION METADATA DATA TRANSPORT LOCATION METADATA SYSTEM MONITORING AND CONTROL SYSTEM MONITORING AND CONTROL LOGGING METADATA DATA SEARCH & DISCOVERY CONTENT METADATA ANNOTATION & HISTORY METADATA DATA ANALYSIS & VISUALIZATION DATA ANALYSIS & VISUALIZATION AGGREGATION METADATA DATA BROWSING CATALOGUING METADATA

11 Fox (pfox@ucar.edu) 11 AISRP April 4-6, 2005 OPeNDAP and Grid systems âDODS since ~ 1995 was based on http and cgi-style architecture âTwo concerns âApplication support and performance of HTTP âHousekeeping abilities of cgi architecture âSolution: evolve OPeNDAP, the discipline neutral aspect of DODS

12 Fox (pfox@ucar.edu) 12 AISRP April 4-6, 2005 OPeNDAP ctd. âData transport protocol and access protocol separated âRevised server architecture âAddress Grid-style authentication âMemory management âException handling âAll these changes and retain interoperation with HTTP and cgi âAdvanced requirements: URL should support more than one dataset, or object, i.e. aggregation

13 Fox (pfox@ucar.edu) 13 AISRP April 4-6, 2005 OPeNDAP 3.x vs OPeNDAP-g Architecture Simple and easy to install One CGI process per URL request Limited memory management – external Limited scalability Limited status reporting to web server Returns data stream from one format Standalone server or httpd module Can manage multiple daemon processes Strong memory management – internal Reuse processes, scales Coupled to OPeNDAP server for status Returns multiple formats in a single stream, multiple protocols

14 Fox (pfox@ucar.edu) 14 AISRP April 4-6, 2005

15 Fox (pfox@ucar.edu) 15 AISRP April 4-6, 2005 Application development

16 Fox (pfox@ucar.edu) 16 AISRP April 4-6, 2005 Status âOperational/production release of standalone OPeNDAP server (no dependence on web server) for ESG âRun OPeNDAP server as a client to GridFTP or HTTP server âMulti-protocol support: file, http, GridFTP, ftp, etc. âFile format support: netCDF, CDF, FITS, CEDAR, … âRe-architected for aggregation support and performance âPortal application client in production, netCDF client operational âAuthentication is handled outside OPeNDAP server framework âURL syntax is more complex but more expressive âWill become part of community OPeNDAP release very soon

17 Fox (pfox@ucar.edu) 17 AISRP April 4-6, 2005 ESG: Framework experience  ESG is a highly collaborative effort allowing users to quickly access data (petabytes of raw or processed data in an application independent manner).  Payoffs of this distributed collaborative infrastructure have included: Distributed data-sharing, RLS works! SRM/HRM work! OPeNDAP-g works! Simplified data discovery of climate data, the work on metadata paid off! Scalability? Large-scale climate data processing and analysis via highly integrated portal Increased collaboration among climate research scientists, people use it! Aid in climate assessments and estimates of future climate variability and trends, IPCC!

18 Fox (pfox@ucar.edu) 18 AISRP April 4-6, 2005 ESG: Framework experience âTransport - GridFTP versus HTTP 3Server to server 3Very good performance 7Depends on a very specific version of GRIDftp server (stripped) 7Clients are not as capable due to ‘weight’ of globus, revert to HTTP âScalability and response times (data AND metadata) 3Framework architecture supports re-layered for tuning âService monitoring 3to support the distributed collaborative infrastructure 7need lots or all services to really make a production environment work  Try out ESG by visiting the website at: http://www.earthsystemgrid.org http://www.earthsystemgrid.org

19 Fox (pfox@ucar.edu) 19 AISRP April 4-6, 2005 Success?  Users are generally happy, developers are very happy  Exploited new technology components  Integration - when and how does it work and scale? 3XML -> SQL 3DODS -> OPeNDAP and OPeNDAP-g  Globus provides a suite of framework components, some are easier to integrate than others, some just don’t fit our use-cases and architecture  Data framework - e.g. OPeNDAP has been extremely successful  Carrying this to space science (solar-terrestrial)

20 Fox (pfox@ucar.edu) 20 AISRP April 4-6, 2005 Vision for building science cyberinfrastructure  Use-case, then requirements  Then derive architecture and choose technology components  Build a working system for users from the start  Get your funding source and community to commit to an evolving architecture  If you choose a major framework technology, e.g. Globus, OPeNDAP, THREDDS, partner with them

21 Fox (pfox@ucar.edu) 21 AISRP April 4-6, 2005 One paradigm Goal - find the right balance of data/model holdings, portals and client software that a researchers can use without effort or interference as if all the materials were available on his/her local computer. E.g. The Virtual Solar-Terrestrial Observatory (VSTO) is proposed to be: a distributed, scalable education and research environment for searching, integrating, and analyzing observational, experimental and model databases in the fields of solar, solar-terrestrial and space physics Comprises: a system-like framework which provides virtual access to specific data, model, tool and material archives containing items from a variety of space- and ground-based instruments and experiments, as well as individual and community modeling and software efforts bridging research and educational use

22 Fox (pfox@ucar.edu) 22 AISRP April 4-6, 2005 Virtual Observatory? Need better glue Basic problem: schema are categorized rather than developed from an object model/class hierarchy -> significantly limits non-human use. However, they all form the basis to organize catalog interfaces for all types of data, images, etc. This limits data systems utilizing frameworks and prevents frameworks from truly interoperating (SOAP, WSDL only a start) Directories, e.g. NASA GCMD, CEDAR catalog, FITS (flat) keyword/ value pairs, are being turned into ontologies (SWEET, VSTO) Markup languages, e.g. ESML, SPDML, ESG/ncML are excellent bases Evolve, recast, merge (where appropriate) using formal processes, tools with intended use in mind - for interface specifications, reasoning, validation, etc. beyond the usual search and access

23 Fox (pfox@ucar.edu) 23 AISRP April 4-6, 2005 Summary  Basic success in both data systems and data frameworks  Satisfying user and sponsor needs (from ‘just’ to ‘outstanding’)  Experience with Globus ranges from very good, to not ready for our need  Experience with OPeNDAP is very good, esp. with core services  Scalability and performance require an adaptable architecture which is something system-level interfaces can still hide from the user  Challenge - to bring these attributes to a framework, i.e. in which the user is more exposed

24 Fox (pfox@ucar.edu) 24 AISRP April 4-6, 2005 Plans  IDL application level access to new OPeNDAP server framework  Outreach to NASA communities/data centers to install and test new capabilities (server and client)  Joint development of accompanying semantic catalogs for Sun-Earth Connection datasets within the OPeNDAP framework  SPDML-enabled OPeNDAP server

Download ppt "Fox 2 AISRP April 4-6, 2005  Earth System Grid  Grid-enabled OPeNDAP  Architecture - Server and Application access  Framework experience."

Similar presentations

National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Data Grids for Collection Federation Reagan W. Moore University.

National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Data Grids for Collection Federation Reagan W. Moore University.
LEAD Portal: a TeraGrid Gateway and Application Service Architecture Marcus Christie and Suresh Marru Indiana University LEAD Project (http://lead.ou.edu)

LEAD Portal: a TeraGrid Gateway and Application Service Architecture Marcus Christie and Suresh Marru Indiana University LEAD Project (
30-31 Jan 2003J G Jensen, RAL/WP5 Storage Elephant Grid Access to Mass Storage.

30-31 Jan 2003J G Jensen, RAL/WP5 Storage Elephant Grid Access to Mass Storage.
Data Management Expert Panel - WP2. WP2 Overview.

Data Management Expert Panel - WP2. WP2 Overview.
A. Sim, CRD, L B N L 1 ANI and Magellan Launch, Nov. 18, 2009 Climate 100: Scaling the Earth System Grid to 100Gbps Networks Alex Sim, CRD, LBNL Dean N.

A. Sim, CRD, L B N L 1 ANI and Magellan Launch, Nov. 18, 2009 Climate 100: Scaling the Earth System Grid to 100Gbps Networks Alex Sim, CRD, LBNL Dean N.
Earth System Curator Spanning the Gap Between Models and Datasets.

Earth System Curator Spanning the Gap Between Models and Datasets.
Metadata Development in the Earth System Curator Spanning the Gap Between Models and Datasets Rocky Dunlap, Georgia Tech.

Metadata Development in the Earth System Curator Spanning the Gap Between Models and Datasets Rocky Dunlap, Georgia Tech.
OPeNDAP’s Server4 Building a High Performance Data Server for the DAP Using Existing Software Building a High Performance Data Server for the DAP Using.

OPeNDAP’s Server4 Building a High Performance Data Server for the DAP Using Existing Software Building a High Performance Data Server for the DAP Using.
1 SRM-Lite: overcoming the firewall barrier for large scale file replication Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory April, 2007.

1 SRM-Lite: overcoming the firewall barrier for large scale file replication Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory April, 2007.
1 Introduction to XML. XML eXtensible implies that users define tag content Markup implies it is a coded document Language implies it is a metalanguage.

1 Introduction to XML. XML eXtensible implies that users define tag content Markup implies it is a coded document Language implies it is a metalanguage.
Office of Science U.S. Department of Energy Grids and Portals at NERSC Presented by Steve Chan.

Office of Science U.S. Department of Energy Grids and Portals at NERSC Presented by Steve Chan.
Toni Saarinen, Tite4 Tomi Ruuska, Tite4 Earth System Grid - ESG.

Toni Saarinen, Tite4 Tomi Ruuska, Tite4 Earth System Grid - ESG.
Data Grids: Globus vs SRB. Maturity SRB  Older code base  Widely accepted across multiple communities  Core components are tightly integrated Globus.

Data Grids: Globus vs SRB. Maturity SRB  Older code base  Widely accepted across multiple communities  Core components are tightly integrated Globus.
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.

Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
Data Grid Web Services Chip Watson Jie Chen, Ying Chen, Bryan Hess, Walt Akers.

Data Grid Web Services Chip Watson Jie Chen, Ying Chen, Bryan Hess, Walt Akers.
TeraGrid Gateway User Concept – Supporting Users V. E. Lynch, M. L. Chen, J. W. Cobb, J. A. Kohl, S. D. Miller, S. S. Vazhkudai Oak Ridge National Laboratory.

TeraGrid Gateway User Concept – Supporting Users V. E. Lynch, M. L. Chen, J. W. Cobb, J. A. Kohl, S. D. Miller, S. S. Vazhkudai Oak Ridge National Laboratory.
Commodity Grid (CoG) Kits Keith Jackson, Lawrence Berkeley National Laboratory Gregor von Laszewski, Argonne National Laboratory.

Commodity Grid (CoG) Kits Keith Jackson, Lawrence Berkeley National Laboratory Gregor von Laszewski, Argonne National Laboratory.
The Virtual Solar-Terrestrial Observatory The Virtual Observatory Peter Fox HAO/ESSL/NCAR November 28, 2005.

The Virtual Solar-Terrestrial Observatory The Virtual Observatory Peter Fox HAO/ESSL/NCAR November 28, 2005.
Data Management Kelly Clynes Caitlin Minteer. Agenda Globus Toolkit Basic Data Management Systems Overview of Data Management Data Movement Grid FTP Reliable.

Data Management Kelly Clynes Caitlin Minteer. Agenda Globus Toolkit Basic Data Management Systems Overview of Data Management Data Movement Grid FTP Reliable.
Presented by The Earth System Grid: Turning Climate Datasets into Community Resources David E. Bernholdt, ORNL on behalf of the Earth System Grid team.

Presented by The Earth System Grid: Turning Climate Datasets into Community Resources David E. Bernholdt, ORNL on behalf of the Earth System Grid team.

Similar presentations

Ads by Google