GrADS-DODS Server An open-source tool for distributed data access and analysis Joe Wielgosz, Brian Doty, Jennifer Adams COLA/IGES - Calverton, MD

Slides:

Advertisements

Similar presentations

1 NASA CEOP Status & Demo CEOS WGISS-25 Sanya, China February 27, 2008 Yonsook Enloe.

Advertisements

Jennifer M. Adams and Brian Doty IGES/COLA

Unidata Seminar Series - 30 January 2004 OPeNDAP and THREDDS: Access and Discovery of Distributed Scientific Data Yuan Ho Ethan Davis UCAR Unidata.

Hyrax Installation and Customization ESIP ‘08 Summer Meeting Best Practices in Services and Data Interoperability Dan Holloway James Gallagher.

OPeNDAP’s Server4 Building a High Performance Data Server for the DAP Using Existing Software Building a High Performance Data Server for the DAP Using.

Integrating NOAA’s Unified Access Framework in GEOSS: Making Earth Observation data easier to access and use Matt Austin NOAA Technology Planning and Integration.

Brian Doty and Jennifer Adams

Streaming NetCDF John Caron July What does NetCDF do for you? Data Storage: machine-, OS-, compiler-independent Standard API (Application Programming.

7 +/- 2 Maybe Good Ideas John Caron June (1) NetCDF-Java (aka CDM) has lots of functionality, but only available in Java – NcML Aggregation – Access.

GrADS 1.9 and the GrADS-DODS Server Jennifer Adams, Brian Doty, Joe Wielgosz Center for Ocean-Land-Atmosphere Studies (COLA) AMS/IIPS 13 January 2004.

Best Practices to Promote Data Interoperability Chris Lynnes Joe Glassy Technology Infusion Working Group.

1 CS6320 – Why Servlets? L. Grewe 2 What is a Servlet? Servlets are Java programs that can be run dynamically from a Web Server Servlets are Java programs.

NETOP ONDEMAND What’s new in version 2.1? DECEMBER 09 NETOP ONDEMAND1.

Configuring PHP on IIS7 Making your application rock on IIS7 Taking advantage of the Windows platform Q&A at Open Space.

Quick Tour of the Web Technologies: The BIG picture LECTURE A bird’s eye view of the different web technologies that we shall explore and study.

TPAC Digital Library Talk Overview Presenter:Glenn Hyland Tasmanian Partnership for Advanced Computing & Australian Antarctic Division Outline: TPAC Overview.

Hyrax Installation and Customization Dan Holloway James Gallagher.

Unidata TDS Workshop THREDDS Data Server Overview October 2014.

Tools for accessing distributed in-situ data collections Donald W. Denbo, NOAA/PMEL-JISAO Jason E. Fabritz, NOAA/PMEL-JISAO Bernard J. Kilonsky, Sea Level.

CEOS/WGISS 20, Kyev, September 12-16, WTF-CEOP Implementation Plan #1 Status (WTF-CEOP first prototype, by JAXA) September 12, 2005 Osamu Ochiai.

Session 10 Windows Platform Eng. Dina Alkhoudari.

APAC, Perth, WA OPeNDAP (The Open-source Project for a Network Data Access Protocol) APAC Tutorial October 12, 2007

GrADS: Essential Component of COLA’s Cyberinfrastructure Brian Doty Jennifer Adams.

Unidata’s TDS Workshop TDS Overview – Part II October 2012.

Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.

OPeNDAP Developer’s Workshop Feb OPeNDAP 4 Data Server – Hyrax James Gallagher and Nathan Potter 21 Feb 2007.

Weathertop Consulting, LLC Wednesday, January 14, 2009 IIPS 11A.2 1 A General Purpose System for Server-side Analysis of Earth Science Data Roland Schweitzer.

1 Apache. 2 Module - Apache ♦ Overview This module focuses on configuring and customizing Apache web server. Apache is a commonly used Hypertext Transfer.

Mid-Course Review: NetCDF in the Current Proposal Period Russ Rew

CakePHP is an open source web development framework. It follows Model-View- Controller and is developed using PHP. IT is the basic for user to create.

Ensemble Handling in GrADS

Accomplishments and Remaining Challenges: THREDDS Data Server and Common Data Model Ethan Davis Unidata Policy Committee Meeting May 2011.

NcBrowse A Graphical netCDF/OPeNDAP Browser Donald Denbo 1 & John Osborne 2 1 UW/JISAO-NOAA/PMEL, 2 OceanAtlas Software

Integrated Model Data Management S.Hankin ESMF July ‘04 Integrated data management in the ESMF (ESME) Steve Hankin (NOAA/PMEL & IOOS/DMAC) ESMF Team meeting.

HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) 1HDF and HDF-EOS Workshop XII10/17/2008.

Opendap dev - meeting, Boulder, Feb 2007 OPeNDAP infrastructure in European Operational Oceanography T Loubrieu (IFREMER) T Jolibois (CLS)

Unidata TDS Workshop THREDDS Data Server Overview

Easily Serving and Accessing HDF-EOS2 Datasets Using DODS Technologies Richard Chinman, UCAR-IITA, DODS Project Manager

Supporting HDF5 in GrADS Jennifer M. Adams and Brian E. Doty IGES/COLA.

GDS – The GrADS/DODS Server Jim Kinter Center for Ocean-Land-Atmosphere Studies (COLA) NVODS Workshop 10 September 2003.

NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.

THREDDS Catalogs Ethan Davis UCAR/Unidata NASA ESDSWG Standards Process Group meeting, 17 July 2007.

Securely Synchronize and Share Enterprise Files across Desktops, Web, and Mobile with EasiShare on the Powerful Microsoft Azure Cloud Platform MICROSOFT.

Unidata’s TDS Workshop TDS Overview – Part I July 2011.

DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan.

An Update on COLA’s Software Development Jennifer M. Adams and Brian Doty.

The HDF Group Data Interoperability The HDF Group Staff Sep , 2010HDF/HDF-EOS Workshop XIV1.

Information Technology: GrADS INTEGRATED USER INTERFACE Maps, Charts, Animations Expressions, Functions of Original Variables General slices of { 4D Grids.

August 2003 At A Glance The IRC is a platform independent, extensible, and adaptive framework that provides robust, interactive, and distributed control.

1 Final Report on WTF-CEOP JAXA Prototype system Satoko Horiyama MIURA JAXA/Earth Observation Research Center.

Weathertop Consulting, LLC Server-side OPeNDAP Analysis – Concrete steps toward a generalized framework via a reference implementation using F-TDS Roland.

The New HYCOM.org hycom.org Web Portals and Data Access Services hycom.org – dev.hycom.org (soon to become ) dev.hycom.org

LAS and THREDDS: Partners for Education Roland Schweitzer Steve Hankin Jonathan Callahan Joe Mclean Kevin O’Brien Ansley Manke Yonghua Wei.

OPeNDAP’s Server4: Building a High Performance Data Server for the DAP Using Existing Software James Gallagher*, Nathan Potter*, Patrick West**, Jose Garcia**

1 Earth System Grid Center for Enabling Technologies OPeNDAP Services for ESG March 9, 2016 Peter Fox, Patrick West, Stephan Zednik RPI Performance Measures.

Enabling the Transition of CPC Products to GIS Format Brian Doty Jennifer Adams Michael Halpert Viviane Silva.

1 2.5 DISTRIBUTED DATA INTEGRATION WTF-CEOP (WGISS Test Facility for CEOP) May 2007 Yonsook Enloe (NASA/SGT) Chris Lynnes (NASA)

Security-Enhanced Linux Stephanie Stelling Center for Information Security Department of Computer Science University of Tulsa, Tulsa, OK

GO-ESSP The Earth System Grid The Challenges of Building Web Client Geo-Spatial Applications Eric Nienhouse NCAR.

9/21/04 James Gallagher Server-Side: The Basics This part of the workshop contains an overview of the two servers which OPeNDAP has developed. One uses.

Update on Unidata Technologies for Data Access Russ Rew

TSDS (HPDE DAP). Objectives (1) develop a standard API for time series-like data, (2) develop a software package, TSDS (Time Series Data Server), that.

Discover How You Can Increase Collaboration with External Partners While Reducing Your Cost in Managing an Extranet from the Azure Cloud MICROSOFT AZURE.

DreamFactory for Microsoft Azure Is an Open Source REST API Platform That Enables Mobilization of Data in Minutes across Frameworks and Storage Methods.

Created with Logic Apps on Azure for Health Care Innovation and Interoperability, Corolar Cloud Is a Smarter and Faster Health Care Integration Engine.

MERRA Data Access and Services

Access HDF5 Datasets via OPeNDAP’s Data Access Protocol (DAP)

Live Access Server (LAS)

OPeNDAP’s Server4: Building a High Performance Data Server for the DAP

Mark Quirk Head of Technology Developer & Platform Group

Presentation transcript:

GrADS-DODS Server An open-source tool for distributed data access and analysis Joe Wielgosz, Brian Doty, Jennifer Adams COLA/IGES - Calverton, MD OPeNDAP Working Conference - March 19, 2003 <dataset name = "test” file = "testdata/little_endian.ctl“>...

GrADS-DODS ServerJoe Wielgosz - Make GrADS-readable datasets - both gridded and in-situ - accessible across the network, to a diverse range of clients What the GDS can do Perform server-side analysis and comparisons against other distributed datasets serversclients and more... web browser LAS GDS OPeNDAP server GDS comparison data GRIB HDF4 netCDF BUFR Unidata IDV ncBrowse GrADS Ferret Matlab IDL OPeNDAP binary GrADS Java servlet

GrADS-DODS ServerJoe Wielgosz - What's new since 1.1 New data type support Station data - GrADS format and BUFR Remote OPeNDAP data Subsampling ("striding") for gridded data Core code refactored Anagram - generic data server framework Swappable, reusable modules Designed for efficiency - streaming I/O XML-based configuration, with more flexibility in: Dataset loading Logging Security Resource management Improved web interface Custom links to help, home, dataset info URL-based administration interface Scales better to 1000's of datasets Organizes data catalog into directories Faster startup and smarter caching

GrADS-DODS ServerJoe Wielgosz - Stable and fast COLA public GDS currently handling >600k hits/month Administrator-friendly Complete online documentation: <dataset name="test" file="testdata/big_endian.ctl" format="ctl" /> <log mode="rotate" frequency="monthly" file="log/gds.log" level="info" /> Install in minutes (really!) No root privileges needed Cross-platform Java and ANSI C Easy to configure Edit one (simple) XML file, and make updates on-the-fly Secure Restrict dataset access & resource usage by IP address And more... Automatic scans for new datasets Detailed logging Graceful handling of heavy loads Easily integrated with Apache...

GrADS-DODS ServerJoe Wielgosz - In use, well-received Positive response from: COLA scientists GrADS user community - research, corporate, hobbyists NOAA/CIRES CDC (earliest adopters outside COLA) Some public GDS servers: (google on "grads dods server") COLA Public Data Server: cola8.iges.org:9191 (soon to be wxdata.org) COLA Monsoon Data Server: monsoondata.org NOAA/CIRES CDC: FNMOC / GODAE: usgodae.org NCEP: nomad2.ncep.noaa.gov NASA / GSWP: voda.gsfc.nasa.gov:9090 NASA / NSIPP:beta.gsfc.nasa.gov:9090 plus activity at centers in France, Britain (BADC), and Japan...

GrADS-DODS ServerJoe Wielgosz - What to expect Continuing refinements to existing code - uncovering bugs (some suspected to still exist) - improving documentation where needed - addressing usability issues Station data support - Finishing up BUFR I/O interface in GrADS - Adding daily station reports to COLA public server - Releasing station-data-enabled GrADS client - Working on interoperability with other archives and clients Under consideration: - built-in metric collection - catalog server - login-based authentication - near-line data support - listening for ideas

GrADS-DODS ServerJoe Wielgosz - COLA thoughts on OPeNDAP "DODS" vs "OPeNDAP" With apologies, waiting for don't want to confuse user community New protocol I see DAP 4.0 as a chance to reduce complexity and increase robustness We all have limited development resources - keep size of protocol proportional Interoperability means common ground, not sum of all features Anagram / Server-side Java Java is an ideal server platform - stable, cross-platform, easy to code Consider Anagram framework + Java-DODS as servlet counterpart to DAP C++ CGI framework? Slot in custom functionality to a standard core - good for gateways, access control / resource limits, web services, etc. Analysis protocol Increasing usage at COLA - perceived by scientists as extremely valuable Time to think about interoperability? Complex issues involved Modeler's needs Grid / ESG II Authentication important for collaborative projects Can OPeNDAP provide a way for small shops to get Grid-enabled?