Presentation is loading. Please wait.

Presentation is loading. Please wait.

Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive Doug Schuster and Steve Worley NCAR.

Published byAnnabel McCarthy Modified over 8 years ago

Similar presentations

Presentation on theme: "Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive Doug Schuster and Steve Worley NCAR."— Presentation transcript:

1 Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive Doug Schuster and Steve Worley NCAR

2 Topic Outline Archive Content Data Discovery and Access

3 Archive Content Common Data Management Tools Data acquisition Metadata extraction and mapping Data archival

4 Data Management Tools Old System: Specialized Software to manage each data input. Inefficient Difficult to Scale RDA Metadata Database RDA Metadata Database Unidata Server University Server NWP Server NWP Server Online Disk Tape Storage GCMD Metadata Server GCMD Metadata Server RDA Data Server Specialized Software Package 2 Specialized Software Package 3 Specialized Software Package 1 New System: Common RDA tools that homogenize data management. Efficient Scalable RDA Data Management Common Tool Set

5 Archive Content Popular Datasets In situ and satellite based observations Gridded analysis and forecast model data High resolution re-analysis

6 Popular RDA Datasets Platform Observations Dataset TitleCoverageUpdate Frequency NCEP GDAS observations (PREPBUFR)Global 1999 - PresentDaily NCEP GDAS observations (BUFR)Global 1999 - PresentDaily NCEP GDAS Satellite Radiance Data (BUFR)Global 2004 - PresentDaily Unidata IDD GTS based observations (NetCDF)Global 2002 - PresentDaily NCEP operational observations (ON-29 Format)Global 1975 - 2007Fixed International Comprehensive Ocean-Atmosphere Data Set (ICOADS) Global 1662 – Present Monthly 16622010Global Platform Observations

7 Popular RDA Datasets Analysis and Forecast Model Data Dataset TitleCoverageUpdate Frequency Thorpex Interactive Grand Global Ensemble (TIGGE)Global 2006 - PresentHourly Unidata IDD (GFS 0.5deg, RUC 20km, NAM 12km) Global and Regional 2002 - Present Daily NCEP ETA/NAM (40km) North America 1995 - Present Monthly ECMWF Operational Deterministic (1.25 x 1.25 Deg)Global 1985 - PresentBi-Yearly NCEP GDAS Final Analysis (1x1 Deg)Global 1999 - PresentDaily NCEP OI Global SST (1x1 Deg)Global 1981 - PresentWeekly NOAA OI Global SST (0.25 x 0.25 Deg)Global 1981 - PresentMonthly Hadley Centre Global Sea Ice and SSTGlobal 1850 - PresentMonthly 18502010Analysis and Forecast Model Data

8 Popular RDA Datasets High Resolution Re-Analysis Dataset TitleCoverageUpdate Frequency ERA-15 (N80 Gaussian)Global 1979 - 1993Static Set ERA-40 (T159)Global 1957 - 2002Static Set ERA-Interim (N128 Gaussian)Global 1989 - PresentYearly 18902010High Resolution Re-Analysis JRA-25 (1.125 Deg Gaussian)Global 1979 – PresentYearly NCEP/DOE (T62)Global 1979 - PresentStatic Set NCEP/NCAR (T62)Global 1948 - PresentQuarterly NARR (32 x 32 km) North America 1979 - Present Quarterly CFSRGlobal 1979 - PresentTBD NOAA-CIRES 20 th CenturyGlobal 1890 – 2008Static Set

9 Data Discovery and Access Data Discovery Popular dataset list Google style search Drill down style search Enhanced content metadata

10 Data Discovery Log in Popular Dataset List Google Style Search Drill Down Style Search Enhanced Content Metadata

11 Data Discovery and Access Data Access Methods User created “dynamic” file list and download options Server default “static” file list and download options Delayed mode request and download options Specialized project portals

12 Data Access Internet Download Options Matrix Data Group Title Discover and Access Online Data Files for Immediate Download (40 TB) Online Data Example 1: User Created “Dynamic” File List and Access Options

13 Data Access Example 1: User Created “Dynamic” File List Select Desired Parameter(s) Select Type of ProductSelect Vertical LevelEnter Desired Date RangeUpdate the List

14 Data Access Example 1: User Created “Dynamic” File List Select Files for download Volume Summary Download as Single Tar File Download Full Files with Wget Script Download Select Parameters with cURL Script

15 Data Access Example 2: Server Default “Static” File List Online Data Example 2: Server Default Static File List Click on Magnifying Glass Icon to Access File Level Content Metadata Internet Download Options Matrix

16 Data Access Internet Download Options Matrix Discover and Request for Offline Data Files to be Staged Online for Delayed Mode Download (500 TB) Example 3: Request Access to Offline Data Files

17 Data Access Example 3: Request Access to Offline Data Files Email Notification when Delayed Mode Request is Ready for Download

18 Data Access Internet Download Options Matrix Email a Question Regarding Data Access

19 Special Project Portals TIGGE Portal –THORPEX Interactive Global Grand Ensemble Archive (TIGGE) –Supports Weather research leading to an integrated Global Interactive Forecast System Integrated across multiple international NWP Centers 10 NWP Centers contributing to TIGGE Future –Model Validation Portal (under design) –Upper Air Observations Portal (under development)

20 Special Project Portals TIGGE Portal

21 http://dss.ucar.edu

Download ppt "Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive Doug Schuster and Steve Worley NCAR."

Similar presentations

ECMWF 14-16 June 2006Slide 1 Access to ECMWF data for Research Manuel Fuentes Data and Services Section, ECMWF ECMWF Forecast Products User Meeting.

ECMWF June 2006Slide 1 Access to ECMWF data for Research Manuel Fuentes Data and Services Section, ECMWF ECMWF Forecast Products User Meeting.
Managing Dataset DOIs and Versions in a Changing Archive Steven Worley Bob Dattore Zaihua Ji National Center for Atmospheric Research Boulder, Colorado,

Managing Dataset DOIs and Versions in a Changing Archive Steven Worley Bob Dattore Zaihua Ji National Center for Atmospheric Research Boulder, Colorado,
Data management in SCD Steven Worley General Categories –The Mass Storage System –NCAR user file services (home directories) –Computer attached storage.

Data management in SCD Steven Worley General Categories –The Mass Storage System –NCAR user file services (home directories) –Computer attached storage.
New Resources in the Research Data Archive Doug Schuster.

New Resources in the Research Data Archive Doug Schuster.
“Where America’s Climate, Weather and Ocean Services Begin” NCEP CONDUIT UPDATE Brent A Gordon NCEP Central Operations January 31, 2006.

“Where America’s Climate, Weather and Ocean Services Begin” NCEP CONDUIT UPDATE Brent A Gordon NCEP Central Operations January 31, 2006.
ICOADS Archive Practices at NCAR JCOMM ETMC-III 9-12 February 2010 Steven Worley.

ICOADS Archive Practices at NCAR JCOMM ETMC-III 9-12 February 2010 Steven Worley.
The Research Data Archive at NCAR Doug Schuster and Steve Worley NCAR.

The Research Data Archive at NCAR Doug Schuster and Steve Worley NCAR.
The International Surface Pressure Databank (ISPD) and Twentieth Century Reanalysis at NCAR Thomas Cram - NCAR, Boulder, CO Gilbert Compo & Chesley McColl.

The International Surface Pressure Databank (ISPD) and Twentieth Century Reanalysis at NCAR Thomas Cram - NCAR, Boulder, CO Gilbert Compo & Chesley McColl.
Operational Dataset Update Functionality Included in the NCAR Research Data Archive Management System 1 Zaihua Ji Doug Schuster Steven Worley Computational.

Operational Dataset Update Functionality Included in the NCAR Research Data Archive Management System 1 Zaihua Ji Doug Schuster Steven Worley Computational.
NOAA National Climate Model Portal D. Michael Grogan Lead Systems Engineer National Climatic Data Center.

NOAA National Climate Model Portal D. Michael Grogan Lead Systems Engineer National Climatic Data Center.
Introduction Downloading and sifting through large volumes of data stored in differing formats can be a time-consuming and sometimes frustrating process.

Introduction Downloading and sifting through large volumes of data stored in differing formats can be a time-consuming and sometimes frustrating process.
October 16-18, 20011 Research Data Set Archives Steven Worley Scientific Computing Division Data Support Section.

October 16-18, Research Data Set Archives Steven Worley Scientific Computing Division Data Support Section.
EGU 2011 TIGGE, TIGGE LAM and the GIFS T. Paccagnella (1), D. Richardson (2), D. Schuster(3), R. Swinbank (4), Z. Toth (3), S.

EGU 2011 TIGGE, TIGGE LAM and the GIFS T. Paccagnella (1), D. Richardson (2), D. Schuster(3), R. Swinbank (4), Z. Toth (3), S.
TIGGE Archive Highlights. First Service Date ECMWF – October 2006 NCAR – October 2006 CMA – June 2007.

TIGGE Archive Highlights. First Service Date ECMWF – October 2006 NCAR – October 2006 CMA – June 2007.
Research Data at NCAR 1 August, 2002 Steven Worley Scientific Computing Division Data Support Section.

Research Data at NCAR 1 August, 2002 Steven Worley Scientific Computing Division Data Support Section.
AERONET Web Data Access and Relational Database David Giles Science Systems and Applications, Inc. NASA Goddard Space Flight Center.

AERONET Web Data Access and Relational Database David Giles Science Systems and Applications, Inc. NASA Goddard Space Flight Center.
Data for Climate and Energy Studies Steven Worley Computational and Information Systems Laboratory NCAR.

Data for Climate and Energy Studies Steven Worley Computational and Information Systems Laboratory NCAR.
TIGGE Data Archive and Access System at NCAR 5th GIFS-TIGGE Working Group South African Weather Service Pretoria 11-13 March 2008 Steven Worley Doug Schuster.

TIGGE Data Archive and Access System at NCAR 5th GIFS-TIGGE Working Group South African Weather Service Pretoria March 2008 Steven Worley Doug Schuster.
Scientific Investigations; Support from Research Data Archives for Joint Office for Science Support 26 February, 2002 Steven Worley SCD/DSS.

Scientific Investigations; Support from Research Data Archives for Joint Office for Science Support 26 February, 2002 Steven Worley SCD/DSS.
A Comparison of the Northern American Regional Reanalysis (NARR) to an Ensemble of Analyses Including CFSR Wesley Ebisuzaki 1, Fedor Mesinger 2, Li Zhang.

A Comparison of the Northern American Regional Reanalysis (NARR) to an Ensemble of Analyses Including CFSR Wesley Ebisuzaki 1, Fedor Mesinger 2, Li Zhang.

Similar presentations

Ads by Google