BADC Workshop 2: BADC Services to Data Suppliers Royal Met. Soc. Conference – 14 September 2005 Ag Stephens et al.

2 Workshop Outline Purpose of Workshop –To briefly present BADC provision to Data Suppliers –To gain feedback from the user community Workshop plan – Presentation: BADC Services to Data Suppliers – Points for discussion… How can we serve Data Suppliers better? Which services need improving, and how? What new services are required? – Evaluation form

3 Introduction and scope -The BADC and its data suppliers -The NERC Data Policy Support to data suppliers -Data management planning -Archival, distribution & service infrastructure -Metadata -File names -Data format -Data submission -Campaign support -Getting help Discussion Presentation Outline

4 The BADC The NERC-designated Data Centre for atmospheric science is the BADC (under NCAS) It currently holds over 30 TB of data including NWP forecasts, climate runs, instrumental and satellite products It serves 7,000 users in the UK and overseas The BADC’s data suppliers NERC-funded researchers (e.g. through Directed Mode Programmes or using a NERC facility such as the FAAM or UFAM instruments) — also the BADC data users! Other research or data centres (e.g. Met Office, ECMWF, Eumetsat, ESA, NASA) International research programmes (e.g. NDSC, EC-funded such as NitroEurope) The BADC and its data suppliers

5 The NERC Data Policy stipulates: The Data Centres’ duties NERC grant holders’ duties, e.g. –Get acquainted with the NERC Data Policy Handbook (*) (*) –Offer the data generated by a NERC-funded project to the designated Data Centre –Ensure appropriate data custody, validation, documentation, cataloguing and dissemination –Maintain and promote data stewardship standards –Set up data protocols (conditions of submission, access and use) –Assist UK researchers in locating and accessing data, including fetching data from external sources –Handle data-related queries The NERC Data Policy

6 Data Management Planning At the outset of a research programme/project/experiment: Scoping study to determine: - scientific goals- external data needs - project duration - data sharing needs - staff and collaborators- investigators’ wishes - details on data to be produced & archived (nature, volume, flow,…) Data management plan (DMP) proposal and adoption (for large programmes): common dispositions and technical measures to meet the programme needs — in accordance with policies possibly already in force (e.g. international data policy, Freedom of Information Act, etc.) Data protocol (DMP executive summary) - submission time-frame- conditions of access - retention time-period- conditions of use and publication

7 Metadata Metadata = data about the data Metadata are essential to enable the: -user (a human or computer) to understand the data (physical nature, units, error estimates, scientific context, algorithms, instrument or model specifications, publication references, etc. — N.B. error flags, error bars may be integrated into the data body) -get connected information on research context (experiment, project, platform, contact, etc.) -read the data (format and layout) -find out about the existence of data, where the data are held and how to access them (discovery metadata). This information is required by data portals, browsers and search engines. Our NERC DataGrid project is developing metadata formats following international ISO standards to improve data discovery (see: Help on metadata -

8 The BADC archive is based on a browsable file system. We encourage meaningful file names to allow: - identification of the file content without reading file - automated ingestion into the archive - automated use by handling software Help on file names - File names

9 Under development: -NetCDF/NASA Ames file format converter Data Formats -have a history of successive improvements -allow and encourage inclusion of significant metadata (such as NetCDF’s Climate and Forecast [CF] Metadata Convention) -enable an ease of data exchange with collaborators -enable readability by existing software Online BADC tools: -NASA Ames file format checker -NetCDF file CF compliance checker Help on formats - The BADC encourages the use of NASA Ames (ASCII) and NetCDF (binary) data formats which:

10 1.Requirements: you must register as a BADC user you must have been granted access to the relevant dataset 2.Data files are uploaded to the BADC incoming directory via: the BADC web-based data file uploader ftp to ftp.badc.rl.ac.uk Data Submission 3.Data files are ingested (moved to the archive): with some checking of metadata and data format NOTE: This process is data submission to the final archive. Uploading to the online workspace is not submitting the data to the BADC.

11 Campaign Support Campaign Support includes: The provision of meteorological data and products (such as synoptic charts, rain radar images) in near real time - Sources: Met Office, ECMWF The provision of near real time satellite data (MSG) The provision of forecast trajectories calculated with forecast winds, either through the use of the BADC or Reading trajectory service The provision of dedicated online workspaces or FTP space allowing fast exchange of preliminary data Near real time Met Office Products for the Convective Storms Initiation Project (CSIP) CSIP Collaborative Workspace allows secure upload and sharing of preliminary data for collaborators

12 Getting help/info 1 st step nd step - Points for discussion… How can we serve Data Suppliers better? Which services need improving, and how? What new services are required?