1. RAINEX Data Management UCAR Joint Office for Science Support José Meitín Jim Moore Dick Dirks UCAR Joint Office for Science Support José Meitín Jim Moore Dick Dirks RAINEX Planning Meeting, Miami, FL

2. Early involvement in project planning Involvement with PIs to develop data management strategy Consistent implementation of strategy for lifetime of project and beyond Reliable and efficient archive and distribution system Easy and efficient access to datasets by broader community Early involvement in project planning Involvement with PIs to develop data management strategy Consistent implementation of strategy for lifetime of project and beyond Reliable and efficient archive and distribution system Easy and efficient access to datasets by broader community JOSS Data Management RAINEX Planning Meeting, Miami, FL UCAR Office Of Programs Joint Office for Science Support

3. Data Management Considerations Develop a consistent Data Policy Consider Data Types Formats and Documentation (metadata) Data Collection strategies (operations support, supplementary data) Real-time Requirements ( data calibration, intercomparison, and needs for operational decision making) Data Quality Control and Processing responsibilities Data Archive structure, functions and responsibilities Data Distribution protocols (user interface, metadata inventories, browse products) Collaboration with other programs and data centers (interoperability) Develop a consistent Data Policy Consider Data Types Formats and Documentation (metadata) Data Collection strategies (operations support, supplementary data) Real-time Requirements ( data calibration, intercomparison, and needs for operational decision making) Data Quality Control and Processing responsibilities Data Archive structure, functions and responsibilities Data Distribution protocols (user interface, metadata inventories, browse products) Collaboration with other programs and data centers (interoperability) RAINEX Planning Meeting, Miami, FL

5. Data Questionnaire Data Management Plan Project Web Pages Real-time Operations and Data Ingest Field Data Catalog Quality control and dataset development Interactive Data Archive and Distribution Special media products (e.g. CD-Roms) Data Questionnaire Data Management Plan Project Web Pages Real-time Operations and Data Ingest Field Data Catalog Quality control and dataset development Interactive Data Archive and Distribution Special media products (e.g. CD-Roms) JOSS Data Services RAINEX Planning Meeting, Miami, FL UCAR Office Of Programs Joint Office for Science Support

6. JOSS Data Services Data Questionnaire JOSS Data Services Data Questionnaire UCAR Office Of Programs Joint Office for Science Support RAINEX Planning Meeting, Miami, FL

7. Project Web Pages JOSS Data Services RAINEX Planning Meeting, Miami, FL

8. Real-time Operations Support and Data Ingest Satellite Ground Station Forecast Products Numerical Model Analyses Operational Data Streams Real-time Operations Support and Data Ingest Satellite Ground Station Forecast Products Numerical Model Analyses Operational Data Streams JOSS Data Services RAINEX Planning Meeting, Miami, FL UCAR Office Of Programs Joint Office for Science Support

9. RAINEX Planning Meeting, Miami, FL GTS METAR Observation Locations 6-10 November 2004 GTS METAR Observation Locations 6-10 November 2004

10. RAINEX Planning Meeting, Miami, FL GTS Ship and Buoy Observation Locations 6-10 November 2004 GTS Ship and Buoy Observation Locations 6-10 November 2004

11. RAINEX Planning Meeting, Miami, FL Coastal and Marine Observation Locations *NDBC BuoyoTCOON (TX) *CMAN*TABS (TX) oNWLON+LUMCON (LA) oPORTSoWAVCIS (LA) +COMPS (FL)

12. Land Observation Locations *ASOSoLAIS (LA) *AWOS+TX ET (TX) *AWOS modemoFAWN (FL) oGAEMN (GA)+TNRCC (TX) +GFC (GA)oALMESO (AL) RAINEX Planning Meeting, Miami, FL

13. JOSS Data Services Field Data Catalog ( WWW based ) Daily Mission Reports Operations Summary Status Reports Data Analysis Products Authoring Tools JOSS Data Services Field Data Catalog ( WWW based ) Daily Mission Reports Operations Summary Status Reports Data Analysis Products Authoring Tools UCAR Office Of Programs Joint Office for Science Support RAINEX Planning Meeting, Miami, FL

14. JOSS Data Services Quality control and dataset development Processing of research-quality, high resolution data Processing of standard operational data streams Compatible formats Composite datasets and data integration JOSS Data Services Quality control and dataset development Processing of research-quality, high resolution data Processing of standard operational data streams Compatible formats Composite datasets and data integration UCAR Office Of Programs Joint Office for Science Support RAINEX Planning Meeting, Miami, FL

15. Composite Data Sets at UCAR/JOSS A composite dataset is a collection (over some time period and region) of similar data (e.g. surface meteorological) from a variety of sources, put into a common format, and passed through a uniform quality control. Why does UCAR/JOSS develop composites? - Provides data in a uniform format with QC. - Allows determination of network/site problems. - Useful for model applications. - Prevents duplication of effort. A composite dataset is a collection (over some time period and region) of similar data (e.g. surface meteorological) from a variety of sources, put into a common format, and passed through a uniform quality control. Why does UCAR/JOSS develop composites? - Provides data in a uniform format with QC. - Allows determination of network/site problems. - Useful for model applications. - Prevents duplication of effort. RAINEX Planning Meeting, Miami, FL

16. ASOS AWOS ARM Others (7) OKMESO ABLE SCAN LTER HPCN WTXMESO NCAR (4) MADIS Ameriflux KVII GWMD NMSU CVT HRLY Visual MERGE Gross Limit Checks Horizontal QC Visual Hourly Surface Composite IHOP_2002 Hourly Surface Composite Development Examine Statistics CVT RAINEX Planning Meeting, Miami, FL

17. NCAR / RAF NetCDF NOAA / AOC “standard” format JOSS convert “type 5” to NetCDF JOSS RAINEX Archive Integrated RAINEX NetCDF Aircraft Flight Level Datasets

18. JOSS DATA MANAGEMENT TOOLS Field Catalog In-field tool to ingest and display operational and preliminary research data and project documentation for making real-time decisions and evaluating project progress Features: Daily Mission Reports Operations Summary Facility Status Reports Data Analysis Products Authoring Tools Web-based access Field Catalog In-field tool to ingest and display operational and preliminary research data and project documentation for making real-time decisions and evaluating project progress Features: Daily Mission Reports Operations Summary Facility Status Reports Data Analysis Products Authoring Tools Web-based access Interactive Data Archive/Access System Primary means for all project scientists and researchers to browse and retrieve data from any JOSS-supported projects Features: Long-term field project data archival and distribution Interactive data browsing, subsetting, and format conversion Remote Archive Access Value-added datasets Data documentation Interactive Data Archive/Access System Primary means for all project scientists and researchers to browse and retrieve data from any JOSS-supported projects Features: Long-term field project data archival and distribution Interactive data browsing, subsetting, and format conversion Remote Archive Access Value-added datasets Data documentation RAINEX Planning Meeting, Miami, FL

19. UCAR Office Of Programs Joint Office for Science Support JOSS Data Services http://www.joss.ucar.edu/codiac/ Interactive Data Archive and Distribution Ease of Data Access (web based) Data available via FTP, magnetic tape, CD-Rom Metadata descriptions Data Browse, Subsetting, Format Translation Capabilities Quality Control Distributed Data Archives Customized data ordering and reformat JOSS Data Services http://www.joss.ucar.edu/codiac/ Interactive Data Archive and Distribution Ease of Data Access (web based) Data available via FTP, magnetic tape, CD-Rom Metadata descriptions Data Browse, Subsetting, Format Translation Capabilities Quality Control Distributed Data Archives Customized data ordering and reformat RAINEX Planning Meeting, Miami, FL

20. RAINEX Data Management Issues: Data Policy Data Management Plan Real Time Data Needs (model, satellite) Distributed Archive Responsibilities Special Data Processing (surface, upper-air, airborne radar aircraft flight-level) RAINEX Data Management Issues: Data Policy Data Management Plan Real Time Data Needs (model, satellite) Distributed Archive Responsibilities Special Data Processing (surface, upper-air, airborne radar aircraft flight-level)

21. RAINEX Planning Meeting, Miami, FL meitin@ucar.edu jmoore@ucar.edu dirks2@ucar.edu meitin@ucar.edu jmoore@ucar.edu dirks2@ucar.edu http://www.joss.ucar.edu/ UCAR Office Of Programs Joint Office for Science Support

22. PORTS – Physical Oceanographic Real-Time System NWLON – National Water Level Observation Network TCOON – Texas Coastal Ocean Observation Network (TAMU – Corpus) TABS – Texas Automated Buoy System (TAMU) LUMCON – Louisiana Universities Marine Consortium WAVCIS – Wave Current Surge Information System (LSU) COMPS – Coastal Ocean Monitoring and Prediction System (USF) GAEMN – Georgia Automated Environmental Monitoring Network (U of Georgia) GFC – Georgia Forestry Commission ALMESO – Alabama Mesonet (Auburn U) LAIS – Louisiana Agriclimatic Information System (LSU) TX ET – Texas Evapotranspiration Network (TAMU) FAWN – Florida Automated Weather Network (U of Florida) TNRCC – Texas Natural Resources Conservation Commission Acronyms