Presentation is loading. Please wait.

Presentation is loading. Please wait.

Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

Similar presentations

Presentation on theme: "Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010"— Presentation transcript:

1 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
WMO Regional Association V Fifteenth Session Bali, Indonesia 30 April – 6 May Proper Data Management Responsibilities to Meet the Global Ocean Observing System (GOOS) Requirements Dr. William Burnett Data Management and Communications U.S. NOAA/National Data Buoy Center Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

2 NOAA and Climate/Ocean Observations
There is an increasing demand for global climate change & ocean information, services and products – this includes observations from the Indonesian GOOS (InaGOOS) and the Indian Ocean equatorial array (RAMA), Partnerships enhance ocean observations in the region and also build the regional capacity to apply these observations to understand climate risk management, coastal resilience, ecosystems, MPAs, and other socio-economic benefits, NOAA is interested in advancing a strong, equitable and mutually beneficial collaboration with Region V for capacity building, socio-economic applications and ocean observations in the region, and Training & education is important for countries building the next generation ocean observatories. NOAA is pleased to be able to provide USA training & educational opportunities through Memorandums Of Understanding or Agreements. As we have discussed at this session, increased observations (both land and marine) are important to the countries in Region V. Before we can implement data management techniques, we have to first ensure observations are available in the first place. NOAA is working closely with countries in Region V and other areas to increase the amount of QUALITY observations in the area. 2

3 International Cooperation
Tsunami/RAMA cruise RV Baruna Jaya III Sept 2007 Formal bilateral agreements between NOAA and agencies in: Indonesia--signed in 2007 India--signed in 2008 Japan--signed in 2008 France--planned in 2009 ASCLME (9 East African countries) As was stated in the last slide, NOAA is pleased to be able to provide USA support and cooperation through Memorandums Of Understanding or Agreements. The scope of this collaboration extends from PI to PI interaction to more formal Agency to Agency agreements. 3

4 The Initial Global Ocean Observing System for Climate Status against the GCOS Implementation Plan and JCOMM targets Total in situ networks 61% October 2007 87% 100% 59% 81% The Global Climate Observation System and JCOMM developed a requirements list and target for global ocean observations. In 2010, we have reached 61% of our target. Instrumentation systems like drifters and floats have reached 100% of their target. However, other instrumentation systems like the global mooring network and the tropical moored buoy network are still below target. 100% 48% 34% 73% 62% Milestones Drifters 2005 Argo 2007 4

5 RAMA: Implementation Status
Resource Formula: Partners provide ship time NOAA provides most equipment RAMA is an example of NOAA collaborating with other countries to build an equatorial moored array in the Indian Ocean. For this array, NOAA provides equipment and technical support, while countries provide ship time and the ability to transfer moored buoys to their country. Currently the network is approximately 60% of its target size. Open colored symbols are sites for which partners have been identified and cruises planned. 57% of sites occupied by March 2010 (26 of 46) 5

6 NOAA’s Contributions to Indonesian GOOS (InaGOOS)
NOAA is developing a replacement for the ATLAS system Measurements comparable to ATLAS Use more commercially available components Prototype deployment targeted for October 2010 Deploy near existing ATLAS RAMA moorings for comparison As per IA, 2 systems would eventually serve InaGOOS and RAMA Another example is NOAA’s contribution to the Indonesian Global Ocean Observation System. NOAA is working to upgrade the current ATLAS moored buoy by improving instruments and communications. Climate principles to improve climate observatories will be used in this region. 15-20 Days 6

7 Indonesia MMAF Visit to NDBC July 2008
The last example is the collaboration between NOAA and Indonesia in developing an Indonesian Data Buoy Center, and to use the NDBC Data Assembly Center as one method to provide the observations to the GTS. 7

8 Currents Status More global ocean observations
So – the great news is that there are more ocean observations being collected – and countries see more and more value in placing in-situ instrumentation in marine environments.

9 Current Status Expansion in free data sets
Another good news story is that these observations are being delivered to the World Wide Web in real-time, free and available to all users.

10 Current Status “Climate-gate”
However, what could be considered as good or bad news is that all these observations are being held to a very high standard – and this requires good data management practices to ensure these observations meets these standards.

11 Proper Data Management
More than just placing a meteorological, oceanographic or geophysical instrument in the water or on the land, More that just collecting an observation, and More than just disseminating the data via a data portal Countries must understand that proper data management is more than placing an instrument in the water, collecting the ob and then putting it on the web.

12 WMO Strategic Thrusts Strategic Thrust –
Improving Service Quality and Service Delivery 2. Organization-Wide Expected Results – Enhanced capabilities of Members to deliver and improve access to high quality weather, climate and water and related environmental predictions, information and services in response to user’s needs and to enable their use in decision-making by all relevant societal sectors. WMO understands the role of proper data management – note that “improved service quality and delivery” is a key strategic thrust. A recognized result is improved delivery and access to high quality weather, climate and water observations.

13 Conclusion Any and all atmospheric, oceanographic and geophysical observations will be considered as a “climate” or high-quality observations – and should be treated as such.

14 GCOS Climate Monitoring Principles
Assess impact of new systems or changes to existing systems prior to implementation. Ensure a suitable period of overlap for new and old observing systems. The details and history of local conditions, instruments, operating procedures, data processing algorithms and other factors pertinent to interpreting data (i.e., metadata) should be documented and treated with the same care as the data themselves. Regularly assess quality and homogeneity of data as a part of routine operations. Integrate into national, regional and global observing priorities the needs for environmental and climate-monitoring products and assessments, such as IPCC assessments. Maintain operation of historically-uninterrupted stations and observing systems. Focus on data-poor regions, poorly observed parameters, regions sensitive to change, and key measurements with inadequate temporal resolution as high priorities for additional observations. Specify to network designers, operators and instrument engineers at the outset of system design and implementation the long-term requirements, including appropriate sampling frequencies. Promote the conversion of research observing systems to long-term operations in a carefully-planned manner. Data management systems that facilitate access, use and interpretation of data and products should be included as essential elements of climate monitoring systems. There are two guidelines that are provided to the WMO to help countries reach the Organization-Wide Expected Result. Within the 10 climate monitoring principles, metadata (critical to understanding the instruments involved in collecting the observation) is required, regular assessments of quality and homogeneity of data is routine, and data management is included in any data collection system.

15 Seven Data Management Laws
A quality descriptor will accompany every real-time observation distributed to the ocean community. Subject all observations to some level of automated real-time quality test. Sufficiently describe the quality flags and quality test descriptions in the accompanying metadata. Observers should independently verify or calibrate a sensor before deployment. Observers should describe their method / calibration in the real-time metadata. Observers should quantify the level of calibration accuracy and the associated expected error bounds. Manual checks on the automated procedures, the real-time data collected and the status of the observing system must be provided by the observer on a time-scale appropriate to ensure the integrity of the observing system. The U.S. Quality Assurance of Real-Time Ocean Data (QARTOD) has been key in developing seven data management laws – primarily, each observation should have a quality descriptor, there must be automated real-time quality tests, quality flags should accompany the metadata, each and every sensors should be calibrated before (and after) deployment, methods should be described, the level of calibration should be quantified and finally – manual checks on the procedures should be conducted routinely.

16 Recommendation Region V should begin to implement proper data quality techniques into their newly developed marine observation platforms - now – before the instruments are placed in the water. Region V can build these principles into their newly deployed systems – and lead the world on proper data quality observations and proper data management.

17 National Data Buoy Center
To provide a real-time, end-to-end capability beginning with the collection of marine atmospheric and oceanographic data and ending with its transmission, quality control and distribution. Tsunami Warning Centers NDBC & other NOAA observations IOOS Partners Platforms Weather Forecast Offices/ River Forecast Centers NDBC Data Assembly Center NWS Global Telecommunication System (GTS) Operational Bulletins National Centers for Environmental Prediction NDBC is a world-wide leader in implementing proper climate observation systems and data management. Oil & Gas Platforms Emergency Managers National Environmental Satellite, Data, and Information Service (NCDC, NODC, NGDC) HF Radars Voluntary Observing Ships Public DATA COLLECTION DATA DELIVERY Electrolyte to Satellite to Website

18 Location of NDBC Stennis Space Center Mississippi - NDBC
Quickly, NDBC is located next to the Gulf of Mexico oil spill.

19 NWS/NDBC Ocean Observing System of Systems
Weather Buoys that have in place for > 30 Years

20 NDBC’s Ocean Observing Systems
111 met/ocean buoys 4 ocean/waves buoys 49 C-MAN stations 39 DART stations 55 TAO buoys + 4 current profiler moorings 1000+ Voluntary Observing Ship vessels Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

21 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
Standard 3m & 6m Buoys Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

22 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
Other NDBC Platforms: DART® Deep-ocean Assessment and Reporting of Tsunamis (DART) 39 Stations Since March 2008 ~230 Ship Days - Contract or NOAA Vessel Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

23 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
Other NDBC Platforms: TAO Tropical Atmosphere Ocean (TAO) Climate Buoys 55 equatorial Pacific buoys 4 Ocean Current Profilers ship days a year Typically a NOAA vessel Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

24 Growth of NDBC Observing Systems
1999 to The Era of Explosive Growth 50 100 150 200 250 300 Katrina Tsunami Weather & Hurric. TAO DART C-MAN ~30 buoys/stations in the 1980’s now over 250 If add in IOOS partner stations – well over 700 stations being processed at DAC 101 Observing Systems 2 system Types with similar sensors ~ 12 % in Severe Environments USCG Provided all Ship Days 51 CMAN Stations 50 Weather Buoys 49 CMAN Stations 96 Weather Buoys 15 Supplemental Hurricane Buoys 55 TAO Climate Buoy Systems 39 DART Tsunami Systems + 300% 254 Observing Systems 5 system Types with diverse sensors ~ 25 % in Severe Environments Challenge Obtaining Ship Days 24 24

25 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
Engineering Mooring, power system, station design System validation, analysis, evaluation Technology refresh, prototype testing Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

26 Station/Equipment Prep
Buoy repair Mounts, cable fabrication Sensor repair, refurbishment and calibration System integration and testing Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

27 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
Logistics Field service planning Arranging ship support Providing dockside services Shipping equipment worldwide Inventory management Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

28 Field and At-Sea Maintenance
Multi-disciplined team – technicians, engineers, USCG, NOAA Corps Worldwide dockside and at-sea repairs and testing Buoy deployment, retrieval, exchange Vessels of opportunity – USCG, OMAO, Commercial Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

29 Data Assembly Center Oceanographers, meteorologists, IT specialists, programmers Observations ingest, processing, analysis Data processing and QC algorithm development for new systems Observation dissemination and web display Management of station configuration and metadata

30 NDBC WIS / WIGOS Data Assembly Center
24/7/365 support of Data Quality Control Communications Operations Daily Operations Brief More than 700 stations currently supported NDBC Buoys NDBC C-MAN Stations NOS Stations DART TAO VOS Partner Stations During lunch, I will provide a brief on NDBC’s marine observatories – for this talk I will focus on our world-class data management system.

31 Hurricane/DART Cruise 2010 M/V HOS MYSTIQUE
Team: Artalona, Kendrick, Obenhaus, Stinson, Tretbar Winds: WNW kt Scattered Showers Wind waves: 3-5 ft Swell: Negligible 23 Apr Service 41048 Winds: VAR 5-10 kt Wind waves: 1-2 ft Swell: N sec 24 Apr En route to 41047 Winds: S kt Wind waves: 2-3 ft Swell: NNE sec 25 Apr En route to 41047 Winds: SW kt Wind waves: 3-5 ft Swell: NNE sec 26-27 Apr Service 41047 En route to 41046

32 NOAA/NDBC Data Assembly Center
10 Total Observations flowing through DAC 8 6 Million of Observations 4 2 2003 2004 2005 2006 2007 2008 2009 Active reimbursable partners National Marine Sanctuary Program Kennedy Space Center U.S. Marine Corps Goddard Space Flight Center Army Corps of Engineers U.S. Coast Guard NOS/NWLON and other NOAA Obs Independent IOOS observing partners Gulf of Maine Ocean Observing System Stevens Institute (NJ) International SeaKeepers Society Long Island Ferry Boat University of South Florida Louisiana State University Texas General Land Office University of Connecticut Louisiana Universities Marine Consortium UNC and UNC-W Skidaway Institute of Oceanography Caro-COOPS Scripps Institution of Oceanography Forrest Oil Chesapeake Bay Observing System Shell Oil Monterey Bay Aquarium Research Inst Oregon State University University of Southern Mississippi We have had explosive growth in handling not only NDBC’s observations but also many other “partner” observatories – other NOAA and U.S. agency observatories, as well as IOOS partners and international partners (Columbia and Indonesia). 32

33 Number of IOOS Partner Stations Reporting
NOS Availability Reporting 207 Missing 15 Daily we review the status and quality of NDBC’s stations and all our partner stations. The list at the bottom of the slide shows the name of the partners and the number of stations that the partner owns. Green implies the station if operational and red means non-operational. The National Ocean Services water level stations are very large (210) and are listed separately.

34 U.S. IOOS Partner Support
Providing real-time quality control information to IOOS Partner Platforms... Including QC flags and analyst comments via the website. Daily, we provide our partners with information (located on a website) that provides a status of their observatories and the instrumentation that is located on each observatory.

35 Web Pages Viewed NDBC regularly receives approximately 2.3 million visits per day. This spiked to over 4 million during the Chilean earthquake. 35 35

36 WIGOS Regional Marine Instrument Center
Joint WMO-IOC Technical Commission for Oceanography and Marine Meteorology (JCOMM) Regional Marine Instrument Center (RMIC) Training Workshop on April 13-15, 2010 for WMO Regional Association IV (RA-IV) at NDBC. Representatives from ten countries participated in this first Workshop. Countries represented were: Bahamas, Costa Rica, NL Antilles, Guatemala, Belize, Canada, France, Barbados, Morocco, and China. NDBC is servings as a pilot project for a WIGOS Regional Marine Instrument Center (RMIC) – similar to the WMO Regional Instrument Center – however this center will focus on quality assurance and quality control of marine observations.

37 OceanSITES DAC / Global DAC
NDBC also serves as a Global Data Assembly Center for OceanSITES moorings and observatories in a co-DAC responsibility with Ifremer in France.

38 OceanSITES Data Management
Maintains specific OceanSITES platforms, Determines what observations are released to GTS, Assures that the platform is available and provides reliable information, Provides the DAC with the observations in any format the DAC is willing to take, and the metadata necessary to serve as an OceanSITES platform, and QC post-recovery data according to OceanSITES agreed procedures. Observations in any format – may or may not be quality controlled Cronin Send Pattabhi McPhaden PI FTP, Flash Drive, CD Sets up the OceanSITES server according to the approved specifications, Guarantees data availability from the PI, Translates the data to the OceanSITES format, Quality Controls real-time data according to the minimum OceanSITES agreed procedures, Provides the observations via the GTS (if requested by the PI), Provides the data on a FTP server for access by the GDACs Formats observations and provides QC EuroSITES WHOI MBARI DAC FTP This slide shows the current architecture with some (not all) PIs, some (not all) DACs and the GDACs. Responsibilities of the PIs, DACs and GDACs are provided in yellow on the right-hand side. At this point, I would describe the flow of observations using the verbiage on the right side. Provides access to data, checks formats IFremer NDBC Provides centralized access to the DAC data Ensures no data are excluded at the GDAC level, and full high-frequency data sets are available, Keeps only the best version of the data. Additional products like interpolated data are separate optional sets, Check all files daily using the “File Checker” software, Maintains the OceanSITES catalogue, and Synchronizes the catalogues with the second GDAC periodically ( at least daily). GDAC FTP Technical Coordinator User Requests 38

39 “WIS-Data Discovery, Access and Retrieval (DAR)” Overview
SOS (Sensor Observation Service) Table of Contents Metadata Data Values Data Provider Quality Control Observations NDBC is implementing the WIS Data Discovery, Access and Retrieval (DAR) technologies through the use of Sensor Observation Service. Attaching an enormous amount of metadata to each observation will require new technologies to handle the increase in message size. SOS is one way to provide this ability – however, all countries should note that message size and bandwidth will need to be increased to meet data management requirements.

40 International Tsunameter Data Assembly Center
The worldwide tsunami observation network also requires a real-time, data assembly center to provide continual monitoring and quality control of Deep-ocean Assessment and Reporting of Tsunamis (DART®) water pressure/height observations. The Data Assembly Center monitors the various real-time transmission of DART® messages depending on the operating mode of the bottom pressure recorder. Transmission of real-time water level heights occurs when the tsunami detection algorithm is triggered by a seismic event, when interrogated by the NOAA Tsunami Warning Centers (TWCs) or NDBC, or at pre-scheduled intervals. NDBC also operates a number of other data assembly centers – one is the tsunameter DAC. This slide says “International” because NDBC handles the data for countries like Indonesia, Thailand, and Chile.

41 Tropical Atmosphere Ocean Data Assembly Center
NDBC operates the TAO data assembly center for equatorial Pacific array. Processing of Automated Distribution Service messages from Service Argos. It uses both the TAO calibration database and calibration files to convert raw data to engineering units and also calculates buoy positions. An automated real-time QC is performed for gross error checking and then the TAO database is updated with the corrected data. The TAO Real-time Data Monitoring Subsystem supports daily, weekly, and monthly QA/QC activities by providing on-demand data checking functionality to the DAC. In addition to the automated gross error checking, the real-time data monitoring subsystem provides on-demand reports for once-daily, thorough examination of all current buoy data and detailed review of the real-time data.

42 U.S. Voluntary Observing Ship (VOS) Port Meteorological Officers
The United States Voluntary Observing Ship Project Mission The mission of the Voluntary Observing Ship (VOS) project is two-fold: (1) to collect and disseminate critical real-time maritime weather observations through the recruitment and support of ships to fulfill National needs and International agreements supporting commerce, forecasts and warning programs, and the Safety Of Life At Sea (SOLAS) worldwide, and (2) to define the global climate and help measure extreme weather events, climate variability, and long-term climate changes. VOS operates at no cost to the vessel, with communication charges, observing equipment and reporting supplies furnished by the National Weather Service. Port Meteorological Officers Port Meteorological Officers (PMOs) support observing programs aboard Voluntary Observing Ships. They are responsible for recruitment of new vessels as observers, and also for ensuring the quality of observations from vessels actively participating in the program. NDBC is involved in the WMO VOS program.

43 High Frequency Radar
NDBC became the National HF Radar Node on 28 February NDBC will receive HF Radar radials from all HF radar sites and generate vectors. NDBC is also the national node for High Frequency radar – which has been very important to track the oil spill in the Gulf of Mexico.

44 Oil and Gas Partners Apply quality control to real-time ADCP data from deep- water oil platforms and rigs. Speaking of the oil spill, NDBC handles all the Acoustic Doppler Current Profilers (ADCPs) observations that are collected by oil and gas platforms in the Gulf of Mexico. Oil companies are required to place these instruments on their platforms – and required to provide the data for free, and in real-time to NDBC who disseminates the information to the GTS.

45 QC Matrix NDBC is developing a web-based system that provides all our partners with a better understanding of our automated quality control checks for all the sensors we handle.

46 NDBC High Level Data Flow
NWSTG shares to GTS NDBC shares to Web Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

47 NDBC Data Center A world-class data center is required to handle all these data assembly centers. 10 Gigabit Ethernet technology that delivers multi-gigabit bandwidth to all resources while maintaining a 100% uptime status. Designed around high-end enterprise class systems.  Minimum standards include Dual-Core 1.8Ghz systems, 4GB RAM, redundant Gigabit network interface cards, and redundant power. 

48 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
Primary Data Sources Data Feed Type Data Received Geostationary Operational Environmental Satellites (GOES) Weather Buoy, C-MAN, Forecast and VOS IRIDIUM Satellite System/Router Based Unrestricted Digital Interworking Connectivity Solution (RUDICS) DART Data IRIDIUM Satellite System/Department of Defense (DoD)/Short Burst Data service Hurricane buoys, TAO refresh, AIS data, limited weather buoy Service ARGOS, Inc. (ARGOS) satellite system Buoy positioning via LUT, TAO legacy data Integrated Ocean Observing System Participants (IOOS) Partner data via FTP and XML HF Radar High Frequency Radar Data GTS/GODAE Non-NDBC marine observation data in support of the web and OSMC GOES – Provides NDBC yellow fleet buoy and C-Man Station observations, forecast data, and Voluntary Observing Ship data. IOOS – IOOS partners’ provide data via FTP to NDBC’s FTP servers including High Frequency Radar – Received from xxxxx via FTP IRIDIUM/RUDICS –provides DART II data via their commercial gateway in Tempe, AZ IRIDIUM/DOD – provides TAO refresh buoy data, AIS buoy data, limited yellow buoy, and hurricane buoy data via the DoD gateway Short Burst Data Service in Hawaii ARGOS – provides NDBC buoy positioning data via LUT and TAO legacy buoy data via and/or Telnet HF Radar – Received from xxxxx via FTP Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

49 Basic Real-time Processing and Web System
Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

50 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
Tsunameter Data Flow Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

51 TAO Data System Architecture
Iridium SBD Argos ADS NWSTG GTS Real-time Data Ingest and Dissemination Database and File Management Public Web Presentation Delayed Mode Analysis Console Interfaces DAC Analysts Public Users Automated QC and Alerts Scientists The refreshed TAO Data System consists of 6 subsystems: Real-time Data Ingest and Dissemination subsystem – It decodes and stores the real-time Short Burst Data from refreshed buoys via Iridium and Argos messages from legacy buoys via POES. It also manages the GTS dissemination as the results of the data processing. Automated QC and Alerts subsystem – It generates alerting messages to DAC via Data Management Console. Alerts include buoy adrift, transmission outage, etc. Database and File Management subsystem – It stores all data and metadata associated with the TAO moorings. It supports Data Management Console, Delayed Mode Analysis Tool, and the Public Web Presentation. Data Management Console subsystem – It is the primary user interface for DAC Data Quality Analyst to monitor TAO Array data real-time data stream. It includes Daily Status Report, Buoy Position Report, Data Availability Report. It also supports Deployment/Recovery metadata entries. Delayed Mode Analysis subsystem – It is a tool set that processes the recovered data from moorings. The results of delayed mode data processing are uploaded into the database. Public Web Presentation subsystem – It is the primary user interface for the public to view and download TAO datasets. Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

52 H F Radar NDBC Stennis Ocean Surface Currents- speed and direction
NDBC processes FTP transmissions and displays vectors on NDBC Web site via the WWW HF Radar Stations located along US Coast NDBC Stennis

53 Partner Data Processing
Regional Observatories Sensor/ Observation Observers Web-page ftp* NDBC Web Page NDBC QA/QC Public NDBC Dial-A-Buoy Wx. Channel Local Media NWS GATEWAY GTS WFOs NODC NCDC et. al. * Via XML formatted files or the NDBC Meteorological and Oceanographic Data Exchange Module (MODEM) Kit - request from

54 Data Quality at NDBC 1970s - 2005 Started with NDBC in 1970s
Focus on “Top Five” Wind Direction, Wind Speed, Atmospheric Surface Pressure, Air Temperature and Waves Algorithms perform check at NWS Gateway Quality Assurance Group at NDBC provided daily check of marine observations – usually 8 hours to two days after dissemination Approx. 110 platforms

55 Data Quality Control: The Last Line of Defense, after…
Sensor Evaluations Individual Sensor Calibrations Payload Software Testing Burn-In Data Evaluation at deployment Partner Data? Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

56 Quality Control of Observations
No QC Done Onboard the Buoy or C-MAN Automated QC Done in Real-Time at NWSTG Hard-flags: Stop the Release and Archive of Data unless Analyst Overrides. Override in advance – Storm Limits, or Remove before archive Soft-flags: Climatology-based, Advisory Handbook of Automated Quality Control Checks and Procedures of the National Data Buoy Center Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

57 Moored Buoy Observations
Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

58 Quality Control Process Flow
Real-Time: System Parameters (e.g., power) Message Integrity Automated Hard-Flag Automated Soft-Flag Post-Release, Pre-Archive: Data Analyst Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

59 Hard-Flags by hierarchy - highest to lowest:
T Transmission parity error M Missing sensor data W Wave message is short, checksum or parity errors. E Spectral Density are exceeded or are in error D Delete measurement (“permanent failure”) S Invalid statistical parameter (e.g., mean > max) V Failed time continuity. L Failed range (climatological) limits R Related measurement has failed a hard QC check (e.g., WVHGT fails → Periods failed). Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

60 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
Soft-flags for Waves a: Measurement is above monthly, regional limit. b: Measurement is below monthly, regional limit. c: Measurement has been adjusted, or corrected. f: Measurement failed hourly time continuity. m: High frequency spikes detected in the wave spectrum. p: Failed wave height to wave period comparison test. q: Swell direction is from an improbable direction. w: Failed wind direction verses wave direction check. x: Wind wave energy is too high for prevailing wind speed. y: Wind wave energy is too low for prevailing wind speed. Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

61 It all happens at the NWSTG
Most other organizations: NWSTG is a big router For up to 5 minutes

62 Why we halt the data at NWSTG
Needed to process waves Data can be corrected: Can recover incorrect payload parameters Sensor offset or position drift Quality Control Add computed fields Assign WMO code forms, routing identifiers Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

63 Configuration Control File for Each Station:
GOES ID Codes and Routing Identifiers Test or Operational Mode Sensors Failed Sensor Hierarchy Sensor Scaling Station and Sensor Elevations Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

64 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
Derived Measurements Pressure Tendency & Trace Characteristic Extrapolated 10- and 20-m wind speeds Swell and wind wave estimates Wind Chill Ice Accretion Heat Index Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

65 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
Output Formats FM-13 for moored buoys C-MAN code for coastal stations: National code, but well-suited for coastal obs. Contains water level & waves Based on FM-12 land synoptic code FM-64 TESAC (temp., salinity, current profiles) FM-65 WAVEOB (spectral wave data) Will be BUFR Ready by 2012 WMO mandate! Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010 65

66 Real-Time Quality Control
Purpose: Remove gross errors Data rejected: Virtually certain to be degraded Typical causes: Transmission errors, power degradations, broken cables Limitations: Won’t detect minor errors, biases Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

67 Monthly Climatology Check

68 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
Limit Checks Classic way of catching a sensor that “pegs”. However, extreme, but valid, data has been withheld. Climatology based Can be overridden before expected storm Originally set to 3x Standard Deviation but found that could be too restrictive Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010 68

69 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
Time Continuity Check Dependent on time since last valid observation. Max. allowed value = 0.58**SQRT(TimeDiff) Works well for normally distributed measurements Std. dev. Chosen 50% higher climatic standard dev. Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

70 Time Continuity Check

71 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
Time Continuity Check Maximum allowable values in one hour: Sea level pressure 12.2 hPa Air Temperature deg. C Water Temperature 5.0 deg. C Wind Speed m/s Wave Height 3.5 m Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

72 If data fails time continuity, it may still be released if:
Pressure: Both pressures < 1000 hPa Wind Speed: Both pressures < 995 hPa or it agrees within 2 m/s of duplicate Air Temperature: Either wind speed > 7 m/s or wind direction change > 40 deg. Wave Height: Current wind speed > 15 m/s Manually disabled in front of a hurricane Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

73 Internal Consistency Checks
If battery voltage < 10.5 V, pressure not released. Significant wave height and dominant period set to zero if significant wave height < 0.25m. Without this, large periods can result from nearly calm seas. Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

74 Internal Consistency Checks
If dew pt. > air temp, set dew pt. = air temp. (RH can read slightly more than 100%) If ratio of gust to wind speed > 4, don’t send wind speed or gust. Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

75 Post Release Man-Machine Mix
Pre-done Graphics produced: Flagged data first Then all data Tools: Time series plots Spectral Wave Curves Wind Wave scatterplots Surface weather plots Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010

76 Visualization Tool Suite Top Level

77 Visualization Tool Suite Area Level

78 Pre-done Graphics Ocean Pier NC Prob ale corrupted message 78

79 Pre-done Graphics

80 Pre-done Graphics Partner data initial report
Showing data with bad data (sensor or process) issue

81 Time Series Plots: Wind Speed Problem
WDEL – repoting in mps vice KTS prior to intl conversion

82 Time Series Plots: Comparison with GFS Initial Analysis

83 Position Plots

84 Data Monitoring What does the future hold?
Implementing new Unidata WX analysis software with improved vis. tools Look at only suspect or bad data - real time QC associated with a database - more sophisticated logic in algorithms, extended trend analysis - model fields used in real met and wave QC Flags assigned based on quality score Distribute and archive data with flags

85 Terima Kasih. Bill. Burnett@noaa
Terima Kasih ! National Data Buoy Center Stennis Space Center, MS 39529 85

Download ppt "Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010"

Similar presentations

Ads by Google