Presentation on theme: "Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010"— Presentation transcript:
1 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010 WMO Regional Association V Fifteenth Session Bali, Indonesia 30 April – 6 May Proper Data Management Responsibilities to Meet the Global Ocean Observing System (GOOS) RequirementsDr. William BurnettData Management and CommunicationsU.S. NOAA/National Data Buoy CenterRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
2 NOAA and Climate/Ocean Observations There is an increasing demand for global climate change & ocean information, services and products – this includes observations from the Indonesian GOOS (InaGOOS) and the Indian Ocean equatorial array (RAMA),Partnerships enhance ocean observations in the region and also build the regional capacity to apply these observations to understand climate risk management, coastal resilience, ecosystems, MPAs, and other socio-economic benefits,NOAA is interested in advancing a strong, equitable and mutually beneficial collaboration with Region V for capacity building, socio-economic applications and ocean observations in the region, andTraining & education is important for countries building the next generation ocean observatories. NOAA is pleased to be able to provide USA training & educational opportunities through Memorandums Of Understanding or Agreements.As we have discussed at this session, increased observations (both land and marine) are important to the countries in Region V. Before we can implement data management techniques, we have to first ensure observations are available in the first place. NOAA is working closely with countries in Region V and other areas to increase the amount of QUALITY observations in the area.2
3 International Cooperation Tsunami/RAMA cruiseRV Baruna Jaya III Sept 2007Formal bilateral agreements between NOAA and agencies in:Indonesia--signed in 2007India--signed in 2008Japan--signed in 2008France--planned in 2009ASCLME (9 East African countries)As was stated in the last slide, NOAA is pleased to be able to provide USA support and cooperation through Memorandums Of Understanding or Agreements. The scope of this collaboration extends from PI to PI interaction to more formal Agency to Agency agreements.3
4 The Initial Global Ocean Observing System for Climate Status against the GCOS Implementation Plan and JCOMM targetsTotal in situ networks61%October 200787%100%59%81%The Global Climate Observation System and JCOMM developed a requirements list and target for global ocean observations. In 2010, we have reached 61% of our target. Instrumentation systems like drifters and floats have reached 100% of their target. However, other instrumentation systems like the global mooring network and the tropical moored buoy network are still below target.100%48%34%73%62%MilestonesDrifters 2005Argo 20074
5 RAMA: Implementation Status Resource Formula:Partners provide ship timeNOAA provides most equipmentRAMA is an example of NOAA collaborating with other countries to build an equatorial moored array in the Indian Ocean. For this array, NOAA provides equipment and technical support, while countries provide ship time and the ability to transfer moored buoys to their country. Currently the network is approximately 60% of its target size. Open colored symbols are sites for which partners have been identified and cruises planned.57% of sites occupied by March 2010 (26 of 46)5
6 NOAA’s Contributions to Indonesian GOOS (InaGOOS) NOAA is developing a replacement for the ATLAS systemMeasurements comparable to ATLASUse more commercially available componentsPrototype deployment targeted for October 2010Deploy near existing ATLAS RAMA moorings for comparisonAs per IA, 2 systems would eventually serve InaGOOS and RAMAAnother example is NOAA’s contribution to the Indonesian Global Ocean Observation System. NOAA is working to upgrade the current ATLAS moored buoy by improving instruments and communications. Climate principles to improve climate observatories will be used in this region.15-20 Days6
7 Indonesia MMAF Visit to NDBC July 2008 The last example is the collaboration between NOAA and Indonesia in developing an Indonesian Data Buoy Center, and to use the NDBC Data Assembly Center as one method to provide the observations to the GTS.7
8 Currents Status More global ocean observations So – the great news is that there are more ocean observations being collected – and countries see more and more value in placing in-situ instrumentation in marine environments.
9 Current Status Expansion in free data sets Another good news story is that these observations are being delivered to the World Wide Web in real-time, free and available to all users.
10 Current Status “Climate-gate” However, what could be considered as good or bad news is that all these observations are being held to a very high standard – and this requires good data management practices to ensure these observations meets these standards.
11 Proper Data Management More than just placing a meteorological, oceanographic or geophysical instrument in the water or on the land,More that just collecting an observation, andMore than just disseminating the data via a data portalCountries must understand that proper data management is more than placing an instrument in the water, collecting the ob and then putting it on the web.
12 WMO Strategic Thrusts Strategic Thrust – Improving Service Quality and Service Delivery2. Organization-Wide Expected Results –Enhanced capabilities of Members to deliver and improve access to high quality weather, climate and water and related environmental predictions, information and services in response to user’s needs and to enable their use in decision-making by all relevant societal sectors.WMO understands the role of proper data management – note that “improved service quality and delivery” is a key strategic thrust. A recognized result is improved delivery and access to high quality weather, climate and water observations.
13 ConclusionAny and all atmospheric, oceanographic and geophysical observations will be considered as a “climate” or high-quality observations – and should be treated as such.
14 GCOS Climate Monitoring Principles Assess impact of new systems or changes to existing systems prior to implementation.Ensure a suitable period of overlap for new and old observing systems.The details and history of local conditions, instruments, operating procedures, data processing algorithms and other factors pertinent to interpreting data (i.e., metadata) should be documented and treated with the same care as the data themselves.Regularly assess quality and homogeneity of data as a part of routine operations.Integrate into national, regional and global observing priorities the needs for environmental and climate-monitoring products and assessments, such as IPCC assessments.Maintain operation of historically-uninterrupted stations and observing systems.Focus on data-poor regions, poorly observed parameters, regions sensitive to change, and key measurements with inadequate temporal resolution as high priorities for additional observations.Specify to network designers, operators and instrument engineers at the outset of system design and implementation the long-term requirements, including appropriate sampling frequencies.Promote the conversion of research observing systems to long-term operations in a carefully-planned manner.Data management systems that facilitate access, use and interpretation of data and products should be included as essential elements of climate monitoring systems.There are two guidelines that are provided to the WMO to help countries reach the Organization-Wide Expected Result. Within the 10 climate monitoring principles, metadata (critical to understanding the instruments involved in collecting the observation) is required, regular assessments of quality and homogeneity of data is routine, and data management is included in any data collection system.
15 Seven Data Management Laws A quality descriptor will accompany every real-time observation distributed to the ocean community.Subject all observations to some level of automated real-time quality test.Sufficiently describe the quality flags and quality test descriptions in the accompanying metadata.Observers should independently verify or calibrate a sensor before deployment.Observers should describe their method / calibration in the real-time metadata.Observers should quantify the level of calibration accuracy and the associated expected error bounds.Manual checks on the automated procedures, the real-time data collected and the status of the observing system must be provided by the observer on a time-scale appropriate to ensure the integrity of the observing system.The U.S. Quality Assurance of Real-Time Ocean Data (QARTOD) has been key in developing seven data management laws – primarily, each observation should have a quality descriptor, there must be automated real-time quality tests, quality flags should accompany the metadata, each and every sensors should be calibrated before (and after) deployment, methods should be described, the level of calibration should be quantified and finally – manual checks on the procedures should be conducted routinely.
16 RecommendationRegion V should begin to implement proper data quality techniques into their newly developed marine observation platforms - now – before the instruments are placed in the water.Region V can build these principles into their newly deployed systems – and lead the world on proper data quality observations and proper data management.
17 National Data Buoy Center To provide a real-time, end-to-end capability beginning with the collection of marine atmospheric and oceanographic data and ending with its transmission, quality control and distribution.Tsunami Warning CentersNDBC& other NOAAobservationsIOOS PartnersPlatformsWeather Forecast Offices/ River Forecast CentersNDBCData AssemblyCenterNWS Global Telecommunication System (GTS) Operational BulletinsNational Centers for Environmental PredictionNDBC is a world-wide leader in implementing proper climate observation systems and data management.Oil & Gas PlatformsEmergencyManagersNational Environmental Satellite, Data, and Information Service(NCDC, NODC, NGDC)HF RadarsVoluntary Observing ShipsPublicDATA COLLECTIONDATA DELIVERYElectrolyte to Satellite to Website
18 Location of NDBC Stennis Space Center Mississippi - NDBC Quickly, NDBC is located next to the Gulf of Mexico oil spill.
19 NWS/NDBC Ocean Observing System of Systems Weather Buoys that have in place for > 30 Years
20 NDBC’s Ocean Observing Systems 111 met/ocean buoys4 ocean/waves buoys49 C-MAN stations39 DART stations55 TAO buoys + 4 current profiler moorings1000+ Voluntary Observing Ship vesselsRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
21 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010 Standard 3m & 6m BuoysRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
22 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010 Other NDBC Platforms: DART®Deep-ocean Assessment and Reporting of Tsunamis (DART)39 Stations Since March 2008~230 Ship Days - Contract or NOAA VesselRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
23 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010 Other NDBC Platforms: TAOTropical Atmosphere Ocean (TAO) Climate Buoys55 equatorial Pacific buoys4 Ocean Current Profilersship days a yearTypically a NOAA vesselRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
24 Growth of NDBC Observing Systems 1999 to The Era of Explosive Growth50100150200250300KatrinaTsunamiWeather& Hurric.TAODARTC-MAN~30 buoys/stations in the 1980’s now over 250If add in IOOS partner stations – well over 700 stations being processed at DAC101 Observing Systems2 system Types with similar sensors~ 12 % in Severe EnvironmentsUSCG Provided all Ship Days51 CMAN Stations50 Weather Buoys49 CMAN Stations96 Weather Buoys15 Supplemental Hurricane Buoys55 TAO Climate Buoy Systems39 DART Tsunami Systems+ 300%254 Observing Systems5 system Types with diverse sensors~ 25 % in Severe EnvironmentsChallenge Obtaining Ship Days2424
25 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010 EngineeringMooring, power system, station designSystem validation, analysis, evaluationTechnology refresh, prototype testingRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
26 Station/Equipment Prep Buoy repairMounts, cable fabricationSensor repair, refurbishment and calibrationSystem integration and testingRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
27 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010 LogisticsField service planningArranging ship supportProviding dockside servicesShipping equipment worldwideInventory managementRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
28 Field and At-Sea Maintenance Multi-disciplined team – technicians, engineers, USCG, NOAA CorpsWorldwide dockside and at-sea repairs and testingBuoy deployment, retrieval, exchangeVessels of opportunity – USCG, OMAO, CommercialRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
29 Data Assembly CenterOceanographers, meteorologists, IT specialists, programmersObservations ingest, processing, analysisData processing and QC algorithm development for new systemsObservation dissemination and web displayManagement of station configuration and metadata
30 NDBC WIS / WIGOS Data Assembly Center 24/7/365 support ofData Quality ControlCommunicationsOperationsDaily Operations BriefMore than 700 stations currently supportedNDBC BuoysNDBC C-MAN StationsNOS StationsDARTTAOVOSPartner StationsDuring lunch, I will provide a brief on NDBC’s marine observatories – for this talk I will focus on our world-class data management system.
31 Hurricane/DART Cruise 2010 M/V HOS MYSTIQUE Team: Artalona, Kendrick, Obenhaus, Stinson, TretbarWinds: WNW kt Scattered ShowersWind waves: 3-5 ft Swell: Negligible23 AprService 41048Winds: VAR 5-10 ktWind waves: 1-2 ft Swell: N sec24 AprEn route to 41047Winds: S ktWind waves: 2-3 ft Swell: NNE sec25 AprEn route to 41047Winds: SW ktWind waves: 3-5 ft Swell: NNE sec26-27 AprService 41047En route to 41046
32 NOAA/NDBC Data Assembly Center 10Total Observations flowing through DAC86Million of Observations422003200420052006200720082009Active reimbursable partnersNational Marine Sanctuary Program Kennedy Space CenterU.S. Marine Corps Goddard Space Flight CenterArmy Corps of Engineers U.S. Coast GuardNOS/NWLON and other NOAA ObsIndependent IOOS observing partnersGulf of Maine Ocean Observing System Stevens Institute (NJ)International SeaKeepers Society Long Island Ferry BoatUniversity of South Florida Louisiana State UniversityTexas General Land Office University of ConnecticutLouisiana Universities Marine Consortium UNC and UNC-WSkidaway Institute of Oceanography Caro-COOPSScripps Institution of Oceanography Forrest OilChesapeake Bay Observing System Shell OilMonterey Bay Aquarium Research Inst Oregon State UniversityUniversity of Southern MississippiWe have had explosive growth in handling not only NDBC’s observations but also many other “partner” observatories – other NOAA and U.S. agency observatories, as well as IOOS partners and international partners (Columbia and Indonesia).32
33 Number of IOOS Partner Stations Reporting NOS AvailabilityReporting207Missing15Daily we review the status and quality of NDBC’s stations and all our partner stations. The list at the bottom of the slide shows the name of the partners and the number of stations that the partner owns. Green implies the station if operational and red means non-operational. The National Ocean Services water level stations are very large (210) and are listed separately.
34 U.S. IOOS Partner Support Providing real-time quality control information to IOOS Partner Platforms... Including QC flags and analyst comments via the website.Daily, we provide our partners with information (located on a website) that provides a status of their observatories and the instrumentation that is located on each observatory.
35 Web Pages ViewedNDBC regularly receives approximately 2.3 million visits per day. This spiked to over 4 million during the Chilean earthquake.3535
36 WIGOS Regional Marine Instrument Center Joint WMO-IOC Technical Commission for Oceanography and Marine Meteorology (JCOMM) Regional Marine Instrument Center (RMIC) Training Workshop on April 13-15, 2010 for WMO Regional Association IV (RA-IV) at NDBC. Representatives from ten countries participated in this first Workshop. Countries represented were: Bahamas, Costa Rica, NL Antilles, Guatemala, Belize, Canada, France, Barbados, Morocco, and China.NDBC is servings as a pilot project for a WIGOS Regional Marine Instrument Center (RMIC) – similar to the WMO Regional Instrument Center – however this center will focus on quality assurance and quality control of marine observations.
37 OceanSITES DAC / Global DAC NDBC also serves as a Global Data Assembly Center for OceanSITES moorings and observatories in a co-DAC responsibility with Ifremer in France.
38 OceanSITES Data Management Maintains specific OceanSITES platforms,Determines what observations are released to GTS,Assures that the platform is available and provides reliable information,Provides the DAC with the observations in any format the DAC is willing to take, and the metadata necessary to serve as an OceanSITES platform, andQC post-recovery data according to OceanSITES agreed procedures.Observations in any format – may or may not be quality controlledCroninSendPattabhiMcPhadenPIFTP, Flash Drive, CDSets up the OceanSITES server according to the approved specifications,Guarantees data availability from the PI,Translates the data to the OceanSITES format,Quality Controls real-time data according to the minimum OceanSITES agreed procedures,Provides the observations via the GTS (if requested by the PI),Provides the data on a FTP server for access by the GDACsFormats observations and provides QCEuroSITESWHOIMBARIDACFTPThis slide shows the current architecture with some (not all) PIs, some (not all) DACs and the GDACs. Responsibilities of the PIs, DACs and GDACs are provided in yellow on the right-hand side. At this point, I would describe the flow of observations using the verbiage on the right side.Provides access to data, checks formatsIFremerNDBCProvides centralized access to the DAC dataEnsures no data are excluded at the GDAC level, and full high-frequency data sets are available,Keeps only the best version of the data. Additional products like interpolated data are separate optional sets,Check all files daily using the “File Checker” software,Maintains the OceanSITES catalogue, andSynchronizes the catalogues with the second GDAC periodically ( at least daily).GDACFTPTechnical CoordinatorUserRequests38
39 “WIS-Data Discovery, Access and Retrieval (DAR)” Overview SOS(SensorObservationService)Table ofContentsMetadataData ValuesDataProviderQualityControlObservationsNDBC is implementing the WIS Data Discovery, Access and Retrieval (DAR) technologies through the use of Sensor Observation Service. Attaching an enormous amount of metadata to each observation will require new technologies to handle the increase in message size. SOS is one way to provide this ability – however, all countries should note that message size and bandwidth will need to be increased to meet data management requirements.
40 International Tsunameter Data Assembly Center The worldwide tsunami observation network also requires a real-time, data assembly center to provide continual monitoring and quality control of Deep-ocean Assessment and Reporting of Tsunamis (DART®) water pressure/height observations. The Data Assembly Center monitors the various real-time transmission of DART® messages depending on the operating mode of the bottom pressure recorder.Transmission of real-time water level heights occurs when the tsunami detection algorithm is triggered by a seismic event, when interrogated by the NOAA Tsunami Warning Centers (TWCs) or NDBC, or at pre-scheduled intervals.NDBC also operates a number of other data assembly centers – one is the tsunameter DAC. This slide says “International” because NDBC handles the data for countries like Indonesia, Thailand, and Chile.
41 Tropical Atmosphere Ocean Data Assembly Center NDBC operates the TAO data assembly center for equatorial Pacific array.Processing of Automated Distribution Service messages from Service Argos. It uses both the TAO calibration database and calibration files to convert raw data to engineering units and also calculates buoy positions. An automated real-time QC is performed for gross error checking and then the TAO database is updated with the corrected data.The TAO Real-time Data Monitoring Subsystem supports daily, weekly, and monthly QA/QC activities by providing on-demand data checking functionality to the DAC. In addition to the automated gross error checking, the real-time data monitoring subsystem provides on-demand reports for once-daily, thorough examination of all current buoy data and detailed review of the real-time data.
42 U.S. Voluntary Observing Ship (VOS) Port Meteorological Officers The United States Voluntary Observing Ship Project MissionThe mission of the Voluntary Observing Ship (VOS) project is two-fold: (1) to collect and disseminate critical real-time maritime weather observations through the recruitment and support of ships to fulfill National needs and International agreements supporting commerce, forecasts and warning programs, and the Safety Of Life At Sea (SOLAS) worldwide, and (2) to define the global climate and help measure extreme weather events, climate variability, and long-term climate changes. VOS operates at no cost to the vessel, with communication charges, observing equipment and reporting supplies furnished by the National Weather Service.Port Meteorological OfficersPort Meteorological Officers (PMOs) support observing programs aboard Voluntary Observing Ships. They are responsible for recruitment of new vessels as observers, and also for ensuring the quality of observations from vessels actively participating in the program.NDBC is involved in the WMO VOS program.
43 High Frequency Radar http://hfradar.ndbc.noaa.gov NDBC became the National HF Radar Node on 28 February NDBC will receive HF Radar radials from all HF radar sites and generate vectors.NDBC is also the national node for High Frequency radar – which has been very important to track the oil spill in the Gulf of Mexico.
44 Oil and Gas PartnersApply quality control to real-time ADCP data from deep-water oil platforms and rigs.Speaking of the oil spill, NDBC handles all the Acoustic Doppler Current Profilers (ADCPs) observations that are collected by oil and gas platforms in the Gulf of Mexico. Oil companies are required to place these instruments on their platforms – and required to provide the data for free, and in real-time to NDBC who disseminates the information to the GTS.
45 QC MatrixNDBC is developing a web-based system that provides all our partners with a better understanding of our automated quality control checks for all the sensors we handle.
46 NDBC High Level Data Flow NWSTG shares to GTSNDBC shares to WebRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
47 NDBC Data CenterA world-class data center is required to handle all these data assembly centers.10 Gigabit Ethernet technology that delivers multi-gigabit bandwidth to all resources while maintaining a 100% uptime status.Designed around high-end enterprise class systems. Minimum standards include Dual-Core 1.8Ghz systems, 4GB RAM, redundant Gigabit network interface cards, and redundant power.
48 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010 Primary Data SourcesData FeedType Data ReceivedGeostationary Operational Environmental Satellites (GOES)Weather Buoy, C-MAN, Forecast and VOSIRIDIUM Satellite System/Router Based Unrestricted Digital Interworking Connectivity Solution (RUDICS)DART DataIRIDIUM Satellite System/Department of Defense (DoD)/Short Burst Data serviceHurricane buoys, TAO refresh, AIS data, limited weather buoyService ARGOS, Inc. (ARGOS) satellite systemBuoy positioning via LUT, TAO legacy dataIntegrated Ocean Observing System Participants (IOOS)Partner data via FTP and XMLHF RadarHigh Frequency Radar DataGTS/GODAENon-NDBC marine observation data in support of the web and OSMCGOES – Provides NDBC yellow fleet buoy and C-Man Station observations, forecast data, and Voluntary Observing Ship data.IOOS – IOOS partners’ provide data via FTP to NDBC’s FTP servers including High Frequency Radar – Received from xxxxx via FTPIRIDIUM/RUDICS –provides DART II data via their commercial gateway in Tempe, AZIRIDIUM/DOD – provides TAO refresh buoy data, AIS buoy data, limited yellow buoy, and hurricane buoy data via the DoD gateway Short Burst Data Service in HawaiiARGOS – provides NDBC buoy positioning data via LUT and TAO legacy buoy data via and/or TelnetHF Radar – Received from xxxxx via FTPRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
49 Basic Real-time Processing and Web System Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
50 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010 Tsunameter Data FlowRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
51 TAO Data System Architecture Iridium SBDArgos ADSNWSTG GTSReal-time Data Ingest and DisseminationDatabase and File ManagementPublic Web PresentationDelayed Mode AnalysisConsole InterfacesDAC AnalystsPublic UsersAutomated QC and AlertsScientistsThe refreshed TAO Data System consists of 6 subsystems:Real-time Data Ingest and Dissemination subsystem – It decodes and stores the real-time Short Burst Data from refreshed buoys via Iridium and Argos messages from legacy buoys via POES. It also manages the GTS dissemination as the results of the data processing.Automated QC and Alerts subsystem – It generates alerting messages to DAC via Data Management Console. Alerts include buoy adrift, transmission outage, etc.Database and File Management subsystem – It stores all data and metadata associated with the TAO moorings. It supports Data Management Console, Delayed Mode Analysis Tool, and the Public Web Presentation.Data Management Console subsystem – It is the primary user interface for DAC Data Quality Analyst to monitor TAO Array data real-time data stream. It includes Daily Status Report, Buoy Position Report, Data Availability Report. It also supports Deployment/Recovery metadata entries.Delayed Mode Analysis subsystem – It is a tool set that processes the recovered data from moorings. The results of delayed mode data processing are uploaded into the database.Public Web Presentation subsystem – It is the primary user interface for the public to view and download TAO datasets.Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
52 H F Radar NDBC Stennis Ocean Surface Currents- speed and direction NDBC processes FTP transmissions and displays vectors on NDBC Web site via the WWWHF Radar Stations located along US CoastNDBC Stennis
53 Partner Data Processing RegionalObservatoriesSensor/ObservationObserversWeb-pageftp*NDBCWeb PageNDBCQA/QCPublicNDBCDial-A-BuoyWx. ChannelLocal MediaNWSGATEWAYGTSWFOsNODCNCDCet. al.* Via XML formatted files or the NDBC Meteorological and OceanographicData Exchange Module (MODEM) Kit - request from
54 Data Quality at NDBC 1970s - 2005 Started with NDBC in 1970s Focus on “Top Five”Wind Direction, Wind Speed, Atmospheric Surface Pressure, Air Temperature and WavesAlgorithms perform check at NWS GatewayQuality Assurance Group atNDBC provided daily check ofmarine observations – usually8 hours to two days afterdisseminationApprox. 110 platforms
55 Data Quality Control: The Last Line of Defense, after… Sensor EvaluationsIndividual Sensor CalibrationsPayload Software TestingBurn-InData Evaluation at deploymentPartner Data?Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
56 Quality Control of Observations No QC Done Onboard the Buoy or C-MANAutomated QC Done in Real-Time at NWSTGHard-flags:Stop the Release and Archive of Data unless Analyst Overrides.Override in advance – Storm Limits, orRemove before archiveSoft-flags: Climatology-based, AdvisoryHandbook of Automated Quality Control Checks and Procedures of the National Data Buoy CenterRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
57 Moored Buoy Observations Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
58 Quality Control Process Flow Real-Time:System Parameters (e.g., power)Message IntegrityAutomated Hard-FlagAutomated Soft-FlagPost-Release, Pre-Archive: Data AnalystRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
59 Hard-Flags by hierarchy - highest to lowest: T Transmission parity errorM Missing sensor dataW Wave message is short, checksum or parity errors.E Spectral Density are exceeded or are in errorD Delete measurement (“permanent failure”)S Invalid statistical parameter (e.g., mean > max)V Failed time continuity.L Failed range (climatological) limitsR Related measurement has failed a hard QC check (e.g., WVHGT fails → Periods failed).Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
60 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010 Soft-flags for Wavesa: Measurement is above monthly, regional limit.b: Measurement is below monthly, regional limit.c: Measurement has been adjusted, or corrected.f: Measurement failed hourly time continuity.m: High frequency spikes detected in the wave spectrum.p: Failed wave height to wave period comparison test.q: Swell direction is from an improbable direction.w: Failed wind direction verses wave direction check.x: Wind wave energy is too high for prevailing wind speed.y: Wind wave energy is too low for prevailing wind speed.Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
61 It all happens at the NWSTG Most other organizations: NWSTG is a big routerFor up to5 minutes
62 Why we halt the data at NWSTG Needed to process wavesData can be corrected:Can recover incorrect payload parametersSensor offset or position driftQuality ControlAdd computed fieldsAssign WMO code forms, routing identifiersRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
63 Configuration Control File for Each Station: GOES IDCodes and Routing IdentifiersTest or Operational ModeSensors FailedSensor HierarchySensor ScalingStation and Sensor ElevationsRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
64 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010 Derived MeasurementsPressure Tendency & Trace CharacteristicExtrapolated 10- and 20-m wind speedsSwell and wind wave estimatesWind ChillIce AccretionHeat IndexRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
65 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010 Output FormatsFM-13 for moored buoysC-MAN code for coastal stations:National code, but well-suited for coastal obs.Contains water level & wavesBased on FM-12 land synoptic codeFM-64 TESAC (temp., salinity, current profiles)FM-65 WAVEOB (spectral wave data)Will be BUFR Ready by 2012 WMO mandate!Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 201065
66 Real-Time Quality Control Purpose: Remove gross errorsData rejected: Virtually certain to be degradedTypical causes: Transmission errors, power degradations, broken cablesLimitations: Won’t detect minor errors, biasesRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
68 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010 Limit ChecksClassic way of catching a sensor that “pegs”.However, extreme, but valid, data has been withheld.Climatology basedCan be overridden before expected stormOriginally set to 3x Standard Deviation but found that could be too restrictiveRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 201068
69 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010 Time Continuity CheckDependent on time since last valid observation.Max. allowed value = 0.58*Std.dev.*SQRT(TimeDiff)Works well for normally distributed measurementsStd. dev. Chosen 50% higher climatic standard dev.Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
71 Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010 Time Continuity CheckMaximum allowable values in one hour:Sea level pressure 12.2 hPaAir Temperature deg. CWater Temperature 5.0 deg. CWind Speed m/sWave Height 3.5 mRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
72 If data fails time continuity, it may still be released if: Pressure: Both pressures < 1000 hPaWind Speed: Both pressures < 995 hPa or it agrees within 2 m/s of duplicateAir Temperature: Either wind speed > 7 m/s or wind direction change > 40 deg.Wave Height: Current wind speed > 15 m/sManually disabled in front of a hurricaneRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
73 Internal Consistency Checks If battery voltage < 10.5 V, pressure not released.Significant wave height and dominant period set to zero if significant wave height < 0.25m. Without this, large periods can result from nearly calm seas.Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
74 Internal Consistency Checks If dew pt. > air temp, set dew pt. = air temp. (RH can read slightly more than 100%)If ratio of gust to wind speed > 4, don’t send wind speed or gust.Regional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
75 Post Release Man-Machine Mix Pre-done Graphics produced:Flagged data firstThen all dataTools:Time series plotsSpectral Wave CurvesWind Wave scatterplotsSurface weather plotsRegional Association V, Fifteenth Session, Bali, 30 April – 6 May 2010
84 Data Monitoring What does the future hold? Implementing new Unidata WX analysis software with improved vis. toolsLook at only suspect or bad data- real time QC associated with a database- more sophisticated logic in algorithms, extended trend analysis- model fields used in real met and wave QCFlags assigned based on quality scoreDistribute and archive data with flags
85 Terima Kasih. Bill. Burnett@noaa Terima Kasih ! National Data Buoy Center Stennis Space Center, MS 3952985