Meteorological Assimilation Data Ingest System (MADIS) and ISO Data Quality Ted Habermann NOAA National Data Centers MADIS observations on April 29, 2004

MADIS and ISO Metadata MADIS produces and distributes thousands of quality reports for stations in U.S. meteorological mesonets every week. See for MADIS information. The ISO Metadata Standard includes an extensive section devoted to data quality reports. How can these fit together? MADIS mesonet stations in the North- American domain on May 26, 2004

Generating ISO Metadata Rich Inventory MADISTrackingNMMR Existing Quality Resources Existing Metadata Resources ISO Generator ISO Metadata REST Views This schematic shows our approach. Existing quality resources are queried using a web service (REST) for the “beef” of the quality report. The “bun” is provided from views into existing FGDC or new ISO sources. These two are combined into a complete ISO report. The “Beef” The “Bun”

Data Quality Granularity MADIS integrates observations from stations in many meteorological mesonets around the U.S. The MADIS Quality Repository includes: nine statistics for six parameters For thousands of stations from ~150 data networks every week. Quality reports can be generated at any level (or granularity) from this hierarchy.

ISO Quality Granularity DQ_Scope + level: MD_ScopeCode + extent [0..1] : EX_Extent + levelDescription [0..*] : MD_ScopeDescription MD_ScopeDescription + attributes : Set + features : Set + featureInstances : Set + attributeInstances : Set + dataset : CharacterString + other : CharacterString + nonGeographicDataset + dimensionGroup + feature + featureType + propertyType + fieldSession MD_ScopeCode + attribute + attributeType + collectionHardware + collectionSession + dataset + series + software + service + model + tile

ISO Quality Granularity In the geographic jargon an ISO feature is a station and an ISO attribute is a parameter MD_ScopeDescription + attributes : Set + features : Set + featureInstances : Set + attributeInstances : Set + dataset : CharacterString + other : CharacterString MD_ScopeCode = dataset (data network in this case) We considered several options for expressing report granularity in ISO and ended up with a general approach that seems to make sense (at least as a first draft): = list of parameters = list of stations

Select Report Select three weeks Select three stations Select two parameters

The HTML View Three weeks Two Parameters Three Stations The Quality Information

The “Beef” Three weeks Two Parameters Three Stations The Quality Information

The ISO Report

dataset IODOT /levelDescription> SLP,THETA RAMI4,RAVI4,RBUI4 ISO Report Scope Two Parameters Three Stations One Network

ISO Temporal Extent Week One Week Two

The ISO Quantitative Report "Start Date","End Date","Data Network","Station","Parameter","Total Count","Bad Count", "Accum_Sum","Accum_Sum2","Percent Bad","Mean","RMS“ " ", " ","IODOT","RAMI4","THETA",294,0,0,0,0,0,0 " ", " ","IODOT","RAMI4","SLP",0,0,0,0,0,0,0 " ", " ","IODOT","RAVI4","SLP",0,0,0,0,0,0,0 " ", " ","IODOT","RAVI4","THETA",282,5,36.96,276.1,1.77,7.39,7.43 " ", " ","IODOT","RBUI4","THETA",308,0,0,0,0,0,0 " ", " ","IODOT","RBUI4","SLP",0,0,0,0,0,0,0 " ", " ","IODOT","RAMI4","THETA",314,0,0,0,0,0,0 " ", " ","IODOT","RAMI4","SLP",0,0,0,0,0,0,0 " ", " ","IODOT","RAVI4","THETA",291,0,0,0,0,0,0 " ", " ","IODOT","RAVI4","SLP",0,0,0,0,0,0,0 " ", " ","IODOT","RBUI4","THETA",316,0,0,0,0,0,0 " ", " ","IODOT","RBUI4","SLP",0,0,0,0,0,0,0 " ", " ","IODOT","RAMI4","THETA",671,0,0,0,0,0,0 " ", " ","IODOT","RAMI4","SLP",0,0,0,0,0,0,0 " ", " ","IODOT","RAVI4","THETA",633,0,0,0,0,0,0 " ", " ","IODOT","RAVI4","SLP",0,0,0,0,0,0,0 " ", " ","IODOT","RBUI4","THETA",622,0,0,0,0,0,0 " ", " ","IODOT","RBUI4","SLP",0,0,0,0,0,0,0 The Quality Information

Conclusions Yes, MADIS Quality Information can be reported as ISO The ScopeCode is general with details in the LevelDescription We need to learn about describing feature and attribute sets in Feature Catalogs. At present we are using comma separated lists. We need to learn about how to describe the formats of reports. At present we are using CSV (comma separated values)