Documenting Data Quality Ted Habermann, NOAA/NESDIS/NGDC Documentation: It’s not just discovery... 50% change in global average Why? i checked my 2002.

Slides:



Advertisements
Similar presentations
GEOSS AIP Phase 2 Kickoff Workshop September, Boulder Colorado, USA AP ISO 1.0 Jürgen Walther Office of the Interministerial Committee for Geo Information.
Advertisements

Evolution of Metadata Standards: New Features in ISO Ted Habermann NOAA National Data Centers September, 2008
Merging Metadata Standards: FGDC CSDGM and ISO Sharon Shin Federal Geographic Data Committee Metadata Coordinator
NOAA Documentation Improvement Ted Habermann How do we measure and visualize improvements in NOAA Documentation? Record Count Completeness (Rubric Scores)
Analyzing Systems Using Data Dictionaries Systems Analysis and Design, 7e Kendall & Kendall 8 © 2008 Pearson Prentice Hall.
Data Dictionary What does “Backordered item” mean? What does “New Customer info.” contain? How does the “account receivable report” look like?
Overview of key concepts and features
© 2012 The MITRE Corporation. All rights reserved. For internal MITRE use 11 July 2013 Meeting #6 hData Record Format Task Force 1 © 2012 The MITRE Corporation.
CS31: Introduction to Computer Science I Discussion 1A 4/2/2010 Sungwon Yang
Sunday, June 28, 2015 Abdelali ZAHI : FALL 2003 : XML Schemas XML Schemas Presented By : Abdelali ZAHI Instructor : Dr H.Haddouti.
11 Data Interface Standard for Accounting Software Project Progress Report China National Audit Office June, 2015.
NOAA Metadata Update Ted Habermann. NOAA EDMC Documentation Directive This Procedural Directive establishes 1) a metadata content standard (International.
Introduction to Geospatial Metadata – ISO 191** Metadata National Coastal Data Development Center A division of the National Oceanographic Data Center.
National Coastal Data Development Center A division of the National Oceanographic Data Center Please a list of participants at each location to
Evolution of Metadata Standards: New Features in ISO Ted Habermann NOAA National Data Centers December, 2007
ISO Standards: Status, Tools, Implementations, and Training Standards/David Danko.
Presentation Outline Metadata Coverage and Guidelines SMAP ISO Requirement Metadata Accessibility – HDF5 Group/Attribute Multiple Instantiation of the.
AIXM 5.1 Seminar 12 – 13 December 2011
Interoperable Documentation Ted Habermann, NOAA/NESDIS/NGDC NCAR Earth Observing Laboratory, June 2010 Links: GEO-IDE Wiki:
1 CIS336 Website design, implementation and management (also Semester 2 of CIS219, CIS221 and IT226) Lecture 6 XSLT (Based on Møller and Schwartzbach,
Rensselaer Polytechnic Institute CSCI-4380 – Database Systems David Goldschmidt, Ph.D.
Processing of structured documents Spring 2002, Part 2 Helena Ahonen-Myka.
Designing a Database (Part I) -Identify all fields needed to produce the required information -Group related fields into tables -Determine Each Table’s.
Content and Computer Platforms Week 3. Today’s goals Obtaining, describing, indexing content –XML –Metadata Preparing for the installation of Dspace –Computers.
Lis512 lecture 4 XML: documents and records. up until now Relational databases can store information that is internal to an organization. But a lot of.
Metadata Implementation Ted Habermann NOAA National Geophysical Data Center Documentation: It’s not just discovery... 50% change in global average Why?
Transitioning from FGDC CSDGM Metadata to ISO 191** Metadata National Coastal Data Development Center A division of the National Oceanographic Data Center.
National Coastal Data Development Center A division of the National Oceanographic Data Center Please a list of participants at each location to
Chapter 8 Analyzing Systems Using Data Dictionaries Systems Analysis and Design Kendall & Kendall Sixth Edition.
Documentation from NcML to ISO Ted Habermann, NOAA NESDIS NGDC.
Documenting Data Quality Ted Habermann, NOAA National Geophysical Data Center.
Documentation Foundation Spectrum Relational Tables XML/Relational Database (with some fields) XML Blobs (with some fields) File Systems XML Blobs in Database.
Workshop Resources.
® Reading meeting. December 12-14th, 2011 QUAlity aware VIsualisation for the Global Earth Observation system of systems GCI Analysis December 12-14th,
AIXM 5 Metadata. Requirements for AIXM Metadata AIXM Metadata Model Examples Requirements for AIXM Metadata AIXM Metadata Model Examples.
Part One Review Policy Intro to ISO Levels of Metadata Data Discovery.
Transitioning from FGDC CSDGM Metadata to ISO 191** Metadata
National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology Pasadena, California EDGE: The Multi-Metadata.
Meteorological Assimilation Data Ingest System (MADIS) and ISO Data Quality Ted Habermann NOAA National Data Centers MADIS observations on April 29, 2004.
Analyzing Systems Using Data Dictionaries Systems Analysis and Design, 8e Kendall & Kendall 8.
Introduction to Geospatial Metadata – ISO 191** Metadata National Centers for Environmental Information (NCEI)
Documenting UAF Data Ted Habermann NOAA/NESDIS/National Geophysical Data Center.
Creating Good Documentation NOAA National Geophysical Data Center
Why Standardize Metadata?. Why Have a Standard? Think for a moment how hard it would be to… … bake a cake without standard units of measurement. … put.
第三部分 类图的实例 u 动态增减属性 u 地理信息元数据模型. 动态增减属性 u 数据库中的基本单位是表,表的 基本成分是属性(表的项或列) 。表的列数是固定的。但是,可 以通过程序来实现 “ 动态 ” 增减表项 。 u 在类图模型中如何表示属性的动 态增减?
Spatial Databases and Metadata.
Barry Weiss 1/4/ Jet Propulsion Laboratory, California Institute of Technology Quality Elements in ISO Metadata Design for Proposed SMAP Data.
UML Basics and XML Basics Navigating the ISO Standards.
(Winter 2016) Instructor: Craig Duckett Lecture 13: Thursday, February 18 th Mere Mortals: Chap. 9 Summary, Team Work 1.
Metadata Content Entering Metadata Information. Discovery vs. Access vs. Understanding Cannot search on content if it is not documented. Cannot access.
Merging Metadata Standards: FGDC CSDGM and ISO and Sharon Shin Federal Geographic Data Committee Metadata Coordinator
Standard Metadata in Scientific Data Formats September 19, 2007 Flash at:
ISO 191** Overview A “Family” of Standards. Resources ISO Standards Web Page – Technical.
Forecast Model Run Collections and ISO Ted Habermann There has been considerable discussion of describing multiple times in forecast datasets This is not.
A look to the past for the future- The North American Profile Sharon Shin Metadata Coordinator Federal Geographic Data Committee.
Data Quality for Long-Term Datasets
WMO GRIB Edition 3 Enrico Fucile Inter-Program Expert Team on Data Representation Maintenance and Monitoring IPET-DRMM Geneva, 30 May – 3 June 2016.
Computer Science: A Structured Programming Approach Using C1 Objectives ❏ To understand the structure of a C-language program. ❏ To write your first C.
WELL- FORMEDNESS CH 6. Objective Well-formedness rules Text in XML Elements and Tags in Atributes Entity references CDATA sections Comments Unicode XML1.1.
Geospatial metadata Prof. Wenwen Li School of Geographical Sciences and Urban Planning 5644 Coor Hall
CITA 330 Section 2 DTD. Defining XML Dialects “Well-formedness” is the minimal requirement for an XML document; all XML parsers can check it Any useful.
1 XML and XML in DLESE Katy Ginger November 2003.
Building Tables & Defining Relationships
Introduction to the C Language
Software Specification Tools
Gap: Poorly Understood Responsibilities for Integration
Documenting Data Quality Ted Habermann, NOAA/NESDIS/NGDC
GDI ISO Standards GovData Metadata
Proposal of a Geographic Metadata Profile for WISE
Metadata Updates (for S / 4
Presentation transcript:

Documenting Data Quality Ted Habermann, NOAA/NESDIS/NGDC Documentation: It’s not just discovery... 50% change in global average Why? i checked my archives, and here is what i found out: it appears that the current 3rd generation algorithm was implemented into operations around Oct-Nov 2002 time frame. cannot say more precisely, as all correspondence i am looking at, talks about this indirectly. (maybe it's what's referred to as the Phase II algorithm.) At the same time, we had implemented quite a few other changes fixing data bugs and formats: view angle problem, increased digitization in all channel's reflectances and AODs, etc. The jump is deemed due to introducing 3rd generation algorithm, which replaced the 2nd generation. The new numbers (~0.08) look more realistic than the previous ones (~0.05 or so). The changes seen in the data is close to the expected effect of this change. The 3rd gen alg takes into account the exact spectral response, whereas the 2nd gen is generic ("one size fits all"). hopefully this settles the issue..

Data Quality - Documents

Data Quality - Granules

> DQ_Scope + level : MD_ScopeCode + extent [0..1] : EX_Extent + levelDescription [0..*] : MI_Metadata Data Quality - Standards > DQ_Element DQ_ConformanceResult + specification : CI_Citation + explanation : CharacterString + pass : Boolean > MD_EvaluationMethodTypeCode + directInternal + directExternal + indirect DQ_DataQuality + scope : DQ_Scope + standAloneReport 0..1 DQ_QuantitativeResult + valueType [0..1] : RecordType + valueUnit : UnitOfMeasure + errorStatistic [0..1] : CharacterString + value [1..*] : Record > MD_ScopeCode + attribute + feature + attributeType + featureType + collectionHardware + propertyType + collectionSession + fieldSession + dataset + software + series + service + nonGeographicDataset + model + dimensionGroup + tile DQ_CoverageResult + report 0..* DQ_StandaloneReportInformation + reportReference : CI_Citation + abstract: CharacterString DQ_MeasureReferenceDQ_EvaluationDQ_Result + resultScope: DQ_Scope [0..1] DQ_DescriptiveResult > MD_ScopeDescription + attributes : Set + features : Set + featureInstances : Set + attributeInstances : Set + dataset : CharacterString + other : CharacterString LI_Lineage

GOES-R Data Quality - Documents Level 2+ Volcanic Ash: Detection and Height L2+ Volcanic Ash Science Description L2+ Volcanic Ash Algorithm Description L2+ Volcanic Ash Source Information L2+ Volcanic Ash Applicable ATBDs L2+ Volcanic Ash Quality Algorithms L2+ Volcanic Ash Source Data Inputs L2+ Volcanic Ash Production Notes L2+ Volcanic Ash Data Fields (TBR-16) L2+ Volcanic Ash Metadata Description and Definition L2+ Volcanic Ash Expected Periodicity

Documents = Standards Science Description - MD_DataIdentification/abstract Algorithm Description - LE_Algorithm/description Source Information - MD_DistributionInformation Applicable ATDBs - LE_Algorithm/citation Quality Algorithms - DQ_DataQuality/DQ_MeasureReference Source Data Inputs - LI_Lineage/source Production Notes - processStep/description Data Fields- MD_ContentInfo Metadata Description and Definition - seems redundant Expected Periodicity - resourceMaintenance

Documentation Objects = Standards NESDIS Documentation Object Mapping Metadata Document System Description Document System Maintenance Manual Interface Control Document Algorithm Theoretical Basis Document

Multiple Dialects of the Same Content Documents CI_Citation XSLT Translation XML Reference Granules/Catalogs Standards

Community - the Wiki

> DQ_Scope + level : MD_ScopeCode + extent [0..1] : EX_Extent + levelDescription [0..*] : MI_Metadata DQ_DataQuality > DQ_Element DQ_ConformanceResult + specification : CI_Citation + explanation : CharacterString + pass : Boolean > MD_EvaluationMethodTypeCode + directInternal + directExternal + indirect DQ_DataQuality + scope : DQ_Scope + standAloneReport 0..1 DQ_QuantitativeResult + valueType [0..1] : RecordType + valueUnit : UnitOfMeasure + errorStatistic [0..1] : CharacterString + value [1..*] : Record > MD_ScopeCode + attribute + feature + attributeType + featureType + collectionHardware + propertyType + collectionSession + fieldSession + dataset + software + series + service + nonGeographicDataset + model + dimensionGroup + tile DQ_CoverageResult + report 0..* DQ_StandaloneReportInformation + reportReference : CI_Citation + abstract: CharacterString DQ_MeasureReferenceDQ_EvaluationDQ_Result + resultScope: DQ_Scope [0..1] DQ_DescriptiveResult > MD_ScopeDescription + attributes : Set + features : Set + featureInstances : Set + attributeInstances : Set + dataset : CharacterString + other : CharacterString LI_Lineage 1

ISO Lineage Model Source Step Product Processing and Algorithm Descriptions

LI_Lineage

Attributes: role [how many] : object type how many = [minimum..maximum] minimum = 0: optional minimum = 1: required * = any number how many = blank: required, one how many = [1..*] : required, any number how many = [1..2] : required, one or two how many = [0..1] : optional, zero or one how many = [0..*] : optional, any number Type: package abbreviation_type UML package abbreviation = XML namespace = Document section Role: what this object does for me contact: CI_ResponsibleParty description: CharacterString Operations: generally not used in ISO UML UML.1

UML.2 LI_Lineage = the LI_Lineage class is in the Lineage (LI) Package statement [0..1] : CharacterString = LI_Lineage can have up to one statement which is a CharacterString source [0..*] : LI_Source = LI_Lineage can have any number of sources which are LI_Sources processStep [0..*] : LI_Lineage can have any number of processSteps which are LE_ProcessSteps

Volcanic Ash Detection Sources

Volcanic Ash Detection Processing

ISO Lineage DQ_Lineage ( ) MI_Metadata + lineage 0..1 LI_Lineage + statement [0..1] : CharacterString LE_Source + description [0..1] : CharacterString + scaleDenominator [0..1] : MD_RepresentativeFraction + sourceReferenceSystem [0..1] : MD_ReferenceSystem + sourceCitation [0..1] : CI_Citation + sourceExtent [0..*] : EX_Extent + processedLevel[0..1] : MD_Identifier + resolution[0..1] : LE_NominalResolution + sourcemetadata [0..*] : MD_Reference LE_ProcessStep + description : CharacterString + rationale [0..1] : CharacterString + dateTime [0..1] : DateTime + processor [0..*] : CI_ResponsibleParty + extent [0..*] : EX_Extent + reference [0.*] : CI_Citation + source 0..* + processStep 0..* + output, source 0..* If(count(source) + count(processStep) =0) and (DQ_DataQuality.cope.level = 'dataset' or 'series') then statement is mandatory LE_Processing + identifier : MD_Identifier + softwareReference[0..*] : CI_Citation + procedureDescription[0..1] : CharacterString + documentation[0..*] : CI_Citation + runTimeParameters[0..1] : CharacterString LE_Algorithm + citation: CI_Citation + description : CharacterString + processingInformation 0..* + algorithm 0..* LE_ProcessStepReport + name : CharacterString + description[0..1] : CharacterString + fileType[0..1] : CharacterString + report 0..* + sourceStep 0..*

Granule Lineage - 1 Brief Text Brief Text (PUG) Citation Citation Product Anciliary Data Auxiliary Data Lookup Table Product

Granule Lineage - 2 Source:clear_sky_masks Could boil down to

Volcanic Ash Detection Lineage in the Granule Option 1: one identifier: Option 2: lineage group with filenames as unique identifiers: Option 3: lineage group with uniqueIdentifiers: includes processingInformation / algorithm / output

Database and XML Keys Citation ID Title Date Friend_ID Location_ID Citation ID Title Date Friend_ID Location_ID Person ID Name Person ID Name OnlineResource ID Name URL OnlineResource ID Name URL XML …

XML Attributes: Objects and References ISO XML consists of tags, elements (with or without content), and attributes. An attribute is a name/value pair that exists within a start-tag or empty-element tag. Attributes provide additional information about an element which is not part of the data. Attribute values must contain either single or double quotes. This example shows a step element with one attribute, number with a value of “3”: Connect A to B. Many of the XML attributes used in the ISO Standards fall into two groups: identifiers and references: Identifiers: id and uuid References: uuidref and xlink:href Objects that start with upper case letters have identifiers (id and uuid) Roles that start with lower case letters have references (uuidref and xlink:href) object: CI_ResponsibleParty id="JaneDoe" object: CI_ResponsibleParty id="JohnDoe" role: friend xlink:href=#JohnDoerole: friend xlink:href=#JaneDoe

ISO Lineage Model - 2 Source A step ps1 Source B step ps1 Source C step ps2 Source D step ps2 Source E step ps3 Step ps1 source A source B output C Step ps2 source C source D output E Step ps3 source E Product Processing and Algorithm Descriptions

XML Attributes: Objects and References >

ISO Lineage DQ_Lineage ( ) MI_Metadata + lineage 0..1 LI_Lineage + statement [0..1] : CharacterString LE_Source + description [0..1] : CharacterString + scaleDenominator [0..1] : MD_RepresentativeFraction + sourceReferenceSystem [0..1] : MD_ReferenceSystem + sourceCitation [0..1] : CI_Citation + sourceExtent [0..*] : EX_Extent + processedLevel[0..1] : MD_Identifier + resolution[0..1] : LE_NominalResolution + sourcemetadata [0..*] : MD_Reference LE_ProcessStep + description : CharacterString + rationale [0..1] : CharacterString + dateTime [0..1] : DateTime + processor [0..*] : CI_ResponsibleParty + extent [0..*] : EX_Extent + reference [0.*] : CI_Citation + source 0..* + processStep 0..* + output, source 0..* If(count(source) + count(processStep) =0) and (DQ_DataQuality.cope.level = 'dataset' or 'series') then statement is mandatory LE_Processing + identifier : MD_Identifier + softwareReference[0..*] : CI_Citation + procedureDescription[0..1] : CharacterString + documentation[0..*] : CI_Citation + runTimeParameters[0..1] : CharacterString LE_Algorithm + citation: CI_Citation + description : CharacterString + processingInformation 0..* + algorithm 0..* LE_ProcessStepReport + name : CharacterString + description[0..1] : CharacterString + fileType[0..1] : CharacterString + report 0..* + sourceStep 0..* References

> DQ_Scope + level : MD_ScopeCode + extent [0..1] : EX_Extent + levelDescription [0..*] : MI_Metadata DQ_DataQuality > DQ_Element DQ_ConformanceResult + specification : CI_Citation + explanation : CharacterString + pass : Boolean > MD_EvaluationMethodTypeCode + directInternal + directExternal + indirect DQ_DataQuality + scope : DQ_Scope + standAloneReport 0..1 DQ_QuantitativeResult + valueType [0..1] : RecordType + valueUnit : UnitOfMeasure + errorStatistic [0..1] : CharacterString + value [1..*] : Record > MD_ScopeCode + attribute + feature + attributeType + featureType + collectionHardware + propertyType + collectionSession + fieldSession + dataset + software + series + service + nonGeographicDataset + model + dimensionGroup + tile DQ_CoverageResult + report 0..* DQ_StandaloneReportInformation + reportReference : CI_Citation + abstract: CharacterString DQ_MeasureReferenceDQ_EvaluationDQ_Result + resultScope: DQ_Scope [0..1] DQ_DescriptiveResult > MD_ScopeDescription + attributes : Set + features : Set + featureInstances : Set + attributeInstances : Set + dataset : CharacterString + other : CharacterString LI_Lineage 1

DQ_Scope

Attributes: role [how many] : object type how many = [minimum..maximum] minimum = 0: optional minimum = 1: required * = any number how many = blank: required, one how many = [1..*] : required, any number how many = [1..2] : required, one or two how many = [0..1] : optional, zero or one how many = [0..*] : optional, any number Type: package abbreviation_type UML package abbreviation = XML namespace = Document section Role: what this object does for me contact: CI_ResponsibleParty description: CharacterString Operations: generally not used in ISO UML UML.1

> DQ_Scope + level : MD_ScopeCode + extent [0..1] : EX_Extent + levelDescription [0..*] : MD_ScopeDescription UML.2 > DQ_Scope = the DQ_Scope is a DataType in the Data Quality (DQ) Package level : MD_ScopeCode = a DQ_Scope must have one level which is a MD_ScopeCode extent [0..1] : EX_Extent = a DQ_Scope can have up to 1 extent which is an EX_Extent levelDescription [0..*] : MD_ScopeDescription = a DQ_Scope can have any number of levelDescriptions which are MD_ScopeDescriptions

DQ_Scope > DQ_Scope + level : MD_ScopeCode + extent [0..1] : EX_Extent + levelDescription [0..*] : MD_ScopeDescription > MD_ScopeCode + attribute + feature + attributeType + featureType + collectionHardware + propertyType + collectionSession + fieldSession + dataset + software + series + service + nonGeographicDataset + model + dimensionGroup + tile > MD_ScopeDescription + attributes : Set + features : Set + featureInstances : Set + attributeInstances : Set + dataset : CharacterString + other : CharacterString

Dessert Pie Ice Cream Apple Pecan Vanilla Chocolate > Dessert > Dessert > Pie > Pie > Ice Cream > Ice Cream Apple Pecan Vanilla Chocolate Abstract Dessert = "is a" or "can be a"

> EX_Extent +description [0..1]: Character String EX_BoundingPolygon +polygon [0..1]: GM_Object EX_GeographicBoundingBox +westBoundingLongitude: Decimal +eastBoundingLongitude: Decimal +southBoundingLatiitude: Decimal +northBoundingLatiitude: Decimal EX_GeographicDescription +geographicIdentifier: MD_Identifier EX_VerticalExtent +minimumValue: Real +maximumValue: Real EX_TemporalExtent +extent: TM_Primitive EX_SpatialTemporalExtent > EX_GeographicExtent +extentTypeCode [0..1]: Boolean="1" count(description + geographicElement + temporalElement + verticalElement) > 0 EX_Extent

> DQ_Scope + level : MD_ScopeCode + extent [0..1] : EX_Extent + levelDescription [0..*] : MI_Metadata DQ_DataQuality > DQ_Element DQ_ConformanceResult + specification : CI_Citation + explanation : CharacterString + pass : Boolean > MD_EvaluationMethodTypeCode + directInternal + directExternal + indirect DQ_DataQuality + scope : DQ_Scope + standAloneReport 0..1 DQ_QuantitativeResult + valueType [0..1] : RecordType + valueUnit : UnitOfMeasure + errorStatistic [0..1] : CharacterString + value [1..*] : Record > MD_ScopeCode + attribute + feature + attributeType + featureType + collectionHardware + propertyType + collectionSession + fieldSession + dataset + software + series + service + nonGeographicDataset + model + dimensionGroup + tile DQ_CoverageResult + report 0..* DQ_StandaloneReportInformation + reportReference : CI_Citation + abstract: CharacterString DQ_MeasureReferenceDQ_EvaluationDQ_Result + resultScope: DQ_Scope [0..1] DQ_DescriptiveResult > MD_ScopeDescription + attributes : Set + features : Set + featureInstances : Set + attributeInstances : Set + dataset : CharacterString + other : CharacterString LI_Lineage 1

DQ_StandAloneReport

MI_Metadata StandAloneReport DQ_DataQuality + scope : DQ_Scope + standAloneReport 0..1 DQ_StandaloneReportInformation + reportReference : CI_Citation + abstract: CharacterString Global or Variable Attribute:

> DQ_Scope + level : MD_ScopeCode + extent [0..1] : EX_Extent + levelDescription [0..*] : MI_Metadata DQ_DataQuality > DQ_Element DQ_ConformanceResult + specification : CI_Citation + explanation : CharacterString + pass : Boolean > MD_EvaluationMethodTypeCode + directInternal + directExternal + indirect DQ_DataQuality + scope : DQ_Scope + standAloneReport 0..1 DQ_QuantitativeResult + valueType [0..1] : RecordType + valueUnit : UnitOfMeasure + errorStatistic [0..1] : CharacterString + value [1..*] : Record > MD_ScopeCode + attribute + feature + attributeType + featureType + collectionHardware + propertyType + collectionSession + fieldSession + dataset + software + series + service + nonGeographicDataset + model + dimensionGroup + tile DQ_CoverageResult + report 0..* DQ_StandaloneReportInformation + reportReference : CI_Citation + abstract: CharacterString DQ_MeasureReferenceDQ_EvaluationDQ_Result + resultScope: DQ_Scope [0..1] DQ_DescriptiveResult > MD_ScopeDescription + attributes : Set + features : Set + featureInstances : Set + attributeInstances : Set + dataset : CharacterString + other : CharacterString LI_Lineage 1

DQ_Element > DQ_Element DQ_MeasureReference + measureIdentification: MD_Identifier [0..1] + nameOfMeasure: CharacterString [0..*] + measureDescription: CharacterString [0..1] DQ_EvaluationMethod + dateTime: DateTime [0..*] + evaluationMethodDescription: CharacterString [0..1] + evaluationProceedure: CI_Citation [0..1] + referenceDoc: CI_Citation [0..*] + evaluationMethodType: DQ_EvaluationMethodTypeCode [0..1] DQ_Result + dateTime: DateTime [0..*] + resultScope: DQ_ScopeCode [0..1]

DQ_MeasureReference + measureIdentification: MD_Identifier [0..1] + nameOfMeasure: CharacterString [0..*] + measureDescription: CharacterString [0..1]

Measure Registry / Database Quality Measure measure identifier name alias element name basic measure definition description parameter value type value structure source reference example Quality Measure measure identifier name alias element name basic measure definition description parameter value type value structure source reference example DQ_MeasureReference + measureIdentification: MD_Identifier [0..1] + nameOfMeasure: CharacterString [0..*] + measureDescription: CharacterString [0..1]

DQM_Measure + measureIdentifier: MD_Identifier + name: CharacterString + alias: CharacterString [0..*] + elementName: TypeName [1..*] + definition: CharacterString + description: DQM_Description + valueType: TypeName + valueStructure: DQM_ValueStructure + example: DQM_Description [0..*] DQM_SourceReference + citation: CI_Citation DQM_BasicMeasure + name: CharacterString + definition: CharacterString + example: DQM_Description [0..*] + valueType: TypeName DQM_Description + textDescription: CharacterString + extendedDescription: MD_BrowseGraphic DQM_Parameter + name: CharacterString + definition: CharacterString + description: DQM_Description + example: DQM_Description [0..*] + valueType: TypeName + valueStructure: DQM_ValueStructure > DQM_ValueStructure + bag+table + set+ matrix + sequence+coverage

Variable Data Quality - Scalers

Granule Data Quality - Scalers

> DQ_Scope + level : MD_ScopeCode + extent [0..1] : EX_Extent + levelDescription [0..*] : MI_Metadata DQ_DataQuality > DQ_Element DQ_ConformanceResult + specification : CI_Citation + explanation : CharacterString + pass : Boolean > MD_EvaluationMethodTypeCode + directInternal + directExternal + indirect DQ_DataQuality + scope : DQ_Scope + standAloneReport 0..1 DQ_QuantitativeResult + valueType [0..1] : RecordType + valueUnit : UnitOfMeasure + errorStatistic [0..1] : CharacterString + value [1..*] : Record > MD_ScopeCode + attribute + feature + attributeType + featureType + collectionHardware + propertyType + collectionSession + fieldSession + dataset + software + series + service + nonGeographicDataset + model + dimensionGroup + tile DQ_CoverageResult + report 0..* DQ_StandaloneReportInformation + reportReference : CI_Citation + abstract: CharacterString DQ_MeasureReferenceDQ_EvaluationDQ_Result + resultScope: DQ_Scope [0..1] DQ_DescriptiveResult > MD_ScopeDescription + attributes : Set + features : Set + featureInstances : Set + attributeInstances : Set + dataset : CharacterString + other : CharacterString LI_Lineage 1

DQ_Result DQ_ConformanceResult + specification : CI_Citation + explanation : CharacterString + pass : Boolean DQ_QuantitativeResult + valueType [0..1] : RecordType + valueUnit : UnitOfMeasure + errorStatistic [0..1] : CharacterString + value [1..*] : Record DQ_DescriptiveResult + statement: CharacterString QE_CoverageResult + resultFile : MX_DataFile + resultFormat: MD_Format + resultContentDescription: MD_CoverageDescription + resultSpatialRepresentation: MD_SpatialRepresentation + spatialRepresentationType : MD_SpatialRepresentationTypeCode DQ_Result + resultScope: DQ_Scope [0..1]

Granule Metadata (NcML) datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata datadata getNcML NcML Schema Variable Type Definition

<gco:RecordType xlink:href=" of netCDF variableType <gco:Record xlink:href=" Attributes for variable = memberName in granule = granuleIdentifier> Variable Quality - ISO with XML Reference RecordType Record

<gco:RecordType xlink:href=" of netCDF groupType <gco:Record xlink:href=" Quality:ScalerInformation])"> Attributes for variable = Quality:ScalerInformation in granule = granuleIdentifier> Granule Quality - ISO with XML Reference RecordType Record

attribute ash_cloud_height dq:count_cloud_temperature_high_quality int 1024 Standard Data Quality - ISO without reference Value Type Measure Variable

DQ_Result DQ_ConformanceResult + specification : CI_Citation + explanation : CharacterString + pass : Boolean DQ_QuantitativeResult + valueType [0..1] : RecordType + valueUnit : UnitOfMeasure + errorStatistic [0..1] : CharacterString + value [1..*] : Record DQ_DescriptiveResult + statement: CharacterString QE_CoverageResult + resultFile : MX_DataFile + resultFormat: MD_Format + resultContentDescription: MD_CoverageDescription + resultSpatialRepresentation: MD_SpatialRepresentation + spatialRepresentationType : MD_SpatialRepresentationTypeCode DQ_Result + resultScope: DQ_Scope [0..1]

Variable Data Quality - Coverages <attribute name="flag_masks" value="1b 1b 2b 2b 4b 4b 56b 56b 56b 192b 192b 1892b 1892b 1892b 1892b 1892b"/>

MD_Band + peakResponse [0..1] : Real + bitsPerValue [0..1] : Integer + toneGradation [0..1] : Integer MI_CoverageDescription Revisions MD_Metadata +contentInfo 0..* > MD_CoverageContentTypeCode + image + thematicClassification + physicalMeasurement + referenceInformation + qualityInformation + auxilliaryData + modelResult MD_CoverageDescription + attributeDescription : RecordType + contentType [1.*]: MD_CoverageContentTypeCode + processingLevelCode [0..1]: MD_Identifier +dimension 0..* MI_RangeElementDescription + name : CharacterString + definition : CharacterString + rangeElement[1..*] : Record +rangeElementDescription 0..* MD_SampleDimension + minValue [0..1] : Real + maxValue [0..1] : Real + units [0..1] : UnitOfMeasure + scaleFactor [0..1] : Real + offset [0..1] : Real + numberOfValues [0..1] : Integer + meanValue [0..1] : Real + standardDeviation [0..1] : Real + otherAttributeType [0..1] : RecordType + otherAttribute [0..1] : Record MD_RangeDimension + sequenceIdentifier [0..1] : MemberName + name[0..*]: MD_Identifier + description [0..1] : CharacterString minValue, maxValue and units must have units of length. RangeElement, otherAttributeType, and other Attribute have cardinality [0..0] +rangeElementDescription 0..*

ISO Flags Overall_QF High_Quality 0 Overall_QF Low_Quality 1 Flag Meaning Flag Name Flag Value Flag Name Flag Value

DQ_Evaluation + dateTime: DateTime [0..*] + evaluationMethodDescription: CharacterString [0..1] + evaluationProcedure: CI_Citation [0..1] + referenceDoc: CI_Citation [0..*] + evaluationMethodType: DQ_EvaluationMethodTypeCode [0..1] DQ_DataEvaluationDQ_Aggregation + sourceQualityResult: CharacterString [2..*] DQ_FullInspectionDQ_SamplebasedInspection + samplingScheme: CharacterString + lotDescription: CharacterString + samplingRatio: CharacterString DQ_IndirectEvaluation > DQ_EvaluationMethodTypeCode + directInternal + directExternal + indirect

Multiple Dialects of the Same Content Documents CI_Citation XSLT Translation XML Reference Granules/Catalogs Standards

> DQ_Scope + level : MD_ScopeCode + extent [0..1] : EX_Extent + levelDescription [0..*] : MI_Metadata Data Quality - Standards > DQ_Element DQ_ConformanceResult + specification : CI_Citation + explanation : CharacterString + pass : Boolean > MD_EvaluationMethodTypeCode + directInternal + directExternal + indirect DQ_DataQuality + scope : DQ_Scope + standAloneReport 0..1 DQ_QuantitativeResult + valueType [0..1] : RecordType + valueUnit : UnitOfMeasure + errorStatistic [0..1] : CharacterString + value [1..*] : Record > MD_ScopeCode + attribute + feature + attributeType + featureType + collectionHardware + propertyType + collectionSession + fieldSession + dataset + software + series + service + nonGeographicDataset + model + dimensionGroup + tile DQ_CoverageResult + report 0..* DQ_StandaloneReportInformation + reportReference : CI_Citation + abstract: CharacterString DQ_MeasureReferenceDQ_EvaluationDQ_Result + resultScope: DQ_Scope [0..1] DQ_DescriptiveResult > MD_ScopeDescription + attributes : Set + features : Set + featureInstances : Set + attributeInstances : Set + dataset : CharacterString + other : CharacterString LI_Lineage

Questions? Questions / Comments / Suggestions:

> DQ_Scope + level : MD_ScopeCode + extent [0..1] : EX_Extent + levelDescription [0..*] : MD_Metadata DQ_DataQuality LI_Lineage > DQ_Element + nameOfMeasure [0..*] : CharacterString + measureIdentification [0..1] : MD_Identifier + measureDescription [0..1] : CharacterString + evaluationMethodType [0..1] : DQ_EvaluationMethodTypeCode + evaluationMethodDescription [0..1] : CharacterString + evaluationProcedure [0..1] : CI_Citation + dateTime [0..*] : DateTime + result [1..2] : DQ_Result DQ_ConformanceResult + specification : CI_Citation + explanation : CharacterString + pass : Boolean > MD_EvaluationMethodTypeCode + directInternal + directExternal + indirect DQ_DataQuality + scope : DQ_Scope + lineage 0..1 DQ_QuantitativeResult + valueType [0..1] : RecordType + valueUnit : UnitOfMeasure + errorStatistic [0..1] : CharacterString + value [1..*] : Record "report" or "linage" role is mandatory if scope.DQ_Scope.level = 'dataset' "levelDescription" is mandatory if "level" notEqual 'dataset' or 'series' > DQ_Result > MD_ScopeCode + attribute + feature + attributeType + featureType + collectionHardware + propertyType + collectionSession + fieldSession + dataset + software + series + service + nonGeographicDataset + model + dimensionGroup + tile DQ_CoverageResult + report 0..*

DQ_Element > DQ_Element + nameOfMeasure [0..*] : CharacterString + measureIdentification [0..1] : MD_Identifier + measureDescription [0..1] : CharacterString + evaluationMethodType [0..1] : DQ_EvaluationMethodTypeCode + evaluationMethodDescription [0..1] : CharacterString + evaluationProcedure [0..1] : CI_Citation + dateTime [0..*] : DateTime + result [1..2] : DQ_Result > DQ_Element + measure [0..*] : DQ_MeasureReference + evaluation [0..1] : DQ_Evaluation + result [1..2] : DQ_Result DQ_MeasureReference + measureIdentification: MD_Identifier [0..1] + nameOfMeasure: CharacterString [0..*] + measureDescription: CharacterString [0..1]