Metadata Understanding the Value and Importance of Proper Data Documentation Michael Moeller Metadata Specialist NOAA Coastal Services Center

What is Metadata?

Simply put, metadata is information about your data. What is Metadata?

This is the metadata for this. What’s Missing? Emily and Madison

Author(s) Boullosa, Carmen. Title(s) They're cows, we're pigs / by Carmen Boullosa Place New York : Grove Press, Physical Descr viii, 180 p ; 22 cm. Subject(s) Pirates Caribbean Area Fiction. Format Fiction Author(s) Boullosa, Carmen. Title(s) They're cows, we're pigs / by Carmen Boullosa Place New York : Grove Press, Physical Descr viii, 180 p ; 22 cm. Subject(s) Pirates Caribbean Area Fiction. Format Fiction This is the metadata for this. While the card-catalog entry is a form of metadata, it does not address topics such as quality, accuracy, or scale. Well-written geospatial metadata describes these and many more aspects of the data.

Identification_Information Citation Citation_Information Originator: NOAA Coastal Services Center Publication_Date: Title: Hurricane Storm Surge Geospatial_Data_Presentation_Form: Map Publication_Information Publication_Place: Charleston, SC Publisher: NOAA Coastal Services Center Larger_Work_Citation Citation_Information Identification_Information Citation Citation_Information Originator: NOAA Coastal Services Center Publication_Date: Title: Hurricane Storm Surge Geospatial_Data_Presentation_Form: Map Publication_Information Publication_Place: Charleston, SC Publisher: NOAA Coastal Services Center Larger_Work_Citation Citation_Information This is the metadata for this.

We often use metadata without even knowing it.

Metadata contains vital information. Imagine, if you will… You are given two identical cans without labels. One contains cat food, the other contains tuna (dolphin-safe, of course). You must choose one these cans, and then eat the contents.

Entity Attributes A Common, Everyday Example

Other Examples? What is Metadata?

Metadata as a Component of Data

Properly documented data provides vital information to interested parties. A Component of Data

Metadata is that component of data which describes it. Environmental Sensitivity Index Data Metadata RARNUM - unique combination of species, concentration, and seasonality CONC (concentration) = Density species is found at location Season_ID = seasonality code linked to the seasonal table Element - Biology group A Component of Data

It’s data about a data set. Title Scale Source Content Location Publication Access Title Scale Source Content Location Publication Access MetadataMetadata GIS files Imagery Geospatial databases GPS data GIS files Imagery Geospatial databases GPS data Data set A Component of Data

Metadata describes… CONTENT CONDITION QUALITY Characteristics of the data Characteristics of the data A Component of Data

Because metadata provides vital information about a dataset, it should never be viewed or treated as a separate entity.

Metadata Non-spatial or attributes Spatial Take Home Message A Component of Data Metadata is a critical and integral component of any complete data set. Metadata is a critical and integral component of any complete data set.

The Value of Metadata

The Value of Metadata The Current Concept Primary external value Discovery Assessment Access Use

The Value of Metadata The Current Concept Primary internal value “Inheritance” “Properly documenting a data set is the key to preserving its usefulness through time.”

The Value of Metadata An Emerging Concept A data management tool Internal value Discovery Assessment Access Use

Benefits as a data management tool The Value of Metadata Data Currency Date of last edit/update Age of source files Data Utility Track source file usage Track distribution frequency

The Value of Metadata Monitoring Data Development Data processing steps Status of development Estimate Development Costs Data processing – time and extent Source file availability Benefits as a data management tool

The Value of Metadata Obstacles to metadata production Examples include: Metadata standards are too extensive and difficult to implement. Metadata production requires time and other resources. Few immediate and tangible benefits, and few incentives to produce metadata.

The Value of Metadata To realize the full potential of metadata under this new concept, metadata creation must become integral to the data development process. The question is “How?” Make metadata part of the process

Approach metadata development from a business perspective. Sell it! The Value of Metadata Preserves data investment Limits liability Helps manage data resources Aids in external data acquisition Facilitates data access and transfer Provides for efficient data distribution

The Value of Metadata Build administrative support Garner administrative support by stressing the organizational benefits Data archive Data assessment Data management Data discovery Data transfer Data distribution

Stress the individual benefits of metadata Build technical support The Value of Metadata Reduces workload over the long term Field fewer data inquiries Provides a means of documenting personal contributions Facilitates sharing of reliable information

Develop strong staff support The Value of Metadata Incorporate metadata expectations into job descriptions and performance standards Build technical support Provide staff development opportunities The three “T’s”  Training  Tools  Time

Develop templates to facilitate efficient and consistent metadata creation Build organizational support The Value of Metadata Identify pertinent fields within the metadata structure Populate fixed fields  Use standardized language  Define distribution methods  Cite standards used Build source and contact libraries

Map metadata fields to the work flow Establish and assign responsibilities Distribute the effort The Value of Metadata  Technicians - lineage  Analysts – process and methodology  Field Scientists – accuracy assessments  I.T. Managers – tools, automated collection methods, information management

Mandate the use of standards and templates. Develop boilerplate metadata deliverable language for data contractors. Require publication of metadata. Create and publish a metadata SOP to document policies and procedures. Establish standard policies The Value of Metadata

Standardized Metadata

Why Have a Standard? Think for a moment how hard it would be to… … bake a cake without standard units of measurement. … put gas into your car without standard nozzle sizes. … plug a lamp into a socket without standard electrical outlets.

Why Have a Standard? The standard for metadata ensures a level of consistency in data documentation. Standards ensure consistency.

Establishing a Standard The Federal Geographic Data Committee (FGDC) was organized in 1990 under the Office of Management and Budget to promote the coordinated use, sharing, and dissemination of geospatial data on a national basis. The FGDC was tasked with creating a metadata standard to meet these objectives. The Metadata Standard

"... each agency shall document all new geospatial data it collects or produces, either directly or indirectly, using the standard under development by the FGDC, and make that standardized documentation electronically accessible to the Clearinghouse network." The Content Standard for Digital Geospatial Metadata (CSDGM) Executive Order 12906, 1994 Establishing a Standard

This “Content Standard” serves as a uniform summary description of the data set. Establishing a Standard

The Content Standard utilizes... Common terms Common definitions Common language Common structure Access constraints Citation currentness entity attribute domain lineage Process step Establishing a Standard

The Content Standard… Establishes names of compound elements and data elements. Defines the information about the values that are to be provided for the data elements. Type=“free text” Domain=Real currentness entity attribute Citation lineage Establishing a Standard

The Content Standard helps the user determine... If a set of geospatial data is available and fit for a particular use. How to access and transfer the data set. Establishing a Standard

Who Who collected the data? Who processed the data? Who wrote the metadata? Who to contact for questions? Who to contact to order? Who owns the data? Where Where were the data collected? Where were the data processed? Where are the data located? What What are the data about? What project were they collected under? What are the constraints on their use? What is the quality? What are appropriate uses? What parameters were measured? What format are the data in? When When were the data collected? When were the data processed? How How were the data collected? How were the data processed? How do I access the data? How do I order the data? How much do the data cost? How was the quality assessed? Why Why were the data collected? Metadata written using the Content Standard answers these important questions: Establishing a Standard

The FGDC Workbook

FGDC’s Metadata Workbook Defines the 334 metadata elements.

What do I use “The Workbook” for? It is the definitive resource for applying the FGDC Content Standard. However, it does not define the production rules. It describes element domain values, which are valid values that can be assigned to the data element. It provides section and element definitions.

It is a quick reference for production rules and structure. Use the “Graphical Representation” for quick access. You will still need to use the workbook to find the definition of a particular element and its domain.

Playing the Metadata Game

The Content Standard is organized using numbered chapters called “sections.” There are 7 main sections and 3 supporting sections. Each section is organized into series of elements that define the information content for metadata to document a set of digital geospatial data. Organization of the Content Standard

Data Quality Information Spatial Data Organization Information Spatial Reference Information Entity and Attribute Information Metadata The Three Supporting Sections 9 Time Period Information 10 Contact Information 8 Citation Information Distribution Information Metadata Reference Information Identification Information Organization of the Content Standard The Seven Main Sections

Organization of the Content Standard Each section begins with the name and definition of the section. These are followed by the component elements of the section. Each section provides the names and definitions of its component elements, information about the types of values that can be provided for the elements, and information about the elements that are mandatory or repeatable.

Interpreting the Graphical Production Rules The workbook uses graphics to illustrate the production rules of the standard. These graphics include most of the information provided by the production rules, including: How elements are grouped. What is mandatory and what is not. What elements can repeat and how many times they can repeat.

Interpreting the Graphical Production Rules Section Sections are depicted by this symbol. Compound Element Compound elements are depicted using a 2-dimensional box. Data Element Data elements are depicted using a 3-dimensional box with shadow.

Interpreting the Graphical Production Rules Data Element A data element is a logically primitive item of data. Data elements are the things that you “fill in.” The form for the definition of a data element is: Data element name -- definition. Type: (choice of “integer”, “real”, “text”, “date”, or “time”) Domain: (describes valid values that can be assigned) An example of the definition of a data element is: Abstract -- a brief narrative summary of the data set. Type: text Domain: free text Note: Data element definitions are contained in the text of the Content Standard, not in the graphical production rules.

Compound Element 1 Compound Element 1.1 Data Element Data Element Data Element 1.2 How Elements Are Grouped Compound elements are composed of other compound and data elements. The composition is represented by nested boxes. Compound Element 1 is composed of Compound Element 1.1 and Data Element 1.2. Compound Element 1.1 is composed of Data Element and Data Element

Mandatory - must be provided. Meaning Data Element Compound Element What’s Mandatory? What’s Not? Mandatory if Applicable - must be provided if the data set exhibits the defined characteristic. Optional - provided at the discretion of the data set producer.

If an element can be repeated independently from other elements, it will be indicated as such below the element name. Repeating Elements Compound Element 1 (can be repeated unlimited times) Compound Element 1.1 Data Element Data Element Data Element 1.2 This group of elements would repeat. Compound Element 1 Compound Element 1.1 Data Element Data Element Data Element 1.2

Using the Graphics to Make Decisions All elements are colored yellow, so all are mandatory and must be reported. Compound Element 1 Compound Element 1.1 Data Element Data Element Data Element 1.2

Compound Element 1 is mandatory. Compound Element 1.1 is optional. If yes, Data Elements and are mandatory. If no, do not report Compound Element 1.1, Data Element or 1.1.2, and skip to Data Element 1.2. Data Element 1.2 is mandatory. Compound Element 1 Data Element Data Element Data Element 1.2 Compound Element 1.1 Using the Graphics to Make Decisions

Compound Element 1 is mandatory. Compound Element 1.1 is mandatory. Data Element is mandatory. Data Element is mandatory if applicable. Data Element 1.2 is optional. Compound Element 1 Compound Element 1.1 Data Element Data Element Data Element 1.2 Using the Graphics to Make Decisions

Compound Element 1 is mandatory if applicable. If not applicable to the data set, do not report any elements. If applicable, it is mandatory and: Compound Element 1.1 is mandatory. Data Element is mandatory if applicable. If not applicable, do not report it. If applicable, it is mandatory. Data Element is mandatory. Data Element 1.2 is optional. Compound Element 1 Compound Element 1.1 Data Element Data Element Data Element 1.2 Using the Graphics to Make Decisions

Adapting the CSDGM Extending the Standard Extended elements are elements outside the Standard, but needed by the data set producer. These elements may be defined by the data set producer or the user community. The official requirements and rules for User Defined Extensions are documented in Appendix D of the FGDC CSDGM.

A profile is a document that describes the application of the Standard to a specific user community A profile always contains the Standard, plus modifications to the optionality or repeatability of non- mandatory elements in the Standard. Modifications to the domains of standard elements can also be made where permitted by the Standard. Profiles may also contain extended elements. Adapting the CSDGM Creating Profiles

Biological Data Profile of the CSDGM (Biological Data Working Group) Metadata Profile for Shoreline Data (Bathymetric Subcommittee) CSDGM: Extensions for Remote Sensing Metadata (Standards Working Group) FGDC Endorsed Profiles or Extensions of the CSDGM, version 2.0

Content Standard for Digital Geospatial Metadata (CSDGM - version 2.0) CSDGM, Part 1: Biological Data Profile Spatial Data Transfer Standard (SDTS), Part 5: Raster Profile and Extensions SDTS, Part 6: Point Profile SDTS Part 7: Computer-Aided Design and Drafting (CADD) Profile Cadastral Data Content Standard Vegetation Classification Standard Soils Geographic Data Standard Geospatial Positioning Accuracy Standard, Part 1 - Reporting Methodology Part 2 - Geodetic Control Networks Part 3 - National Standard for Spatial Data Accuracy Content Standard for Digital Orthoimagery, Content Standard for Remote Sensing Swath Data FGDC Endorsed Standards

Exercise 1 Using the Green Book

Writing Metadata

It’s not THAT bad! First records are the hardest. Not all fields may need to be filled in. Tools are available. Training classes can be taken. Can often be produced automatically. Can (and should) be reviewed for updates.

Before you begin writing, get organized. Writing Metadata

Document your data as you go. Writing Metadata

Write so others can understand. Writing Metadata

Always review your document. Writing Metadata

Items required Sense of Humor! Chocolate FGDC Workbook Metadata entry tool Coffee

Write simply but completely. Document for a general audience. Be consistent in style and terminology. Keep your readers in mind. Writing Metadata

Define all acronyms. Avoid using jargon. Clearly state data limitations. Writing Metadata Keep your readers in mind.

Write a complete title that includes: What Where When Scale Who Writing Metadata

The title is critical in helping others find your data. Which is better? Greater Yellowstone Rivers from 1:126,700 Forest Visitor Maps ( ) Writing Metadata

Be specific. Quantify when you can. Vague: We checked our work and it looks complete. Specific: We checked our work using 3 separate sets of check plots reviewed by 2 different people. We determined our work to be 95% complete based on these visual inspections. Writing Metadata

Select your key words wisely. Use unambiguous words. Use descriptive words. Fully qualify geographic locations. Writing Metadata

Have someone else read it. If you’re the only reviewer, put it away and read it again later. Check for clarity and omissions. Review your final product. Writing Metadata

Can a novice understand what you wrote? Are your data properly documented for posterity? When you review your work, ask: Writing Metadata

Does the documentation present all the information needed to use or reuse the data? Are any pieces missing? When you review your work, ask: Writing Metadata

Write so that others will understand. Writing Metadata

Finally... Remember, metadata is an integral component of your data, and can provide many benefits at various levels within an organization by making the metadata process more streamlined and efficient.

Michael Moeller

Writing metadata