1. U.S. Department of the Interior U.S. Geological Survey Tutorials on Data Management Lesson 3: Describe (Metadata, Documentation) CC image by bonus on Flickr

2. Describe Metadata/Documentation Provided by DataONE  Definition of metadata  Examine information included in a metadata record  Examples of metadata standards and how to choose  Illustrate the value of metadata to data users, data providers, and organizations  Describe the utility of metadata for a variety of scenarios CC image by Alec Couros on Flickr Lesson Topics

3. Describe Metadata/Documentation Provided by DataONE After completing this lesson, the participant will be able to:  Define science metadata  Give examples of metadata that you are likely to encounter in the ‘real world’ (i.e., outside of a research context)  Identify and list the types of information typically included in metadata records for environmental datasets  Identify 3 reasons metadata is of value to data users, data developers, and organizations  List 3 uses for metadata, beyond discovery of data  Identify and describe factors that may determine which metadata standards are most appropriate for a given dataset Learning Objectives

4. Describe Metadata/Documentation Provided by DataONE The Data Life Cycle

5. Describe Metadata/Documentation Provided by DataONE CC image by Justin See on Flickr CC image by CIMMYT on Flickr CC image by acordova on Flickr CC image by kukkurovaca on Flickr CC image by SEDAC on Flickr CC image by ISAS on Flickr Data Collection

6. Describe Metadata/Documentation Provided by DataONE Average Temperature of Observation for Each Species SpeciesAverage Temperature Temperature Standard Deviation Number of Observations Minimum Temperature Maximum Temperature Northern Red-legged Frog 4.4---14.4 Tailed Frog7.03.03410 Arizona Toad10.0---110 Strecker's Chorus Frog 10.52.011916 Oregon Spotted Frog 11.015.52022 New Jersey Chorus Frog 11.54.517322 Wood Frog12.55.5897028.8 Spring Peeper13.25.656932 Red-legged Frog13.35.916427 From Field Notes to Datasets

7. Describe Metadata/Documentation Provided by DataONE CC image by Heather Kennedy on Flickr From Datasets to Published Papers

8. Describe Metadata/Documentation Provided by DataONE  Definition: A collection of data  Generally datasets can be defined as:  Spatial – a collection of logically related features arranged in a prescribed manner such as GIS map layers, water features, etc  Tabular – a file, spreadsheet, data in a table  Many tabular datasets are inherently “spatial”, e.g. water- quality samples associated with stream collection sites  Elements in a dataset can include:  Values, measures, points, coordinates, conditions, qualities, frequencies, or attributes that are a result of an observational study What is a Dataset?

9. Describe Metadata/Documentation Provided by DataONE provide  When you provide data to someone else, what types of information would you want to include with the data? receive  When you receive a dataset from an external source, what types of details do you want to know about the data? Working with Data

10. Describe Metadata/Documentation Provided by DataONE  Providing data:  Why were the data created?  What limitations, if any, do the data have?  What does the data mean?  How should the data be cited if it is re-used in a new study?  Receiving data:  What are the data gaps?  What processes were used for creating the data?  Are there any fees associated with the data?  In what scale were the data created?  What do the values in the tables mean?  What software do I need in order to read the data?  What projection are the data in?  Can I give these data to someone else? Working with Data

11. Describe Metadata/Documentation Provided by DataONE Need for Documentation  Without documentation, data becomes unusable  Decisions based on undocumented data are not defensible  Undocumented data cannot be reused  Institutional knowledge in data is not preserved  Undocumented data leads to redundant efforts  Wastes money Provided by Tom Chatfield, BLM

12. Describe Metadata/Documentation Provided by DataONE Metadata is: Data ‘reporting’  WHO created the data?  WHAT is the content of the data?  WHEN were the data created?  WHERE is it geographically?  HOW were the data developed?  WHY were the data developed? Photo by Michelle Chang. All Rights Reserved What is Metadata?

13. Describe Metadata/Documentation Provided by DataONE Benefits of Metadata  Protects USGS’s investments in data  Counters personnel changes  Preserves institutional memory  Enables data sharing with other agencies  Reduces redundant data collection  Limits potential liability  Saves time and money  Facilitates understanding of the data  Enables discovery of available data within and outside USGS  Publicizes data to public Provided by Tom Chatfield, BLM

14. Describe Metadata/Documentation Provided by DataONE Mandates for Metadata  OMB Circular A-16  Bureau’s 1283-1 Data Administration and Management Handbook - All data requires documentation with metadata that is stored in an accessible repository  Geospatial:  Executive Order 12906 (April 1994) sec. 3(b) requires collection of metadata for new and existing GeoSpatial data collections and “make that standardized documentation electronically accessible to the Clearinghouse network”  The Federal Geographic Data Committee (FGDC) has developed a standard for metadata collection that all Federal agencies (including USGS) are required to follow Provided by Tom Chatfield, BLM

15. Describe Metadata/Documentation Provided by DataONE  Metadata is all around… Author(s) Boullosa, Carmen. Title(s) They're cows, we're pigs / by Carmen Boullosa Place New York : Grove Press, 1997. Physical Descr viii, 180 p ; 22 cm. Subject(s) Pirates Caribbean Area Fiction. Format Fiction Author(s) Boullosa, Carmen. Title(s) They're cows, we're pigs / by Carmen Boullosa Place New York : Grove Press, 1997. Physical Descr viii, 180 p ; 22 cm. Subject(s) Pirates Caribbean Area Fiction. Format Fiction CC image by USDAgov on Flickr CC image by Mskadu on Flickr Metadata in Real Life

16. Describe Metadata/Documentation Provided by DataONE  A Standard provides a structure to describe data with:  Common terms to allow consistency between records  Common definitions for easier interpretation  Common language for ease of communication  Common structure to quickly locate information  In search and retrieval, standards provide:  Documentation structure in a reliable and predictable format for computer interpretation  A uniform summary description of the dataset CC image by ccarlstead on Flickr What is a Metadata Standard?

17. Describe Metadata/Documentation Provided by DataONE CC image by I like on Flickr What Does a Metadata Record Look Like?

18. Describe Metadata/Documentation Provided by DataONE Metadata Content Sections  Identification  Data Quality  Spatial Data Organization  Spatial Reference  Entity and Attribute  Distribution  Metadata Reference Provided by Tom Chatfield, BLM

19. Describe Metadata/Documentation Provided by DataONE Identification  Citation  Description  Time Period of Content  Status  Spatial domain  Keywords  Access & Use constraints  Point of Contact  Browse Graphic  Data Set Credit  Security  Native Data Set Environment  Cross Reference Provided by Tom Chatfield, BLM

20. Describe Metadata/Documentation Provided by DataONE Data Quality  Attribute Accuracy  Logical Consistency Report  Completeness Report  Positional Accuracy  Lineage  Cloud Cover (optional) Provided by Tom Chatfield, BLM

21. Describe Metadata/Documentation Provided by DataONE Spatial Data Organization & Spatial Reference  Indirect Spatial Reference  Direct Spatial Reference Method  Point & Vector Object Info or  Raster Object Info  Horizontal/Vertical Coordinate  System Definition Provided by Tom Chatfield, BLM

22. Describe Metadata/Documentation Provided by DataONE Entity and Attribute  Entity/Attribute Name  Entity/Attribute Type  Entity/Attribute Description Provided by Tom Chatfield, BLM

23. Describe Metadata/Documentation Provided by DataONE Distribution  Distributor  Resource Description  Distribution Liability  Ordering process  Technical Prerequisites  Availability schedule Provided by Tom Chatfield, BLM

24. Describe Metadata/Documentation Provided by DataONE Metadata Reference  Date  Review Date  Contact  Standard Name & Version  Access & Use Constraints  Security  Online Linkage Provided by Tom Chatfield, BLM

25. Describe Metadata/Documentation Provided by DataONE Data developers Data users Organizations Metadata helps… The Value of Metadata

26. Describe Metadata/Documentation Provided by DataONE Even if the value of data documentation is recognized, concerns remain as to the effort required to create metadata that effectively describe the data. CC image by waterlilysage on Flickr Concerns About Creating Metadata

27. Describe Metadata/Documentation Provided by DataONE ConcernSolution workload required to capture accurate robust metadata incorporate metadata creation into data development process – distribute the effort time and resources to create, manage, and maintain metadata include in grant budget and schedule readability / usability of metadata use a standardized metadata format discipline specific information and ontologies ‘profile’ standard to require specific information and use specific values Concerns About Creating Metadata

28. Describe Metadata/Documentation Provided by DataONE  Metadata allows data developers to:  Avoid data duplication  Share reliable information  Publicize efforts – promote the work of a scientist and his/her contributions to a field of study CC image by US Embassy Guyana on Flickr What is the Value to Data Developers?

29. Describe Metadata/Documentation Provided by DataONE  Metadata gives a user the ability to:  Search, retrieve, and evaluate data set information from both inside and outside an organization  Find data: Determine what data exists for a geographic location and/or topic  Determine applicability: Decide if a data set meets a particular need  Discover how to acquire the dataset you identified; process and use the dataset CC image by ASEE on Flickr What is the Value to Data Users?

30. Describe Metadata/Documentation Provided by DataONE  Metadata helps ensure an organization’s investment in data:  Documentation of data processing steps, quality control, definitions, data uses, and restrictions  Ability to use data after initial intended purpose  Transcends people and time:  Offers data permanence  Creates institutional memory  Advertises an organization’s research:  Creates possible new partnerships and collaborations through data sharing CC image by mambol on Flickr What is the Value to Organizations?

31. Describe Metadata/Documentation Provided by DataONE Information Entropy DATA DETAILS Time of data development Specific details about problems with individual items or specific dates are lost relatively rapidly General details about datasets are lost through time Accident or technology change may make data unusable Retirement or career change makes access to “mental storage” difficult or unlikely Loss of data developer leads to loss of remaining information TIME (From Michener et al 1997)

32. Describe Metadata/Documentation Provided by DataONE TIME DATA DETAILS Sound information management, including metadata development, can arrest the loss of dataset detail. Information Entropy

33. Describe Metadata/Documentation Provided by DataONE  Metadata can support:  data distribution  data management  project management  If it is:  considered a component of the data  created during data development  populated with rich content derive classify collect planimetricimagery analysis alternative committee review PLAN charette meta A Closer Look: The Utility of Metadata

34. Describe Metadata/Documentation Provided by DataONE metadata publication data portals data discovery Data Distribution via Metadata

35. Describe Metadata/Documentation Provided by DataONE  The descriptive content of the metadata file can be used to identify, assess, and access available data resources. online access order process contacts use constraints access constraints data quality availability/pricing keywords geographic location time period attributes Distribution: Data Discovery

36. Describe Metadata/Documentation Provided by DataONE  A metadata collection can be published to the internet via:  website catalog  web accessible folder (WAF)  Z39.50 metadata clearinghouse  metadata service  geospatial data portal Internet Metadata CollectionUser Query Internet / Intranet Dataset Distribution: Metadata Publication

37. Describe Metadata/Documentation Provided by DataONE  Examples of metadata search portals:  Data.gov  Federal e-gov geospatial data portal  http://www.geo.data.gov http://www.geo.data.gov  Metacat  Repository for data and metadata  http://knb.ecoinformatics.org/index.jsp http://knb.ecoinformatics.org/index.jsp  US Geological Survey  USGS Core Science Metadata Clearinghouse:  http://mercury.ornl.gov/clearinghouse http://mercury.ornl.gov/clearinghouse  ArcGIS Online  ESRI sponsored national geospatial data portal  http://www.geographynetwork.com http://www.geographynetwork.com CC image by RGB12 on Flickr Distribution: Geospatial Data Portals

38. Describe Metadata/Documentation Provided by DataONE Example: USGS Core Science Metadata Clearinghouse

39. Describe Metadata/Documentation Provided by DataONE Data Accountability Discovery & Re-use Maintenance & Update Data Liability Data Management via Metadata

40. Describe Metadata/Documentation Provided by DataONE  Metadata records can used to track data provenance accuracy  Data Maintenance:  Are the data current?  Do we have data older than ten years?  was before some political or geophysical event that resulted in significant change?  Are the data valid?  prior to most current source data  prior to most current methodologies  Data Update:  Contact information  Distribution policies, availability, pricing, URLs  New derivations of the dataset Management: Maintenance and Update

41. Describe Metadata/Documentation Provided by DataONE If you create metadata, other people can discover your data If you create metadata, you can find your own data CC image by Oceanit Daily Photo on Flickr Discovery: Data Reuse

42. Describe Metadata/Documentation Provided by DataONE  Find your data by:  themes / attributes  geographic location time ranges  analytical methods used  sources and contributors  data quality Discoverable data is usable data! CC image by NASA Goddard Spece Flight Center on Flickr Management: Data Discovery & Reuse

43. Describe Metadata/Documentation Provided by DataONE  Metadata allows you to repeat scientific process if:  methodologies are defined  variables are defined  analytical parameters are defined  Metadata allows you to defend your scientific process:  demonstrate process  increasingly GIS-savvy public requires metadata for consumer information INPUT RESULTS Management: Data Accountability

44. Describe Metadata/Documentation Provided by DataONE  Metadata is an exercise in data accountability. It requires you to assess:  What do you know about the dataset?  What don’t you know about the dataset?  What should you know about the dataset? Are you willing to associate yourself with the metadata record ? Management: Data Accountability

45. Describe Metadata/Documentation Provided by DataONE  Metadata is a declaration of:  Purpose  the originator’s intended application of the data  Use Constraints  inappropriate applications of the data  Completeness  features or geographies excluded from the data  Distribution Liability  explicit liability of the data producer and assumed liability of the consumer What to do… What not to do… Management: Data Liability

46. Describe Metadata/Documentation Provided by DataONE Project Monitoring Project Coordination Project Planning Contract Deliverables Project Management via Metadata

47. Describe Metadata/Documentation Provided by DataONE  Metadata records can serve as a project design document:  descriptions & intent of project  geographic and temporal extent of project  source data of project  attribute requirements of project  Benefits:  expectations are clearly outlined  metadata is integrated into the process  provides a medium to record progress Project Management: Project Planning

48. Describe Metadata/Documentation Provided by DataONE  Use metadata to monitor:  data development status  QA/QC assessments  needed changes in approach Monitoring requires that the metadata be actively maintained and reviewed! time milestones Project Management: Project Monitoring

49. Describe Metadata/Documentation Provided by DataONE  Metadata can be a means to improve communications among project participants using common:  descriptions & parameters  keywords, vocabularies, thesauri  contact information  attributes  distribution information  If reviewed regularly by all participants, metadata created early and updated during the project improves opportunity for coordinating:  source data  analytical methods  new information Project Management: Project Coordination

50. Describe Metadata/Documentation Provided by DataONE  As a key component of the data, metadata should be part of any data deliverable  For quality metadata from a deliverable, the record should provide:  Citation information  Data quality information  Accurate geospatial information  Clearly defined entities and attributes  Distribution information Project Management: Contract Deliverables

51. Describe Metadata/Documentation Provided by DataONE Image courtesy of Viv Hutchinson Choosing Metadata Standards

52. Describe Metadata/Documentation Provided by DataONE  Dublin Core Element Set  Emphasis on web resources, publications  http://dublincore.org/documents/dces/  FGDC Content Standard for Digital Geospatial Metadata (CSDGM)  Emphasis on geospatial data  Biological Data Profile (BDP) of the CSDGM  Profile to the CSDGM emphasis on biological data (and geospatial)  http://www.fgdc.gov/metadata/geospatial-metadata-standards  ISO 19115/19139 Geographic information: Metadata  Emphasis on geospatial data and services  http://www.fgdc.gov/metadata/geospatial-metadata- standards#fgdcendorsedisostandards Multiple Metadata Standards Exist: Examples

53. Describe Metadata/Documentation Provided by DataONE  Ecological Metadata Language (EML)  Focus on ecological data  http://knb.ecoinformatics.org/eml_metadata_guide.ht ml  Darwin Core  Emphasis on museum specimens  http://rs.tdwg.org/dwc/index.htm  Geography Markup Language (GML)  Emphasis on geographic features (roads, highways, bridges)  http://www.opengeospatial.org/standards/gml Multiple Metadata Standards Exist: Examples

54. Describe Metadata/Documentation Provided by DataONE EMLFGDC Title Abstract Entity DescriptionEntity Type Definition Intellectual RightsUse Constraints Comparing Metadata Standards

55. Describe Metadata/Documentation Provided by DataONE  Many standards collect similar information  Factors to consider:  Your data type:  Are you working mainly with GIS data? Rastor/vector or point data? Do you have biological or shoreline information in your dataset? - Consider the FGDC Content Standard for Digital Geospatial Metadata with one of its profiles: the Biological Data Profile or the Shoreline Data Profile.  Are you working with data retrieved from instruments such as monitoring stations or satellites? Are you using geospatial data services such as applications for web-mapping applications or data modeling? - If so, then consider using the ISO 19115-2 standard  Are you mainly working with ecological data? - Consider Ecological Metadata Language (EML) Choosing a Metadata Standard

56. Describe Metadata/Documentation Provided by DataONE  More Factors to consider:  Your organization’s policies: do they state which standard to use?  What resources are available to create metadata? Examples of Tools:  FGDC CSDGM: - Mermaid (NOAA) http://www.ncddc.noaa.gov/metadata- standards/mermaid/ - Metavist (Forest Service) http://ncrs.fs.fed.us/pubs/viewpub.asp?key=2737 - TKME (USGS) http://geology.usgs.gov/tools/metadata/tools/doc/tkme.html  EML: - Morpho (http://knb.ecoinformatics.org/morphoportal.jsp)  ISO: (http://www.fgdc.gov/metadata/iso-metadata-editor-review) - XML Spy or Oxegyn - CatMD  Other factors: Availability of human support; instructional materials; use of controlled vocabularies; output formats Choosing a Metadata Standard "Any use of trade, product, or firm names is for descriptive purposes only and does not imply endorsement by the U.S. Government.”

57. Describe Metadata/Documentation Provided by DataONE  Metadata is documentation of data  A metadata record captures critical information about the content of a dataset  Metadata allows data to be discovered, accessed, and re-used  A metadata standard provides structure and consistency to data documentation  Standards and tools vary – select according to defined criteria such as data type, organizational guidance, and available resources  Metadata is of critical importance to data developers, data users, and organizations  Metadata can be effectively used for:  data distribution  data management  project management  Metadata completes a dataset. Creating robust metadata is in your OWN best interest! Summary

58. Describe Metadata/Documentation Provided by DataONE Resources 1. Chatfield, T., Selbach, R. February, 2011. Data Management for Data Stewards. Data Management Training Workshop. Bureau of Land Management (BLM). 2. DataONE education modules Accessed August, 2012 at https://www.dataone.org/education-modules