1. SJSK 1 HDF-EOS Workshop II Metadata Requirements for EOSDIS Data Providers Siri Jodha Singh Khalsa khalsa@colorado.edu

2. SJSK 2 HDF-EOS Workshop II Topics Why metadata is important Types of metadata in HDF-EOS files Required metadata How metadata is encoded and delivered

3. SJSK 3 HDF-EOS Workshop II What is Metadata? Metadata is information that identifies and characterizes an information product. Sometimes called “data about data”

4. SJSK 4 HDF-EOS Workshop II Users Need Metadata Metadata is needed to answer questions such as: -What time and location does this data apply to? -Why type of instrument and processing produced the data? -What other inputs were used to generate the data? -What QA has been performed on this data? -Who do I contact if I have questions about this data?

5. SJSK 5 HDF-EOS Workshop II Metadata is Essential Large data archive systems cannot function without metadata. Metadata is used to keep track of such things as: -where the data is -what type of operations are possible on the data -whether there are any access restrictions on the data -how individual data files are logically grouped into “collections.”

6. SJSK 6 Key Concepts A granule is the smallest aggregation of data that is independently described and inventoried by the ECS. A granule consists of 1 or more physical files. A collection is a logical grouping of granules. The ECS Data Model allows for: -“Core” attributes -“Product-Specific” Attributes (PSAs)

7. SJSK 7 HDF-EOS Workshop II Types of Metadata Metadata in HDF files -stored as global text attributes Types of Metadata used in HDF-EOS files: -Structural Metadata -Core Metadata (inventory, can include PSAs) -Archive Metadata (non-searchable, product-specific) Collection level metadata -core and product-specific

8. SJSK 8 HDF-EOS Workshop II Required Metadata Origins of metadata requirements: -what is required to archive and retrieve files -what is required to provide search and other services on data -what is federally mandated (FGDC) There are 287 attributes in the ECS data model -only a subset are used for any given product -101 are applicable at the granule level

9. SJSK 9 HDF-EOS Workshop II Metadata Coverage Science Data that are delivered for archiving in ECS must meet what is called the Intermediate level of metadata coverage. This involves as few as: -31 collection level attributes -4 granule level attributes Compliance at this level is not enforced by the system.

10. SJSK 10 HDF-EOS Workshop II Collection-Level Metadata for Intermediate Coverage -ShortName -LongName -CollectionDescription -VersionID -ArchiveCenter -RevisionDate -VersionDescription -CollectionState -MaintenanceandUpdateFrequency -ECSDisciplineKeyword -ECSTopicKeyword -ECSTermKeyword -ECSVariableKeyword -ContactOrganizationName -Role -SpatialCoverageType -PointLatitude -PointLongitude -TimeType -DateType -TemporalRangeType -PrecisionofSeconds -EndsatPresentFlag -CalendarDate -TimeofDay -GuideName -DataCenter -DocumentVersion -DocumentUpdated -Title -DocumentCreated

11. SJSK 11 HDF-EOS Workshop II Granule-Level Metadata for Intermediate Coverage There are only four granule-level metadata attributes required: -ShortName -VersionID -SizeMBECSDataGranule -ProductionDateTime ShortName and VersionID are identical to the collection- level attributes with these names. For granules coming into ECS, SizeMBECSDataGranule and ProductionDateTime are supplied by the system upon insertion.

12. SJSK 12 HDF-EOS Workshop II How is Metadata Supplied? Collection-level metadata is carried in an Earth Science Data Type (ESDT) Descriptor file. Granule-level metadata is defined in the descriptor file and populated using a Metadata Configuration File (MCF). Granule-level metadata is delivered in the HDF- EOS granule *or* in a populated MCF accompanying a non-HDF granule. The DAAC where a collection will reside is responsible for descriptors and ingest routines.

13. SJSK 13 HDF-EOS Workshop II Metadata Work Flow for External Data Providers

14. SJSK 14 Metadata Resources on the Web ECS Metadata Homepage http://ecsinfo.hitc.com/metadata/metadata.html Metadata Works (ESDT Descriptor Tool) http://et3ws1.HITC.COM/metadata_works/ EOSDIS Information Architecture http://spsosun.gsfc.nasa.gov/InfoArch.html Federal Geographic Data Committee http://www.fgdc.gov/

15. SJSK 15 Q&A w/ Experts Panel Q: “If you are a new data provider, how do you get your data into an HDF-EOS granule, given the bewildering array of utilities and tools available? What is the simplest solution for this?” A: The recommended solution is to obtain the HCR package, which includes the HDF-EOS and HDF libraries. For populating the required metadata in the granule, obtain the Metadata/Time Toolkit_MDT. The steps would be: 1. Write an HCR and use the tools to turn this into a skeletal HDF-EOS granule. (This step is optional). 2. Use the HDF-EOS library to create a granule. (If starting with a skeletal HDF-EOS file generated from an HCR then plain HDF calls can be used to insert data into the granule ). 3. Use Toolkit_MDT calls to insert metadata into the granule. This requires generation of an MCF in ODL. Metadata_Works is available for doing this. As an alternative, a simple HDF call can be used to attach minimum metadata (in ODL) to an HDF file. Note: if the data are going to reside in a DAAC, or in an archive that must be interoperable with ECS, you will need to generate collection-level metadata. Metadata_Works is the recommended tool for this.