Metadata for your Digital Collections Jenn Riley Metadata Librarian IU Digital Library Program

3/6/07 INCOLSA Workshop2 Many definitions of metadata “Data about data” “Structured information about an information resource of any media type or format.” (Caplan) “Any data used to aid the identification, description and location of networked electronic resources.” (IFLA) …

3/6/07 INCOLSA Workshop3 Refining a definition Other characteristics Structure Control Origin Machine-generated Human-generated In practice, the term often covers data and meta-metadata

3/6/07 INCOLSA Workshop4 Some uses of metadata By information specialists Describing non-traditional materials Cataloging Web sites Navigating digital objects Managing digital objects over the long term Managing corporate assets By novices Preparing Web sites for search engines Describing Eprints Managing personal CD collections

3/6/07 INCOLSA Workshop5 Metadata and cataloging Depends on what you mean by: metadata, and cataloging! But, in general: Metadata is broader in scope than cataloging Much metadata creation takes place outside of libraries Good metadata practitioners use fundamental cataloging principles in non-MARC environments Metadata created for many different types of materials Metadata is NOT only for Internet resources!

3/6/07 INCOLSA Workshop6 Metadata in digital library projects Searching Browsing Display for users Interoperability Management of digital objects Preservation Navigation

3/6/07 INCOLSA Workshop7 Some types of metadata TypeUse Descriptive metadataSearching Browsing Display Interoperability Technical metadataInteroperability Digital object management Preservation Preservation metadataInteroperability Preservation Rights metadataInteroperability Digital object management Structural metadataNavigation

3/6/07 INCOLSA Workshop8 How metadata is used

3/6/07 INCOLSA Workshop9 Creating descriptive metadata Digital library content management systems ContentDM ExLibris Digitool Greenstone Library catalogs Spreadsheets & databases XML

3/6/07 INCOLSA Workshop10 Creating other types of metadata Technical Stored in content management system Stored in separate Excel spreadsheet Structural Created and stored in content management system METS XML GIS Using specialized software Content markup In XML

3/6/07 INCOLSA Workshop11 Descriptive metadata Purpose Description Discovery Some common general schemas Dublin Core (unqualified and qualified) MARC MARCXML MODS LOTS of domain-specific schemas

3/6/07 INCOLSA Workshop12 Simple Dublin Core (DC) 15-element set National and international standard 2001: Released as ANSI/NISO Z39.85ANSI/NISO Z : Released as ISO 15836ISO Maintained by the Dublin Core Metadata Initiative (DCMI) Other players DC Usage Board DCMI Communities DCMI Task Groups

3/6/07 INCOLSA Workshop13 DCMI mission The mission of DCMI is to make it easier to find resources using the Internet through the following activities: Developing metadata standards for discovery across domains, Defining frameworks for the interoperation of metadata sets, and, Facilitating the development of community- or disciplinary-specific metadata sets that are consistent with items 1 and 2

3/6/07 INCOLSA Workshop14 DC Principles Original principles “Core” across all knowledge domains No element required All elements repeatable 1:1 principle DC Abstract Model “A reference against which particular DC encoding guidelines can be compared” model Two schools of thought on its development Clarifies model underlying the metadata standard Overly complicates a standard intended to be simple

3/6/07 INCOLSA Workshop15 Content/value standards for DC None required Some elements recommend a content or value standard as a best practice Relation Source Subject Type Coverage Date Format Language Identifier Coverage Date Format Language Identifier

3/6/07 INCOLSA Workshop16 Some limitations of DC Can’t indicate a main title vs. other subordinate titles No method for specifying creator roles W3CDTF format can’t indicate date ranges or uncertainty Can’t by itself provide robust record relationships

3/6/07 INCOLSA Workshop17 Good times to use DC Cross-collection searching Cross-domain discovery Metadata sharing Describing some types of simple resources Metadata creation by novices

DC [record]record QDC [record]record [collection]collection MARC [record]record [collection]collection MARCXML [record]record MODS [record]record [collection]collection Record format XML RDF (X)HTML Field labelsText Reliance on AACR None Common method of creation By novices, by specialists, and by derivation

3/6/07 INCOLSA Workshop19 Qualified Dublin Core (QDC) Adds some increased specificity to Unqualified Dublin Core Same governance structure as DC Same encodings as DC Same content/value standards as DC Listed in DMCI TermsDMCI Terms Additional principles Extensibility Dumb-down principle

3/6/07 INCOLSA Workshop20 Types of DC qualifiers Additional elements Element refinements Encoding schemes Vocabulary encoding schemes Syntax encoding schemes

3/6/07 INCOLSA Workshop21 DC qualifier status Recommended Conforming Obsolete Registered

3/6/07 INCOLSA Workshop22 Limitations of QDC Widely misunderstood No method for specifying creator roles W3CDTF format can’t indicate date ranges or uncertainty Split across 3 XML schemas No encoding in XML (yet) officially endorsed by DCMI

3/6/07 INCOLSA Workshop23 Best times to use QDC More specificity needed than simple DC, but not a fundamentally different approach to description Want to share DC with others, but need a few extensions for your local environment Describing some types of simple resources Metadata creation by novices

DC [record]record QDC [record]record [collection]collection MARC [record]record [collection]collection MARCXML [record]record MODS [record]record [collection]collection Record format XML RDF (X)HTML XML RDF (X)HTML Field labelsText Reliance on AACR None Common method of creation By novices, by specialists, and by derivation

3/6/07 INCOLSA Workshop25 MAchine Readable Cataloging (MARC) Format for the records in library catalogs Used for library metadata since 1960s Adopted as national standard in 1971 Adopted as international standard in 1973 Maintained by: Network Development and MARC Standards Office at the Library of Congress Standards and the Support Office at the National Library of Canada

3/6/07 INCOLSA Workshop26 More about MARC Actually a family of MARC standards throughout the world U.S. & Canada use MARC21 Structured as a binary interchange format ANSI/NISO Z39.2 ISO 2709 Field names Numeric fields Alphabetic subfields

3/6/07 INCOLSA Workshop27 Content/value standards for MARC None required by the format itself But US record creation practice relies heavily on: AACR2r ISBD LCNAF LCSH

3/6/07 INCOLSA Workshop28 Limitations of MARC Use of all its potential is time-consuming OPACs don’t make full use of all possible data OPACs virtually the only systems to use MARC data Requires highly-trained staff to create Local practice differs greatly

3/6/07 INCOLSA Workshop29 Good times to use MARC Integration with other records in OPAC Resources are like those traditionally found in library catalogs Maximum compatibility with other libraries is needed Have expert catalogers for metadata creation

DC [record]record QDC [record]record [collection]collection MARC [record]record [collection]collection MARCXML [record]record MODS [record]record [collection]collection Record format XML RDF (X)HTML XML RDF (X)HTML ISO 2709 [ANSI Z39.2] Field labelsText Numeric Reliance on AACR None Strong Common method of creation By novices, by specialists, and by derivation By specialists

3/6/07 INCOLSA Workshop31 MARC in XML (MARCXML) Copies the exact structure of MARC21 in an XML syntax Numeric fields Alphabetic subfields Implicit assumption that content/value standards are the same as in MARC

3/6/07 INCOLSA Workshop32 Limitations of MARCXML Not appropriate for direct data entry Extremely verbose syntax Full content validation requires tools external to XML Schema conformance

3/6/07 INCOLSA Workshop33 Best times to use MARCXML As a transition format between a MARC record and another XML-encoded metadata format Materials lend themselves to library-type description Need more robustness than DC offers Want XML representation to store within larger digital object but need lossless conversion to MARC

DC [record]record QDC [record]record [collection]collection MARC [record]record [collection]collection MARCXML [record]record MODS [record]record [collection]collection Record format XML RDF (X)HTML XML RDF (X)HTML ISO 2709 [ANSI Z39.2] XML Field labelsText Numeric Reliance on AACR None Strong Common method of creation By novices, by specialists, and by derivation By specialists By derivation

3/6/07 INCOLSA Workshop35 Metadata Object Description Schema (MODS) Developed and managed by the Library of Congress Network Development and MARC Standards Office For encoding bibliographic information Influenced by MARC, but not equivalent Usable for any format of materials First released for trial use June 2002 MODS 3.2 released late 2006

3/6/07 INCOLSA Workshop36 MODS differences from MARC MODS is “MARC-like” but intended to be simpler Textual tag names Encoded in XML Some specific changes Some regrouping of elements Removes some elements Adds some elements

3/6/07 INCOLSA Workshop37 Content/value standards for MODS Many elements indicate a given content/value standard should be used Generally follows MARC/AACR2/ISBD conventions But not all enforced by the MODS XML schema Authority attribute available on many elements

3/6/07 INCOLSA Workshop38 Limitations of MODS No lossless round-trip conversion from and to MARC Still largely implemented by library community only Some semantics of MARC lost

3/6/07 INCOLSA Workshop39 Good times to use MODS Materials lend themselves to library-type description Want to reach both library and non-library audiences Need more robustness than DC offers Want XML representation to store within larger digital object

DC [record]record QDC [record]record [collection]collection MARC [record]record [collection]collection MARCXML [record]record MODS [record]record [collection]collection Record format XML RDF (X)HTML XML RDF (X)HTML ISO 2709 [ANSI Z39.2] XML Field labelsText Numeric Text Reliance on AACR None Strong Implied Common method of creation By novices, by specialists, and by derivation By specialists By derivation By specialists and by derivation

3/6/07 INCOLSA Workshop41 Visual Resources Association (VRA) Core From Visual Resources Association Separates Work from Image Library focus Inspiration from Dublin Core Version 3.0 released on 2002 Version 4.0 currently in Beta

3/6/07 INCOLSA Workshop42 Categories for the Description of Works of Art (CDWA) Lite Reduced version of the Categories for the Description of Works of Art (512 categories) From J. Paul Getty Trust Museum focus Conceived for record sharing

3/6/07 INCOLSA Workshop43 Structure standards for learning materials Gateway to Educational Materials (GEM) From the U.S. Department of Education Based on Qualified Dublin Core Adds elements for instructional level, instructional method, etc. “GEM's goal is to improve the organization and accessibility of the substantial collections of materials that are already available on various federal, state, university, non-profit, and commercial Internet sites.”* IEEE Learning Object Metadata (LOM) Elements for technical and descriptive metadata about learning resources * From

3/6/07 INCOLSA Workshop44 Text Encoding Initiative (TEI) TEI in Libraries For encoding full texts of documents Literary texts Letters …etc. Requires specialized search engine Delivery requires specialized software or offline conversion to HTML

3/6/07 INCOLSA Workshop45 Encoded Archival Description (EAD) Maintained by the Society for American Archivists EAD Working Group Markup language for archival finding aids Designed to accommodate multi-level description Requires specialized search engine Delivery requires specialized software or offline conversion to HTML EAD 1.0 released in 1998 EAD2002 finalized in December 2002

3/6/07 INCOLSA Workshop46 Levels of control Data structure standards (e.g., MARC) Data content standards (e.g., AACR2r) Encoding schemes Vocabulary Syntax High-level models (e.g., FRBR) Very few metadata standards include a counterpart to the AACR “chief source of information”

3/6/07 INCOLSA Workshop47 Some data content standards Anglo-American Cataloging Rules, 2nd edition (AACR2) Scheduled to be replaced by RDA in 2009 Describing Archives: A Content Standard (DACS) Replaces APPM Cataloging Cultural Objects (CCO) First content standard explicitly designed for these materials

3/6/07 INCOLSA Workshop48 When there’s no data content standard…

3/6/07 INCOLSA Workshop49 Vocabulary encoding schemes TGM I TGM II TGN GeoNet AAT LCSH LCNAF DCMI Type MIME Types …etc.

3/6/07 INCOLSA Workshop50 Syntax encoding schemes ISO8601 W3CDTF URI AACR2r …etc.

3/6/07 INCOLSA Workshop51 Functional Requirements of Bibliographic Records (FRBR) model WORK EXPRESSION MANIFESTATION ITEM is realized through is embodied in is exemplified by

3/6/07 INCOLSA Workshop52 Using FRBR principles in metadata creation Don’t need to take the model literally For unique materials, much simplification is possible Make sure you know how your practices conform to the high-level model Be consistent in these practices

3/6/07 INCOLSA Workshop53 How do I pick standards? (1) Institution Nature of holding institution Resources available for metadata creation What others in the community are doing Capabilities of your delivery software The standard Purpose Structure Context History

3/6/07 INCOLSA Workshop54 How do I pick standards? (2) Materials Genre Format Likely audiences What metadata already exists for these materials Project goals Robustness needed for the given materials and users Describing multiple versions Mechanisms for providing relationships between records Plan for interoperability, including repeatability of elements More information on handouthandout

3/6/07 INCOLSA Workshop55 Assessing materials for ease of metadata creation Number of items? Homogeneity of items? Foreign language? Published or unpublished? Specialist needed? How much information is known? Any existing metadata?

3/6/07 INCOLSA Workshop56 Assessing currently existing metadata Machine-readable? Divided into fields? What format? What content standards? Complete?

3/6/07 INCOLSA Workshop57 Assessing software capabilities Are there templates for standard metadata formats? Can you add/remove fields to a template? Can you create new templates? Can you add additional clarifying information without creating a separate field? Personal vs. corporate names Subject vocabulary used Is there an XML export? Does it produce valid records?

3/6/07 INCOLSA Workshop58 Case studies in choosing standards Describe your institution Describe one collection you’d like to digitize Describe your technical infrastructure

3/6/07 INCOLSA Workshop59 Beyond descriptive metadata Technical metadata Preservation metadata Rights metadata Structural metadata

3/6/07 INCOLSA Workshop60 Technical metadata For recording technical aspects of digital objects For long-term maintenance of data Migration Emulation Much can be generate automatically, but not all Some examples: NISO Z39.87: Data Dictionary – Technical Metadata for Digital Still Images & MIX NISO Z39.87: Data Dictionary – Technical Metadata for Digital Still ImagesMIX Schema for Technical Metadata for Text Forthcoming standard for audio from the Audio Engineering Society LC VMD draft schema for technical metadata for video files LC VMD draft schema

3/6/07 INCOLSA Workshop61 Image technical metadata Might include: Color space Bit depth Byte order Compression scheme Camera settings Operator name

3/6/07 INCOLSA Workshop62 Text technical metadata Might include: Character set Byte order Font/script Language

3/6/07 INCOLSA Workshop63 Audio technical metadata Might include: Byte order Checksum Sample rate Duration Number of channels

3/6/07 INCOLSA Workshop64 Video technical metadata Might include: Bits per sample Calibration information Sample format Signal format

3/6/07 INCOLSA Workshop65 Preservation metadata The set of everything you need to know to preserve digital objects over the long term Information that supports and documents the digital preservation process Includes technical metadata but also other elements Covers elements such as checksums, creation environment, and change history PREMIS is the prevailing model PREMIS

3/6/07 INCOLSA Workshop66 Rights metadata Machine- or human-readable indications of rights information for a resource Can be used to determine if a user can access a resource Can indicate rights holder of a resource for payment purposes Some current schemas METS rights XrML ODRL

3/6/07 INCOLSA Workshop67 Structural metadata For creating a logical structure between digital objects Multiple copies/versions of same item Multiple pages within item Multiple sizes of each page Meaningful groups of content Often handled transparently by a delivery system METS is the current primary standard METS

3/6/07 INCOLSA Workshop68 Why you should care about these standards You will migrate from your current system to another, probably in the next few years File formats become obsolete We have too many interesting collections to have to re-do work we’ve already done Standards promote interoperability

3/6/07 INCOLSA Workshop69 Building “Good digital collections”* Interoperable – with the important goal of cross-collection searching Persistent – reliably accessible Re-usable – repositories of digital objects that can be used for multiple purposes *Institute for Museum and Library Services. A Framework of Guidance for Building Good Digital Collections. Washington, D.C.: Institute for Museum and Library Services, November Framework of Guidance for Building Good Digital Collections.

3/6/07 INCOLSA Workshop70 Building “Good digital collections” Interoperable – with the important goal of cross-collection searching Persistent – reliably accessible Re-usable – repositories of digital objects that can be used for multiple purposes Good metadata promotes good digital collections.

3/6/07 INCOLSA Workshop71 Sharing your metadata Harvesting Collects metadata, processes it, and stores it locally to respond to user queries Open Archives Initiative Protocol for Metadata Harvesting Open Archives Initiative Protocol for Metadata Harvesting Federated searching Transmits user queries to multiple destinations in real time ILS vendors currently offering these products Protocols used Z39.50 SRU

3/6/07 INCOLSA Workshop72 OAI Protocol Structure Intentionally designed to be simple Data providers Have metadata they want to share “Expose” their metadata to be harvested Service providers Harvest metadata from data providers Provide searching of harvested metadata from multiple sources Can also provide other value-added services

3/6/07 INCOLSA Workshop73 Data Providers Set up a server that responds to harvesting requests Required to expose metadata in simple Dublin Core (DC) format Can also expose metadata in any other format expressible with an XML schema

3/6/07 INCOLSA Workshop74 Service Providers Harvest and store metadata Generally provide search/browse access to this metadata Can be general or domain-specific Can choose to collect metadata in formats other than DC Generally link out to holding institutions for access to digital content

3/6/07 INCOLSA Workshop75 Advantages for Libraries Any existing rules for description can be used Can share metadata without sacrificing local granularity Location of unique materials by many users Domain-specific service providers Middle ground between Google and OCLC One of a suite of tools to provide users with access to all of your materials

3/6/07 INCOLSA Workshop76 Why share metadata? Benefits to users One-stop searching Aggregation of subject-specific resources Benefits to institutions Increased exposure for collections Broader user base Bringing together of distributed collections Don’t expect users will know about your collection and remember to visit it.

3/6/07 INCOLSA Workshop77 Preparing your metadata for sharing Map to common formats; also called “crosswalking” To create “views” of metadata for specific purposes Mapping from robust format to more general format is common Mapping from general format to more robust format is ineffective

3/6/07 INCOLSA Workshop78 Crosswalks (1) For transforming between metadata formats Usually refers to transforming between content standards rather than structure standards, but not always Mapping from more robust format to less robust format effective; mapping from simpler format to more robust format less so Good practice to create and store most robust metadata format possible, then create other views for specific needs

3/6/07 INCOLSA Workshop79 Crosswalks (2) Can be in many formats Logical sets of rules [example]example Actual code [example]example Often need to tweak a generic crosswalk for a specific implementation Accommodating local practice Adding institution-specific information Adding context not available locally

3/6/07 INCOLSA Workshop80 Types of mapping logic Mapping the complete contents of one field to another Splitting multiple values in a single local field into multiple fields in the target schema Translating anomalous local practices into a more generally useful value Splitting data in one field into two or more fields Transforming data values Boilerplate values to include in output schema

3/6/07 INCOLSA Workshop81 Metadata as a view of the resource There is no monolithic, one-size-fits-all metadata record Metadata for the same thing is different depending on use and audience Harry Potter as represented by… a public librarypublic library an online bookstoreonline bookstore a fan sitefan site

3/6/07 INCOLSA Workshop82 Choice of vocabularies as a view Names LCNAF: Michelangelo Buonarroti, ULAN: Buonarroti, Michelangelo Places LCSH: Jakarta (Indonesia) TGN: Jakarta Subjects LCSH: Neo-impressionism (Art) AAT: Pointillism

3/6/07 INCOLSA Workshop83 Finding the right balance Metadata providers know the materials Document encoding schemes and controlled vocabularies Document practices Ensure record validity Aggregators have the processing power Format conversion Reconcile known vocabularies Normalize data Batch metadata enhancement

3/6/07 INCOLSA Workshop84 What does this record describe? identifier: publisher: Museum of Zoology, Fish Field Notes format:jpeg rights: These pages may be freely searched and displayed. Permission must be received for subsequent distribution in print or electronically. type:image subject: ; 1926; 0812; 18; Trib. to Sixteen Cr. Trib. Pine River, Manistee R.; JAM26-460; 05; 1926/05/18; R10W; S26; S27; T21N language: UND source: Michigan 1926 Metzelaar, ; description: Flora and Fauna of the Great Lakes Region Example courtesy of Sarah Shreeves, University of Illinois at Urbana-Champaign

3/6/07 INCOLSA Workshop85

3/6/07 INCOLSA Workshop86 Shareable metadata defined Metadata for aggregation with records from other institutions Promotes search interoperability - “the ability to perform a search over diverse sets of metadata records and obtain meaningful results” (Priscilla Caplan) Is human understandable outside of its local context Is useful outside of its local context Preferably is machine processable

3/6/07 INCOLSA Workshop87 Five C’s of shareable metadata Consistency Coherence Content Context Conformance

3/6/07 INCOLSA Workshop88 Consistency Records in a set should all reflect the same practice Fields used Vocabularies Syntax encoding schemes Allows aggregators to apply same enhancement logic to an entire group of records

3/6/07 INCOLSA Workshop89 Coherence Record should be self-explanatory Values must appear in appropriate elements Repeat fields instead of “packing” to explicitly indicate where one value ends and another begins

3/6/07 INCOLSA Workshop90 Content Choose appropriate vocabularies Choose appropriate granularity Make it obvious what to display Exclude unnecessary “filler” Make it clear what links point to

3/6/07 INCOLSA Workshop91 Context Include information not used locally Exclude information only used locally Current safe assumptions Users discover material through shared record User then delivered to your environment for full context Context driven by intended use

3/6/07 INCOLSA Workshop92 Conformance To standards Metadata standards (and not just DC) Vocabulary and encoding standards Descriptive content standards (AACR2, CCO, DACS) Technical standards (XML, Character encoding, etc)

3/6/07 INCOLSA Workshop93 A final word on interoperability We can no longer afford to only think about our local users Creating shareable metadata will require more work on your part Indiana is moving toward a portal of Indiana- related digital content – you should be planning for this now

3/6/07 INCOLSA Workshop94 Putting it all into practice Develop written documentation Develop a quality control workflow for metadata creation Share your findings with others

3/6/07 INCOLSA Workshop95 Further information These presentation slides: Metadata librarians listserv: Priscilla Caplan: Metadata Fundamentals for all Librarians, 2003