1. OAI Overview DLESE OAI Workshop April 29-30, 2002 John Weatherley (jweather@ucar.edu)

2. DLESE OAI April 29-30, 20022 Workshop Schedule Day 1 Morning Overview of OAI Look at OAI tools and resources Afternoon DLESE OAI software installation, configuration and setup Day 2 Morning Overview of NDSL and DLESE interoperability architecture NSDL metadata overview Metadata and OAI

3. DLESE OAI April 29-30, 20023 Resources Workshop presentation slides, links to tools and other OAI resources are located at: http://oai.dlese.org http://oai.dlese.org

4. DLESE OAI April 29-30, 20024 What is DLESE and NSDL? DLESE: Digital Library for Earth System Education: provides access to digitally accessible resources for learning about the Earth system NSDL: National Science (STEM) Digital Library: network of scholarly and educational digital libraries related to science (DLESE will be part of this network)

5. DLESE OAI April 29-30, 20025 1. What is the OAI? What is the Open Archive Initiative (OAI)? Organization dedicated to solving problems of digital library interoperability by defining simple protocols and standards Grew out of the e-prints (arXiv) community at Los Alamos What is the OAI Protocol for Metadata Harvesting (OAI-PMH)? Protocol to transfer metadata from a source archive to a destination archive How is the OAI-PMH Being Used by the NSDL and DLESE? The OAI-PMH has been adopted as a primary means of gathering and sharing metadata among contributors Also used to facilitate internal management of metadata stores

6. DLESE OAI April 29-30, 20026 What is Metadata? Data refers to digital objects e.g. the resources themselves Metadata is data about data e.g. a description about a resource, not the resource itself OAI is used to transmit metadata

7. DLESE OAI April 29-30, 20027 2. Definitions / Concepts Basic Principles Harvesting vs. Federation Data Providers vs. Service Providers Underlying Technology HTTP and XML XML Namespaces and Schema Protocol Policies and Conventions Basic Policies Sets

8. DLESE OAI April 29-30, 20028 Harvesting vs. Federation Competing approaches to interoperability Federation is when services such as searching are run remotely Harvesting is when metadata is transferred from remote sources to the destination where the services are located Federation requires more effort at the remote site but is easier for the local system Harvesting requires less effort at the remote site; Services are provided by the local system OAI uses the harvesting model

9. DLESE OAI April 29-30, 20029 Data Providers vs. Service Providers Data Providers refer to entities who possess metadata and are willing to share this with others (e.g. collection builders) Service Providers are entities who harvest data from Data Providers in order to provide higher-level services to users (e.g. searching, browsing, recommender systems, etc.). The NSDL and DLESE are examples.

10. DLESE OAI April 29-30, 200210 Features of the OAI Approach Lightweight: Low overhead for Data Providers Protocol is relatively simple to implement Many plug-and-play tools publicly available Transports any metadata framework that can be made available in XML form (details to come) Details of searching, browsing, annotation and other advanced services are handled by the Service Provider

11. DLESE OAI April 29-30, 200211 Data Providers: (collection builders) … … … Service Provider (DLESE, NSDL) Harvested Records 3. Provide searching, browsing, and other services over the data. OAI protocol (over http) 1. Service Provider polls periodically for new records 2. New records downloaded and cached by the Service Provider Metadata Harvesting Framework Library User

12. DLESE OAI April 29-30, 200212 HTTP and XML The OAI-PMH is an almost stateless request/response protocol Requests and responses are sent via the HTTP protocol Requests are encoded as GET/POST operations Responses are well-formed XML documents

13. DLESE OAI April 29-30, 200213 Well-formed and Valid XML Correct Dodge Spirit 1994 you CO Incorrect Dodge Spirit 1994 CO you

14. DLESE OAI April 29-30, 200214 DTD, Schemas & Namespace DTD’s: Document Type Definition Describe the elements of XML instance documents Not well-formed XML Some data-typing Namespaces harder to deal with Schemas Describe the elements of XML instance documents Well-formed XML Strong data-typing Namespaces are easier to deal with Namespace: Collection of related element names identified by a name label (e.g. dc)

15. DLESE OAI April 29-30, 200215 XML Namespaces and Schema Consistency and data quality is ensured by using XML Schema descriptions for each possible response XML Namespaces are used where necessary to clearly define which parts of the responses are actual metadata and which support the OAI-PMH. Example: http://www.cstc.org/cgi- bin/OAI/CSTC.pl?verb=GetRecord&identifier=oai%3ACSTC%3A103&metad ataPrefix=oai_dc

16. DLESE OAI April 29-30, 200216 Basic OAI Policies and Conventions Each metadata record from a given Data Provider must have a unique ID (OAI ID is not necessarily the same as the record ID) Each metadata record must be persistent so that Service Providers can always refer back to the source Each record must have a date stamp indicating creation / modification date Dates provide a mechanism for incremental and continuous transfer of metadata by only requesting records that have changed since the previous harvest Flow Control - Resumption Tokens can be used to return partial results – the client is issued a token which may be presented to the server to receive more results Multiple formats of metadata are allowed Examples: Dublin Core, DLESE IMS

17. DLESE OAI April 29-30, 200217 Sets OAI-PMH mechanism to allow for harvesting of sub- collections Semantics for sets are defined outside of the protocol Sets are defined by conventions established between data and service providers Example sets within DLESE might be: DWEL, COMET, LDEO, etc. Example sets within the NDSL might be: DLESE, DLESE:DWEL, DLESE:COMET, DLESE:LDEO, etc. Sets can be established that enable querying (e.g. by topic, author name, subject area, etc.) Example: The Open Digital Library (Suleman, 2001)

18. DLESE OAI April 29-30, 200218 3. Requirements to be a Data Provider Source of metadata Human or automated resource catalogers Metadata mappings Crosswalks from native formats to DC or other formats Server technology Handled by the OAI software Datestamps Deletions Unique identifiers

19. DLESE OAI April 29-30, 200219 4. The OAI-PMH Service Requests Identify ListMetadataFormats ListSets GetRecord ListIdentifiers ListRecords Date Ranges Resumption Tokens

20. DLESE OAI April 29-30, 200220 Identify Purpose Return general information about the archive and its policies Parameters None Sample URL http://oai.dlese.org/provider?verb=Identify

21. DLESE OAI April 29-30, 200221 ListMetadataFormats Purpose List metadata formats supported by the archive as well as their schema locations and namespaces Parameters Identifier – for a specific record ( O ) Sample URL http://oai.dlese.org/provider?verb=ListMetadataFormats

22. DLESE OAI April 29-30, 200222 ListSets Purpose Provide a hierarchical listing of sets in which records may be organized Parameters None Sample URL http://oai.dlese.org/provider?verb=ListSets

23. DLESE OAI April 29-30, 200223 GetRecord Purpose Returns the metadata for a single identifier in the form on an OAI record Parameters identifier – id for the record ( R ) metadataPrefix – metadata format ( R ) Sample URL http://oai.dlese.org/provider?verb=GetRecord&identifier=dlese%3ADLE SE-000-000-000-002&metadataPrefix=dlese_ims http://oai.dlese.org/provider?verb=GetRecord&identifier=dlese%3ADLE SE-000-000-000-002&metadataPrefix=dlese_ims

24. DLESE OAI April 29-30, 200224 ListIdentifiers Purpose List all unique identifiers corresponding to the record in the repository Parameters from – start date ( O ) until – end date ( O ) resumptionToken – flow control mechanism ( X ) Sample URL http://oai.dlese.org/provider?verb=ListIdentifiers

25. DLESE OAI April 29-30, 200225 ListRecords Purpose Retrieves metadata for multiple records Parameters from – start date ( O ) until – end date ( O ) resumptionToken – flow control mechanism ( X ) set – set to harvest from ( O ) metadataPrefix – metadata format ( R ) Sample URL http://oai.dlese.org/provider?verb=ListRecords&metadataPrefix=dlese_ims

26. DLESE OAI April 29-30, 200226 DLESE Architecture Metadata Repository Collections DLESE Portal Search & Discovery Direct Entry OAI Resources Services: (e.g. What’s New) NSDL OAI Library Users

27. DLESE OAI April 29-30, 200227 References 1.“Building Interoperable Digital Libraries: A Practical Guide to creating Open Archives,” Hussein Suleman (hussein@vt.edu), JCDL 2001 Tutorial.hussein@vt.edu 2.“A Framework for Building Open Digital Libraries,” Hussein Suleman and Edward A. Fox, in D-Lib Magazine, December, 2001. http://www.dlib.org/dlib/december01/suleman/12suleman.html http://www.dlib.org/dlib/december01/suleman/12suleman.html 3.The Open Archives Initiative http://www.openarchives.orghttp://www.openarchives.org