1. Metadata Harvesting Interoperable digital collections

2. Distributed libraries The reality in most digital libraries is that no one location has all the materials that may be of interest. It is often more efficient to allow a number of sites each to retain some of the materials. How can we assure clients that they will see all relevant resources, regardless of which library they search?

3. Two basic approaches One service provider with access to resources stored in multiple locations –Information about all the resources located at the service provider. –Services (DL scenarios) use the information to provide connections to resources at multiple locations Distributed services –Information kept with the resources –Services, local to each collection, interact with other collection sites

4. Two protocols Z39.50 –Developed before the web –Protocol for communicating with collection holders in order to provide services. Open Archives Initiative –Recent innovation –Central service provider gathers information from collection holders

5. Z39.50 - briefly Information Retrieval Service Definition and Protocol Specifications for Library Applications Initially developed over the OSI network standards Protocol for information exchange –Free the information seeker from the need to know the details of the target database configuration Each site provides services –Each service queries remote sites for needed information Information requests mapped to database queries at the collection site. Some inconsistency in the interpretation of queries.

6. Distributed Resources Multiple Services Service provider -- search, browse, compare, etc. Data provider Approach 1 - One service provider gathers information about data and uses it to provide services

7. Distributed data and services Approach 2: Each system is both a data repository and a service provider. Services query other data providers as needed. Search, browse Search, browse, compare

8. Service provider -- search, browse, compare, etc. Data provider Each server likely to have its own clients. Difference is whether the information exchange is periodic or ad hoc Hybrid systems

9. Open Archives Initiative (OAI) Web-based –Uses HTTP to communicate between sites Centralized server –Services provided from a site that has already gathered the information it needs for those services from a distributed collection of sites.

10. Z39.50 Special purpose protocol (machine to machine, not web interface) Gathers information when it is requested, not on a scheduled basis.

11. OAI Compared to Z39.50 Z39.50OAI Content (Objects)Distributed World ViewBibliographic Object PresentationData provider Searching isDistributedCentralized Search done byData providerService provider Metadata searched isUp to dateStale Semantic MappingWhen searchingMetadata delivery Source: oai.grainger.uiuc.edu/FinalReport/JCDL_2003_OAI_Intro.ppt

12. Open Archives Initiative Protocol for Metadata Harvesting -- OAI-PMH Repository OAI Harvester OAI HTTP req (OAI verb) HTTP resp (XML) OAI PMH defines an interface between the Harvester and any number of Repositories Metadata Provider Service Provider Implemented as CGI, ASP, PHP, or other Any system may serve as a harvester, repository, or both

13. OAI components Service Providers and Data Providers Requests and Responses http://www.oaforum.org/tutorial/english/page3.htm#section3

14. Records Metadata of a resource. Three parts –Header (required) Identifier (required: 1 only) Datestamp (required: 1 only) setSpec elements (optional: 0, 1, or more) Status attribute for deleted item –Metadata (required) XML encoded metadata with root tag, namespace Repositories must support Dublin Core, other formats optional –“About” statement (optional) Right statements Provenance statements

15. Identifiers Globally unique identifier Valid URI –Examples oai: : oai:etd.vt.edu:etd-1234567890 –Must resolve to one item No duplicates No reuse of previously used identifiers

16. Datestamps Date of last modification of a record –Used only for harvesting (meta metadata?) Mandatory for each item in the repository Two levels of granularity possible –YYYY-MM-DD –YYYY-MM-DDThh:mm:ssZ T … Z = Time zone -- must be GMT Allows harvesting incrementally -- get only what is new since last visit –Accessed by arguments from and until

17. The OAI-PMH verbs Each requests a specific response from a data repository

18. Identify Function: Description of the archive Example: http://www.language-archives.org/cgi-bin/olaca3.pl?verb=Identify Parameters: none Errors/exceptions: –badArgument (there should not be any) Response format: Element Example Ordinality ‡ repositoryName My Archive 1 baseURL http://archive.org/oai 1 protocolVersion 2.0 1 earliestDatestamp 1999-01-01 1 deleteRecords no, transient, persistent 1 granularity YYYY-MM-DD, YYYY-MM-DDThh:mm:ssZ 1 adminEmail oai-admin@archive.org + compression deflate, compress * description oai-identifier, eprints, friends, … * ‡ Ordinality: 1 = mandatory, 1 only; + = mandatory, 1 only; * = optional, 0 or more

19. 2006-10-17T01:37:44Z http://www.language-archives.org/cgi- bin/olaca3.pl − OLAC Aggregator http://www.language-archives.org/cgi-bin/olaca3.pl 2.0 mailto:haejoong@ldc.upenn.edu 2002-12-14 no YYYY-MM-DD − <!-- maybe later identity --> Actual response from http://www.language-archives.org/cgi-bin/olaca3.pl?verb=Identify Continued

20. − oai OLACA.language-archives.org : oai:ethnologue.com:aaa Continued

21. − http://www.language-archives.org:8082/dp9/ Steven Bird & Gary Simons Coordinators mailto:olac-admin@language-archives.org Open Language Archives Community http://www.language-archives.org/ Philadelphia, U.S.A. − This repository contains all records from OLAC-registered archives. It is intended to be used by services which do not want to harvest individual OLAC archives. − Metadata may be used only subject to the access permissions given by the individual archives.

22. ListMetadataFormats Function: retrieve available metadata formats from archive Example: archive.org/oai-script?verb=ListMetadataFormats& identifier=oai:HUBerlin.de:3000218 Parameters: identifier (optional) Errors/exceptions: –badArgument –idDoesNotExist –noMetadataFormats

23. − 2006-10-17T01:58:06Z http://www.language-archives.org/cgi- bin/olaca3.pl − olac http://www.language-archives.org/OLAC/1.0/olac.xsd http://www.language-archives.org/OLAC/1.0/ − olac_display http://www.language-archives.org/OLAC/1.0/olac.xsd http://www.language-archives.org/OLAC/1.0/ − oai_dc http://www.openarchives.org/OAI/2.0/oai_dc.xsd http://www.openarchives.org/OAI/2.0/oai_dc/ Response to http://www.language-archives.org/cgi-bin/ olaca3.pl?verb=ListMetadataFormatshttp://www.language-archives.org/cgi-bin/ olaca3.pl?verb=ListMetadataFormats

24. ListSets Function: retrieve set structure of a repository Example: archive.org/oai-script?verb=ListSets Parameters: resumptionToken (exclusive) Errors/exceptions: –badArgument –badResumptionToken –noSetHierarchy Sets are optional and are used to divide a repository into separate units that will be of interest to different harvesters.

25. ListIdentifiers Function: abbieviated form of ListRecords, retrieve only headers Example: archive.org/oai-script?verb=ListIdentifiers&metadataPrefix= oai_dc&from=2002-12-01 Parameters: –from (optional) –until (optional) –metadataPrefix (required) –set (optional) –resumptionToken (exclusive) Errors/exceptions: –badArgument –badResumptionToken –cannotDisseminateFormat –noRecordsMatch –noSetHierarchy

26. ListRecords Function: harvest records from a repository Example: archive.org/oai-script?verb=ListRecords& metadataPrefix=oai_dc&set=biology Parameters: –from (optional) –until (optional) –metadataPrefix (required) –set (optional) –resumptionToken (exclusive) Errors/exceptions: –badArgument –badResumptionToken –cannotDisseminateFormat –noRecordsMatch –noSetHierarchy

27. GetRecord Function: retrieve an individual metadata record from a repository Example: archive.org/oai-script?verb=GetRecord&identifier=oai:HUBerlin.de: 3000218 &metadataPrefix=oai_dc Parameters: –Identifier (required) –metadataPrefix (required) Errors/exceptions: –badArgument –cannotDisseminateFormat –idDoesNotExist

30. Interoperability The goal: communication, without human intervention, between information sources –Books that “talk to each other” Live links for references Knowledge of how to find relevant resources when needed Ability to query other information locations

31. Protocols Precise rules for interactions between independent processes –Format of the messages Both structure and content –Specified behavior in response to specific messages Many ways to accomplish the same result, but both sides must have the same understanding of the rules of engagement.

32. Protocol Types RPC model –Point to point –Completely open to definition by developer Verbs (methods) Nouns (objects, resources) –Useful to closed community or group who know about the availability of the resource.

33. SOAP Initial words of the acronym have been discontinued. Initially developed as part of the Microsoft.NET paradigm –Now in W3C committee Stateless, one-way message exchange paradigm XML encoded Flexibility of RPC, but more constrained in the way communication is formatted.

34. REST REpresentational State Transfer An after-the-fact definition of the architecture of the World Wide Web The model is –Client/server –Stateless –Cacheable –Layered Resource interface constrained –Restricted verbs –Restricted content types

35. REST and RPC RPC provides flexibility for any type of interaction between any type of resources REST provides consistency to allow interaction among resources without prior discovery of accepted actions and responses.

36. SOAP and REST Debate in the Web community about which is the better paradigm for application development REST -- restricted, but simple extension of existing Web processes SOAP -- added flexibility with cost in terms of bandwidth, security, complexity for development

37. References Giving SOAP a REST http://www.devx.com/DevX/Article/8155http://www.devx.com/DevX/Article/8155 SOAP Version 1.2 Part 0: Primer http://www.w3.org/TR/2003/REC- soap12-part0-20030624/#L1153http://www.w3.org/TR/2003/REC- soap12-part0-20030624/#L1153 OAI For Beginners - The Open Archives Forum online tutorial: http://www.oaforum.org/tutorial/index.php http://www.oaforum.org/tutorial/index.php Z39.50 Resource Page: http://www.niso.org/standards/resources/Z3950_Resources.html http://www.niso.org/standards/resources/Z3950_Resources.html Z39.50 An Overview of Development and the Future (1995) http://www.cqs.washington.edu/~camel/z/z.html