Presentation is loading. Please wait.

Presentation is loading. Please wait.

Extracting XML from Unicorn with OAI and SRU

Similar presentations


Presentation on theme: "Extracting XML from Unicorn with OAI and SRU"— Presentation transcript:

1 Extracting XML from Unicorn with OAI and SRU
European Unicorn User Group Conference Glasgow Caledonian University September 7th & 8th, 2006 Benoit PAUWELS Université Libre de Bruxelles (ULB) Brussels

2 Agenda Introduction – Unicorn interfaces
Part 1: An OAI frontend for Unicorn Part 2: An SRU frontend for Unicorn Short description of OAI and SRU protocols Overview of technical implementation Use cases and demos

3 Introduction OAI and SRU are ‘open’ protocols that permit exchange of metadata between information systems Well-known Unicorn interfaces: Unicorn API server Unicorn Webcat/iBistro/iLink server Unicorn Z39.50 server All comply to the philosophy of request/response sequences

4 Unicorn interfaces: API server
Catalogue database [ Records and indexes ] TCPIP/Socket API request SirsiDynix Character client C Workflows client Java Themes client TCPIP/Socket API response API datacodes/values Client system Unicorn server Communication protocol TCPIP/Socket Information exchange protocol proprietary SirsiDynix API requests/responses Returned record structure proprietary SirsiDynix format (data-codes and -values)

5 Unicorn interfaces: iLink
Web Server iLink Catalogue database [ Records and indexes ] HTTP iLink request (URL) Any Web browser HTTP HTML page HTML Client system Unicorn server Communication protocol HTTP Information exchange protocol URL requests / HTML responses Returned record structure HTML

6 Unicorn interfaces: Z39.50 Client system Unicorn server
Catalogue database [ Records and indexes ] Z39.50 Z39.50 request Any Z3950 client Z3950 Z3950 response MARC21 Client system Unicorn server Communication protocol Z39.50 specific Information exchange protocol Z39.50 specific Returned record structure typically MARC21

7 Unicorn interfaces API: Proprietary
low interoperability level HTML: Record data not well structured low reusability level Z39.50: Protocol specific more difficult to implement (high learning curve) Z39.50 is statefull Difficult to integrate into today’s web services environments communication: use HTTP information exchange: use open protocols (like OAI and SRU) record data structure: use XML (according to well-defined XML Schema)

8 2 new Unicorn interfaces
HTTP / Open / XML OAI-PMH: Open Archives Initiative – Protocol for Metadata Harvesting SRU: Search and Retrieve via URL

9 OAI-PMH : the protocol Service Provider Data Provider Web Server
OAI Frontend Document Archive HTTP embedded OAI requests HTTP embedded OAI responses Service Provider Data Provider

10 OAI-PMH: the protocol ‘Harvester collects metadata from archives’
Stateless protocol: sequence of OAI requests/responses over HTTP Just harvesting -- NOT searching

11 OAI-PMH: the protocol OAI requests HTTP GET|POST requests Syntax
BASE URL host + port + path of OAI request handler key=value pairs Examples: cgi-bin/OAI20/catalog? verb=Identify _ oai/1.1/bmcoai.asp? verb=GetRecord&identifier=oai:bmc: &metadataPrefix=oai_dc

12 OAI-PMH: the protocol OAI responses
XML encoded bytestreams, containing the records Record = triplet header (unique OAI identifier) metadata about Metadata schemes XML Schema Minimum: unqualified Dublin Core Community specific Example of a record (catkey from ULB catalogue): oai_dc marc21 umods

13 OAI-PMH: the protocol Simple : 6 OAI requests/responses Identify
_ ListMetadataFormats [identifier] _ ListSets _ GetRecord identifier, metadataPrefix verb=GetRecord&identifier=oai:ulbcat:245000&metadataPrefix=marc21 _

14 OAI-PMH: the protocol Simple : 6 OAI requests/responses
ListRecords metadataPrefix, [from,until,set] verb=ListRecords&metadataPrefix=oai_dc _ verb=ListRecords&metadataPrefix=mhld21&set=elper _ verb=ListRecords&metadataPrefix=marc21&from= _ ListIdentifiers metadataPrefix, [from,until,set] verb=ListIdentifiers&metadataPrefix=oai_dc _

15 OAI frontend for Unicorn
Implementation of the data provider functionality (2001) pick a template and interface with Unicorn through Unicorn database tools Our choice: Object Oriented Perl frontend (H. Suleman – Virginia Tech) _

16 OAI frontend for Unicorn
HTTP embedded OAI request Unicorn Server HTTP server Unicorn database CGI OAI C wrapper call the appropriate OAI request handler retrieve metadata from Unicorn database format in XML fork in ‘sirsi’ environment OAI.pl HTTP embedded OAI response

17 OAI frontend for Unicorn
Example: implementation of the GetRecord request verb=GetRecord&identifier=oai:ulbcat:245000&metadataPrefix=oai_dc 1. Get metadata from Unicorn for catkey $record = `echo $catkey | catalogdump -of | filtermarc -iALL -od -Ds`; _ @dates = split(‘\|’,`echo $catkey | selcatalog -iK -opr`); 2. Convert ANSEL character set into ISO-LATIN-1 3. Map from MARC to oai_dc _ 4. Format into XML

18 OAI frontend for Unicorn
Example: implementation of the ‘set’ parameter of the ListRecords request verb=ListRecords&metadataPrefix=oai_dc&set=elper Precompile set as a file of catkeys name of file: « name of set_catkeys » einstein_albert_catkeys elper_catkeys sd_catkeys all_catkeys through periodic execution of « mkoaisets » custom report

19 OAI frontend for Unicorn
Example: implementation of the ‘from/until’ parameters of the ListRecords request verb=ListRecords&metadataPrefix=oai_dc&from= &until= BRS index on creation/modification date? Every Unicorn record that gets created or modified is ‘touched’ in the ‘textedit’ and ‘browsedit’ directories Custom report ‘cadutext’ saves catkeys to <ud>/Savedkeys/adutext/rptid adds line ‘rptid|date|status’ to <ud>/Lastruns/cadutext Example: « from= &until=  » obtain report ids for all runs of cadutext after and before from the file <ud>/Lastruns/cadutext for each of these report ids: obtain catkeys from <ud>/Savedkeys/adutext/rptid and save them to randomnumber_catkeys file sort and uniq the randomnumber_catkeys file

20 OAI frontend for Unicorn
Limitations of implementation: ListRecords/ListIdentifiers: The from and until parameters are not permitted if the set parameter is given on the request The from and until parameters are permitted if the set parameter is not given on the request, but their values should fall within a certain date range (at this moment arbitrarily set to ‘today - 2 months’ and ‘today’) Deleted records Complete source code and documentation available on the API Repository (http://sirsiapi.org)

21 OAI frontend - use cases @ ULB
Use case 1: Vlink - OpenURL resolver system joint project with Vrije Universiteit Brussel (VUB) Vlink knowledge base ULB iLink JSTOR ISI Web of Science Elsevier ScienceDirect OVID WebSpirs OpenURL HTML extended services

22

23 OAI frontend - use cases @ ULB
Use case 1: Vlink - OpenURL resolver system OpenURL sent from iLink sid=ULB:Webcat&id=oai:ulbcat:617924 This OpenURL does not contain enough metadata for the specific item ==> Vlink does a fetch back to Unicorn through an OAI GetRecord request to obtain a full MARC21 bibliographic description verb=GetRecord&identifier=oai:ulbcat:617924&metadataPrefix=marc21

24 OAI frontend - use cases @ ULB
Use case 1: Vlink - OpenURL resolver system Feed Vlink Knowledge Base through OAI harvesting VLink Vlink Knowledge Base Unicorn OAI-PMH verb=ListRecords&metadataPrefix=mhld21&set=elper

25 OAI frontend - use cases @ ULB
Use case 2: Unicat - Virtual Union Catalog of Belgium University library Catalog Unicorn Aleph VIRTUA VUBIS End User Unicat WWW Gateway Unicat Indexer Unicat Harvester Search/ Browse indexes Union OAI Archive OAI SRU Public Museum Other Central Repository Data providers HTML

26 SRU : the protocol Client System Unicorn Server
Web Server SRU Frontend Catalogue database [ Records and indexes ] HTTP SRU request HTTP SRU response XML Client System Unicorn Server Communication protocol HTTP Information exchange protocol SRU Returned record structure XML

27 SRU: the protocol ‘Client searches and retrieves metadata records from an archive’ Stateless protocol: sequence of SRU requests/responses over HTTP Search and Retrieve (<-> OAI: harvesting)

28 SRU: the protocol SRU requests HTTP GET requests Syntax
BASE URL host + port + path of SRU request handler key=value pairs 3 possible requests (operations) explain serves to record facilities available at an SRU server used by clients to self-configure returned explain record is in XML and follows the ZeeRex Schema Example: _ scan allows the client to request a range of the available terms at a given point within a list of indexed terms enables clients to present an ordered list of values and, if supported, how many hits there would be for a search on that term searchRetrieve

29 SRU: the protocol searchRetrieve operation
searchRetrieve (principal) parameters Version: (of the request); current protocol version: 1.1 query: query expressed in CQL startRecord: position within the sequence of matched records of the first record to be returned maximumRecords: number of records requested to be returned recordSchema: schema requested for the records to be returned stylesheet: URL for an xml stylesheet. The client requests that the server simply return this URL in the response. CQL « Traditionally, query languages have fallen into two camps: Powerful, expressive languages, not easily readable nor writable by non-experts (e.g. SQL, PQF, and XQuery);or simple and intuitive languages not powerful enough to express complex concepts (e.g. CCL and google). CQL tries to combine simplicity and intuitiveness of expression for simple, every day queries, with the richness of more expressive languages to accomodate complex concepts when necessary. » (http://www.loc.gov/standards/sru/cql)

30 SRU: the protocol searchRetrieve operation Examples of CQL queries:
dinosaur title = "complete dinosaur" title exact "the complete dinosaur" dinosaur not reptile dinosaur and bird or dinobird publicationYear < 1980 title all "complete dinosaur" title contains all of the words: ‘complete’, and ‘dinosaur’ title any "dinosaur bird reptile" title contains any of the words: ‘dinosaur’, ‘bird’, or ‘reptile’ ribs prox/distance<=5 chevrons a more specific proximity query: ‘ribs’ within 5 words of ‘chevrons’

31 SRU: the protocol searchRetrieve operation -- examples
_ _ _ all "einstein albert“ _ all "einstein albert“ _ all "einstein albert“&stylesheet=http://bib49.ulb.ac.be/cibleCanevas.xsl _ all "einstein albert“&stylesheet=http://bib49.ulb.ac.be/cibleTypo3.xsl _

32 SRU frontend for Unicorn
Web Server SRU Frontend Catalogue database [ Records and indexes ] HTTP SRU request HTTP SRU response XML Client system Unicorn Server

33 SRU frontend for Unicorn
Z Frontend Catalogue database [ Records and indexes ] SRU/Z Gateway Web Server HTTP SRU request Z3950 Z3950 request HTTP SRU response XML Z3950 Z3950 response Client system SRU/Z39.50 Unicorn Server

34 SRU frontend for Unicorn
SRU/Z39.50 Gateway: YAZ Proxy (Index Data) Implemented at ULB: 7/2006 (2 days) config.xml <target name="cible" default="1"> <url>bib7.ulb.ac.be:2200</url> <xi:include href="explain.xml"/> <cql2rpn>pqf.properties</cql2rpn> </target> <target name=“slavko" default="1"> <url>velma.library.mun.ca:2200</url> <xi:include href="explain.slavko.xml"/> <cql2rpn>pqf.slavko.properties</cql2rpn> explain.xml ZeeRex XML record as response to ‘explain’ operation pqf.properties specifies the mapping of various CQL indexes, relations, etc. into Type-1 query attributes

35 SRU frontend for Unicorn
YAZ Proxy version=1.1&operation=searchRetrieve&maximumRecords=10&startRecord=1& query=title all "einstein albert“& stylesheet=http://bib49.ulb.ac.be/cibleTypo3.xsl _ version=1.1&operation=searchRetrieve&maximumRecords=10&startRecord=1& query=title all "einstein albert“& stylesheet=http://bib49.ulb.ac.be/cibleTypo3.xsl _

36 SRU frontend : use case @ ULB
Seamless integration of catalog searches in CMS Typo3 Example HTML page containing biography of famous belgian historian Henri Pirenne frame pointing to the following URL: version=1.1&operation=searchRetrieve&maximumRecords=10&startRecord=1& query=pirenne%20and%20epub-dnu-* &stylesheet=http://bib49.ulb.ac.be/cibleTypo3.xsl Project Unicorn contains descriptions of databases, websites, etc with local thematic classification codes in 653 create thematic websites within our CMS, containing frames that list available databases per theme


Download ppt "Extracting XML from Unicorn with OAI and SRU"

Similar presentations


Ads by Google