1. Extracting XML from Unicorn with OAI and SRU
European Unicorn User Group Conference
Glasgow Caledonian University
September 7th & 8th, 2006
Benoit PAUWELS
Université Libre de Bruxelles (ULB)
Brussels

2. Agenda Introduction – Unicorn interfaces
Part 1: An OAI frontend for Unicorn
Part 2: An SRU frontend for Unicorn
Short description of OAI and SRU protocols
Overview of technical implementation
Use cases and demos

3. Introduction
OAI and SRU are ‘open’ protocols that permit exchange of metadata between information systems
Well-known Unicorn interfaces:
Unicorn API server
Unicorn Webcat/iBistro/iLink server
Unicorn Z39.50 server
All comply to the philosophy of request/response sequences

4. Unicorn interfaces: API server
Catalogue database
[ Records and
indexes ]
TCPIP/Socket
API request
SirsiDynix
Character client
C Workflows client
Java Themes client
TCPIP/Socket API response
API datacodes/values
Client system
Unicorn server
Communication protocol TCPIP/Socket
Information exchange protocol proprietary SirsiDynix API requests/responses
Returned record structure proprietary SirsiDynix format (data-codes and -values)

5. Unicorn interfaces: iLink
Web Server
iLink
Catalogue database
[ Records and
indexes ]
HTTP
iLink request (URL)
Any Web browser
HTTP HTML page
HTML
Client system
Unicorn server
Communication protocol HTTP
Information exchange protocol URL requests / HTML responses
Returned record structure HTML

6. Unicorn interfaces: Z39.50 Client system Unicorn server
Catalogue database
[ Records and
indexes ]
Z39.50
Z39.50 request
Any Z3950 client
Z3950 Z3950 response
MARC21
Client system
Unicorn server
Communication protocol Z39.50 specific
Information exchange protocol Z39.50 specific
Returned record structure typically MARC21

7. Unicorn interfaces API: Proprietary
low interoperability level
HTML: Record data not well structured
low reusability level
Z39.50: Protocol specific
more difficult to implement (high learning curve)
Z39.50 is statefull
Difficult to integrate into today’s web services environments
communication: use HTTP
information exchange: use open protocols (like OAI and SRU)
record data structure: use XML (according to well-defined XML Schema)

8. 2 new Unicorn interfaces
HTTP / Open / XML
OAI-PMH: Open Archives Initiative – Protocol for Metadata Harvesting
SRU: Search and Retrieve via URL

9. OAI-PMH : the protocol Service Provider Data Provider Web Server
OAI Frontend
Document Archive
HTTP embedded
OAI requests
HTTP embedded
OAI responses
Service Provider
Data Provider

10. OAI-PMH: the protocol ‘Harvester collects metadata from archives’
Stateless protocol: sequence of OAI requests/responses over HTTP
Just harvesting -- NOT searching

11. OAI-PMH: the protocol OAI requests HTTP GET|POST requests Syntax
BASE URL
host + port + path of OAI request handler
key=value pairs
Examples:
cgi-bin/OAI20/catalog? verb=Identify _
oai/1.1/bmcoai.asp? verb=GetRecord&identifier=oai:bmc: &metadataPrefix=oai_dc

12. OAI-PMH: the protocol OAI responses
XML encoded bytestreams, containing the records
Record = triplet
header (unique OAI identifier)
metadata
about
Metadata schemes
XML Schema
Minimum: unqualified Dublin Core
Community specific
Example of a record (catkey from ULB catalogue):
oai_dc marc21 umods

13. OAI-PMH: the protocol Simple : 6 OAI requests/responses Identify_
ListMetadataFormats [identifier] _
ListSets _
GetRecord identifier, metadataPrefix
verb=GetRecord&identifier=oai:ulbcat:245000&metadataPrefix=marc21 _

14. OAI-PMH: the protocol Simple : 6 OAI requests/responses
ListRecords metadataPrefix, [from,until,set]
verb=ListRecords&metadataPrefix=oai_dc _
verb=ListRecords&metadataPrefix=mhld21&set=elper _
verb=ListRecords&metadataPrefix=marc21&from= _
ListIdentifiers metadataPrefix, [from,until,set]
verb=ListIdentifiers&metadataPrefix=oai_dc _

15. OAI frontend for Unicorn
Implementation of the data provider functionality (2001)
pick a template and interface with Unicorn through Unicorn database tools
Our choice: Object Oriented Perl frontend (H. Suleman – Virginia Tech) _

16. OAI frontend for Unicorn
HTTP embedded
OAI request
Unicorn Server
HTTP server
Unicorn
database
CGI
OAI
C wrapper
call the appropriate OAI request handler
retrieve metadata from Unicorn database
format in XML
fork in ‘sirsi’
environment
OAI.pl
HTTP embedded
OAI response

17. OAI frontend for Unicorn
Example: implementation of the GetRecord request
verb=GetRecord&identifier=oai:ulbcat:245000&metadataPrefix=oai_dc
1. Get metadata from Unicorn for catkey
$record = `echo $catkey | catalogdump -of | filtermarc -iALL -od -Ds`; _
@dates = split(‘\|’,`echo $catkey | selcatalog -iK -opr`);
2. Convert ANSEL character set into ISO-LATIN-1
3. Map from MARC to oai_dc _
4. Format into XML

18. OAI frontend for Unicorn
Example: implementation of the ‘set’ parameter of the ListRecords request
verb=ListRecords&metadataPrefix=oai_dc&set=elper
Precompile set as a file of catkeys
name of file: « name of set_catkeys »
einstein_albert_catkeys
elper_catkeys
sd_catkeys
all_catkeys
through periodic execution of « mkoaisets » custom report

19. OAI frontend for Unicorn
Example: implementation of the ‘from/until’ parameters of the ListRecords request
verb=ListRecords&metadataPrefix=oai_dc&from= &until=
BRS index on creation/modification date?
Every Unicorn record that gets created or modified is ‘touched’ in the ‘textedit’ and ‘browsedit’ directories
Custom report ‘cadutext’
saves catkeys to <ud>/Savedkeys/adutext/rptid
adds line ‘rptid|date|status’ to <ud>/Lastruns/cadutext
Example: « from= &until=  »
obtain report ids for all runs of cadutext after and before from the file <ud>/Lastruns/cadutext
for each of these report ids: obtain catkeys from <ud>/Savedkeys/adutext/rptid and save them to randomnumber_catkeys file
sort and uniq the randomnumber_catkeys file

20. OAI frontend for Unicorn
Limitations of implementation:
ListRecords/ListIdentifiers:
The from and until parameters are not permitted if the set parameter is given on the request
The from and until parameters are permitted if the set parameter is not given on the request, but their values should fall within a certain date range (at this moment arbitrarily set to ‘today - 2 months’ and ‘today’)
Deleted records
Complete source code and documentation available on the API Repository (

21. OAI frontend - use cases @ ULB
Use case 1: Vlink - OpenURL resolver system joint project with Vrije Universiteit Brussel (VUB)
Vlink
knowledge base
ULB iLink
JSTOR
ISI Web of Science
Elsevier ScienceDirect
OVID WebSpirs
OpenURL
HTML
extended services

23. OAI frontend - use cases @ ULB
Use case 1: Vlink - OpenURL resolver system
OpenURL sent from iLink
sid=ULB:Webcat&id=oai:ulbcat:617924
This OpenURL does not contain enough metadata for the specific item ==> Vlink does a fetch back to Unicorn through an OAI GetRecord request to obtain a full MARC21 bibliographic description
verb=GetRecord&identifier=oai:ulbcat:617924&metadataPrefix=marc21

24. OAI frontend - use cases @ ULB
Use case 1: Vlink - OpenURL resolver system
Feed Vlink Knowledge Base through OAI harvesting
VLink
Vlink
Knowledge Base
Unicorn
OAI-PMH
verb=ListRecords&metadataPrefix=mhld21&set=elper

25. OAI frontend - use cases @ ULB
Use case 2: Unicat - Virtual Union Catalog of Belgium
University
library
Catalog
Unicorn Aleph VIRTUA VUBIS
End User
Unicat WWW
Gateway
Unicat Indexer
Unicat Harvester
Search/
Browse
indexes
Union OAI
Archive
OAI
SRU
Public
Museum
Other
Central Repository
Data providers
HTML

26. SRU : the protocol Client System Unicorn Server
Web Server
SRU Frontend
Catalogue database
[ Records and
indexes ]
HTTP
SRU request
HTTP SRU response
XML
Client System
Unicorn Server
Communication protocol HTTP
Information exchange protocol SRU
Returned record structure XML

27. SRU: the protocol
‘Client searches and retrieves metadata records from an archive’
Stateless protocol: sequence of SRU requests/responses over HTTP
Search and Retrieve (<-> OAI: harvesting)

28. SRU: the protocol SRU requests HTTP GET requests Syntax
BASE URL
host + port + path of SRU request handler
key=value pairs
3 possible requests (operations)
explain
serves to record facilities available at an SRU server
used by clients to self-configure
returned explain record is in XML and follows the ZeeRex Schema
Example: _
scan
allows the client to request a range of the available terms at a given point within a list of indexed terms
enables clients to present an ordered list of values and, if supported, how many hits there would be for a search on that term
searchRetrieve

29. SRU: the protocol searchRetrieve operation
searchRetrieve (principal) parameters
Version: (of the request); current protocol version: 1.1
query: query expressed in CQL
startRecord: position within the sequence of matched records of the first record to be returned
maximumRecords: number of records requested to be returned
recordSchema: schema requested for the records to be returned
stylesheet: URL for an xml stylesheet. The client requests that the server simply return this URL in the response.
CQL
« Traditionally, query languages have fallen into two camps: Powerful, expressive languages, not easily readable nor writable by non-experts (e.g. SQL, PQF, and XQuery);or simple and intuitive languages not powerful enough to express complex concepts (e.g. CCL and google). CQL tries to combine simplicity and intuitiveness of expression for simple, every day queries, with the richness of more expressive languages to accomodate complex concepts when necessary. » (

30. SRU: the protocol searchRetrieve operation Examples of CQL queries:
dinosaur title = "complete dinosaur" title exact "the complete dinosaur" dinosaur not reptile dinosaur and bird or dinobird publicationYear < 1980
title all "complete dinosaur"
title contains all of the words: ‘complete’, and ‘dinosaur’
title any "dinosaur bird reptile"
title contains any of the words: ‘dinosaur’, ‘bird’, or ‘reptile’
ribs prox/distance<=5 chevrons
a more specific proximity query: ‘ribs’ within 5 words of ‘chevrons’

31. SRU: the protocol searchRetrieve operation -- examples_ _ _
all "einstein albert“ _
all "einstein albert“ _
all "einstein albert“&stylesheet= _
all "einstein albert“&stylesheet= _

32. SRU frontend for Unicorn
Web Server
SRU Frontend
Catalogue database
[ Records and
indexes ]
HTTP
SRU request
HTTP SRU response
XML
Client system
Unicorn Server

33. SRU frontend for Unicorn
Z Frontend
Catalogue database
[ Records and
indexes ]
SRU/Z Gateway
Web Server
HTTP
SRU request
Z3950
Z3950 request
HTTP SRU response
XML
Z3950
Z3950 response
Client system
SRU/Z39.50
Unicorn Server

34. SRU frontend for Unicorn
SRU/Z39.50 Gateway: YAZ Proxy (Index Data)
Implemented at ULB: 7/2006 (2 days)
config.xml
<target name="cible" default="1">
<url>bib7.ulb.ac.be:2200</url>
<xi:include href="explain.xml"/>
<cql2rpn>pqf.properties</cql2rpn>
</target>
<target name=“slavko" default="1">
<url>velma.library.mun.ca:2200</url>
<xi:include href="explain.slavko.xml"/>
<cql2rpn>pqf.slavko.properties</cql2rpn>
explain.xml
ZeeRex XML record as response to ‘explain’ operation
pqf.properties
specifies the mapping of various CQL indexes, relations, etc. into Type-1 query attributes

35. SRU frontend for Unicorn
YAZ Proxy
version=1.1&operation=searchRetrieve&maximumRecords=10&startRecord=1& query=title all "einstein albert“& stylesheet= _
version=1.1&operation=searchRetrieve&maximumRecords=10&startRecord=1& query=title all "einstein albert“& stylesheet= _

36. SRU frontend : use case @ ULB
Seamless integration of catalog searches in CMS
Typo3
Example
HTML page containing biography of famous belgian historian Henri Pirenne
frame pointing to the following URL:
version=1.1&operation=searchRetrieve&maximumRecords=10&startRecord=1& query=pirenne%20and%20epub-dnu-* &stylesheet=
Project
Unicorn contains descriptions of databases, websites, etc with local thematic classification codes in 653
create thematic websites within our CMS, containing frames that list available databases per theme