1. Excellent XML – systems interoperability at the Wellcome Library EIUG 11th Conference, Stirling University 1 & 2 September 2005 Margaret Savage-Jones m.savage-jones@wellcome.ac.uk

2. Wellcome Library Systems  Millennium - Innovative Interfaces Inc. http://catalogue.wellcome.ac.ukhttp://catalogue.wellcome.ac.uk Includes online requesting from closed stack since mid 2003  Calm - Archive system – DS Ltd http://archives.wellcome.ac.ukhttp://archives.wellcome.ac.uk Online access to archive & mss holdings  Miro/MedPhoto image system – System Simulation Ltd http://medphoto.wellcome.ac.uk Online access to over 100,000 images, image retrieval & delivery

3. Underlying protocol: OAI-PMH Open Archives Initiative Protocol for Metadata Harvesting - protocol for sharing and harvesting metadata between different OAI-compliant systems Based on XML and HTTP One system (CALM or MedPhoto) exposes metadata via an OAI repository. This metadata is harvested by the other system (Millennium) and then loaded

4. Motivation With a MARC21, ISAD(G) & a bespoke image repository it was a strategic objective to make these systems interoperate Phase II of the Closed Stack project - Western Manuscripts and Archives had to be requestable online by summer 2004 XML Harvester development by Innovative with Michigan State University 2001-02. Wellcome placed an order for XML Harvester in January 2003 With CALM ver 4 it was possible to export EAD XML

5. Benefits  Online requesting - Western MSS & Archives collections  One circulation system to manage and one set of circ stats  Same interface for all online requests from stack  Archives & manuscripts like other collections  Image sets for library objects displayed in Web OPAC  User can jump from one system to another  No need to rekey user search in other system  Selective harvesting for onward record updating

6. Example: archive record (from Crick Coll.)

7. Harvested archive record in Web OPAC

8. Image of the archive item

9. Encoded Archival Description (EAD) Initially XML Harvester dealt only with EAD and needed encodinganalogs for parsing. Developed with Michigan State University (MSU) whose EAD finding aids had MARC encodinganalogs. Harvester parser read these tags. Encodinganalogs are attributes in XML records indicateing field, subfield, indicators etc. in another descriptive encoding system e.g. MARC21 equivalent to EAD tagged element

10. Archive system metadata Hierarchical, tree structure with collection and component item level records catalogued in General International Standard Archival Description, ISAD(G) Field export from CALM as default subset EAD DTD had some empty fields – had to export as “DServe Natural” XML which includes field tags. Catalog.xml output with catalog.DTD

11. Pilot – used “Haddad” catalogue XML Used small set of 87 XML Arabic records – a local variant of `MASTER’ XML DTD as a pilot to tes XML Harvester Used stylesheets to filter unwanted fields, add encodinganalogs and put 87.xml files in a web server directory ready to be harvested

13. Web crawler Harvester reaches the XML files through port 80. We added a page to the Millennium screens directory listing files with redirections to the web server folder. Harvester opened the page, scanned for `HREF’ strings which directed it to the XML records (file.xml) The XML Harvester parser read tags from encodinganalogs to create MARC21 records, writing to a file for loading

14. Redirection screen Harvester Test Mss Files Sample Screen # 2 Test to confirm if harvester can crawl files deposited on wtcalm01 002 83 82

15. Example – encodinganalogs for 856 - http://http://wisdom.welcome.ac.uk/xml/.html View full manuscript record

16. Harvested MARC21 “Haddad” record

17. Links: to PDF and Request button

18. Lessons  Arabic records would be loaded only once but records from CALM would need regular reharvesting/overlay  Need a more sophisticated approach than crawling a web directory – XML Harvester can harvest from OAI Repository and use datestamps in OAI to harvest records created, or modified in specified date range  XSLT could be used to transform records to MARC21 OAI without using encodinganalogs.

19. Archives OAI repository  Built on CALM server using freeware University of Illinois Provider service tool (Runs under Windows IIS)  Other Requirements: Microsoft 2000 server Microsoft IIS ver 4 or higher Microsoft ASP Microsoft XML Parser (MSXML) 4.0 Microsoft ActiveX Data objects and ODBC compliant datasource i.e. MS Acces97+ database Firewall access on port 80

20. Key decisions  Metadata export – chose full CALM record XML DTD (not EAD)  Matchpoint – decided to load contents of Calm RefNo field to Millennium 001 indexed in `o’ Also had to consider:  Hierarchical record level to harvest  Navigation between the two systems  Millennium parameters

21. Decision: Record level to harvest A “Collection” could consist of more than 40 boxes. Must have 1:1 record relationship to make requesting and retrieval work Decision to exclude archives Collection records & use Component level records. Each of these represent 1 item (box, folder, piece) and links to a single bib records with attached item for circulation in Millennium

22. Decision: Navigation Archivists wanted the archives (CALM) interface to offer the main search route for Western Archives & MSS User is taken from CALM record into Millennium to place their request then back to their CALM record to continue browsing their hit list - – two links were needed Forward: runs cgi script to search Millennium for corresponding bib record Back: 856 with URL link (can be inserted by Harvester)

23. Example: Links Forward: cgi script runs search of Millennium `o’ index for match on CALM RefNo value http://catalogue.wellcome.ac.uk/search/o?SEARCH=PPCRI%2FA%2F1%2F2%2F8 Back: RefNo PP/CRI/A/1/2/8 built into OAI record URL linking to CALM web front end - RefNo value built into search string http://archives.wellcome.ac.uk/DServe/dserve.exe?& dsqIni= Dserve.ini&dsqApp=Archive&dsqCmd=show.tcl& dsqDb= Catalog&dsqPos=0&dsqSearch=((text)='PP/CRI/A/1/2/8')

24. Calm XML export file - Component MS4385/4404 MS.4404 Notes and extracts on Chemistry, Volumetric Analysis, (etc.) c. 1865 Item 1 volume Bentley House Western MSS series 3 - Requestable

25. Mapping Calm XML to Marc21  Fields tags used: 001, 008, 245, 260, 500, 506, 655, 856 And 949 to make the item. Harvester inserts a 99x tag with load identification code e.g. CALM20040820225128  Found that Component records do not have `author’ which is only held at Collection level – but not a problem  `Mock’ bib and item records keyed to Millennium to: - demonstrate navigation & agree content with team - act as a benchmark when harvested records loaded

26. XSLT – eXtensible Style Language Transformation Used XSLT to split the XML single output file into 48,000 component.xml records using the as record delimiter and then transform them to MARC21 OAI records listed to XML Harvester by our OAI repository The OAI repository installed on the CALM staging server uses the University of Illinois Provider service tool - freeware

27. Millennium parameters To cope with `open’ v `closed’ archive collections – new codes were added to archives records and mapped to new Millennium branch codes which would trigger Millcirc rules New branch codes added to Request Rules, Determiner Table, WWWOPTIONS, Locations served New MATTYPE to exclude Western Mss and archives from the Asian Mss scope

28. Config file for archives record harvest @LOGLEVEL=CONFIG @DBNAME=CALM @URL=http://wtcalm02/oai/oai.asp @CREATEOVERLAYFROMURI=true @9XXMARCTAG=991 @USEOAI=true @DATE=20000606000000 @OAIFROMEMAIL=m.savage-jones@wellcome.ac.uk @SHOWMETADATA=true

29. Management interface for XML Harvester

30. Archive record: Request link to Web OPAC

31. Harvested archive record in Millennium

32. Patron login screen to place request

33. Confirmation of request

34. Interoperation sought with image system To integrate MedPhoto, a bespoke photo library system, and Millennium for seamless display and ordering of images MedPhoto holds images and records for more than 60,000 items catalogued in Millennium – Iconographic collection, archives & manuscripts, rare books etc. Specific need for Millennium User to see images associated with library objects

35. Media management interface

36. Config file for image URL harvest @LOGLEVEL=CONFIG @DBNAME=MEDPHOTO @URL=http://aquarius.wellcome.ac.uk:6969/ixbin/hixserv @RECID_MARCTAG=001 @CREATEOVERLAYFROMURI=true @9XXMARCTAG=991 @USEOAI=true @REQUIRE_EADID=false @DATE=20000606000000 @OAIFROMDATE=20050701000000 @OAIUNTILDATE=20050731000000 @OAIFROMEMAIL=m.savage-jones@wellcome.ac.uk @OAISET=bib

37. Selective Harvesting – images Harvest full “bib” set and load to Millennium populating 962s then each month request list of all new image URLs created since the last harvest with a Millennium.b number in their record. <http://medphoto.wellcome.ac.uk:6969/ixbin/hixserv?verb=ListRecords&meta dataPrefix=marc21&set=bib&from=2005-05-01&until=2005-05-31> (for records in May) <http://medphoto.wellcome.ac.uk:6969/ixbin/hixserv?verb=ListRecords&meta dataPrefix=marc21&set=bib&from=2005-06-01&until=2005-06-30> (for records in June and so on)

38. Harvesting: Image OAI repository  OAI repository built by SSL on MedPhoto server  Metadata matchpoint.b bib record no. is common element  Between Millennium and MedPhoto  XML Harvester selectively requests record set “bib” which all  Have.b nos, parses the returned list of MARC21 OAI records and creates a file of MARC records for loading  Matches on.b and overlays inserting 962 for each image  962|u holds URL for thumbnail and |e holds `launchpad`URL

39. MARC21 record ready to load File Name: DONE-MEDPHOTO_20050601192747.marc (411,392 bytes) Offset: 256 Blocks: 1 - 2 LEADER 00403nam a2200085uu 4500 DIRECTORY 001000900000 035001500009 856008000024 962018500104 991002800289 TAGS 1 000 00403nam a2200085uu 4500@ 2 001 L0027751@ 3 035 |a.b12857890@ 4 856 4 1 |uhttp://medphoto.wellcome.ac.uk/ixbin/imageserv?MIDMIRO=L0027751|zView image@ 5 962 |a000:000:URL:b0000000:000000:0:0:0:0:0:0|tImage|vn|uhttp://medphoto.wellcome.ac.uk/ixbin/hixclient.exe?MIROPAC=L0027751|ehttp://medphoto.wellcome.ac.uk/ixbin/i mageserv?MIRO=L0027751@ 6 991 |aMEDPHOTO{228}20050601192747@

41. Example: with |t default

43. “Launch pad” We saw an opportunity for further integration – used Intermediate screen – URL delivered by MedPhoto repository and loaded to 962 |e User can hotlink from this “launch pad” into image system to register, use a light box, email, download or order the image online from the image system before returning to Web OPAC

46. What we used  XML Harvester product (III)  OAI repository software  VBScript – for file splitting operation  Instant Saxon (command line XSLT processor)  Microsoft MSXML core services (e.g. ver 5)  Media Management for 962 (or load URLs to 856)  Three OAI-PMH compliant library systems  Shared Record IDs as matchpoints  Some experience of working with stylesheets  Some experience of load tables and record loading

47. Work in progress  Harvesting legacy catalogues/XML for other Asian MSS e.g.Iskander and Jain project (with Oxford University)  Complete testing and batch loading of 60,000 thumbnail and “launchpad” URLs to 962’s  Establish routines to manage updates for new, deleted or amended records – utilise OAI-PMH selective harvesting  Further automation of routines where practicable

48. Wish List/Enhancements  Global edit for 962 tag  More documentation for XML Harvester  Access to underlying harvester parameters e.g. for XSLT processor and XML parser  Automation of selective harvesting for maintenance

49. Useful links  XML http://www.w3.org/XMLhttp://www.w3.org/XML  EAD http://www.loc.gov/ead/http://www.loc.gov/ead/  OAI software http://oai.grainger.uiuc.edu/projectinfo.htmhttp://oai.grainger.uiuc.edu/projectinfo.htm  XSLT http://saxon.sourceforge.net/saxon6.4.3/instant.htmlhttp://saxon.sourceforge.net/saxon6.4.3/instant.html  http://www.openarchives.org/OAI/openarchivesprotocol.html http://www.openarchives.org/OAI/openarchivesprotocol.html  http://www.openarchives.org/OAI/2.0/guidelines-marcxml.htm http://www.openarchives.org/OAI/2.0/guidelines-marcxml.htm  OAI tutorial http://www.oaiforum.org/tutorialhttp://www.oaiforum.org/tutorial  OAI repository testing http://re.cs.uct.ac.za/http://re.cs.uct.ac.za/

50. Some example records http://catalogue.wellcome.ac.uk/record=b1465521 http://catalogue.wellcome.ac.uk/record=b1580232 http://catalogue.wellcome.ac.uk/record=b1313568http://catalogue.wellcome.ac.uk/record=b1580232 http://catalogue.wellcome.ac.uk/record=b1313568 http://catalogue.wellcome.ac.uk/record=b1613633 http://catalogue.wellcome.ac.uk/search/o?SEARCH=PPCRI%2FA%2F1%2F2%2F8

51. Excellent XML: systems interoperability at the Wellcome Library Thanks for your attention Margaret Savage-Jones Library Systems Administrator m.savage-jones@wellcome.ac.uk