1. September 1 st 2010 Igelu Ghent 2010 1 The on-the-fly conversion circus Matthias Gross (Bavarian State Library)

2. September 1 st 2010 Igelu Ghent 2010 2 Introduction Often the original version of an object is not what the user wants … or what we want the user to get Two basic strategies: Store additional versions Offer them virtually: create them on the fly

3. September 1 st 2010 Igelu Ghent 2010 3 whats common… Common aspect: something has to be converted a)Content e.g. tiff / PDF jpg (for viewing) tiff / jpg PDF (for download) b)Structural metadata (METS) c)Bibliographic metadata e.g. MARC21 DC Common side-effect of conversion: loss of information, sometimes of functionality. In most cases affordable (hi-resolution scans), in some even wanted: -for legal reasons -reduce file size to speed up transfer

4. September 1 st 2010 Igelu Ghent 2010 4 … and whats different Benefits of on-the-fly conversion: Reduce storage costs Keep data structure simple Reduce migration costs when specifications of desired target formats change Price: Server load Runtime for conversion <= waiting time for end user e.g. tiff jpg: waiting time is usually too long to be acceptable Some formats are built to be delivered primarily via on-the- fly conversion: j2k ( jpg)

5. September 1 st 2010 Igelu Ghent 2010 5 normal presentation

6. September 1 st 2010 Igelu Ghent 2010 6 First example: serving the DFG viewer In this example, we have single page PDFs in a METS structure. Let us look at the content first. To serve the so-called DFG viewer (which projects funded by the Deutsche Forschungsgemeinschaft, DFG, are obliged to), three different JPEG versions of each page with a given resolution are needed This would not be easy to implement within DigiTool: how to encode the resolution of a VIEW manifestation so that it can be addressed from outside the system? (Basically, all VIEW manifestations are equal, with an optional VIEW MAIN) The conversion is implemented via a special viewer which calls Ghostscript/Imagemagick. Via simple caching the waiting time can be shortened when the user returns to a previously accessed page.

7. September 1 st 2010 Igelu Ghent 2010 7 presentation in the DFG viewer

8. September 1 st 2010 Igelu Ghent 2010 8 DFG viewer, continued Now let us look at the structural metadata (METS). The DFG viewer expects a different type of METS : the FileSec has to be changed significantly METS (digital entity-XML) can be quite large (>3 MB for 1700 pages) Conversion lasts up to 7 seconds on-the-fly conversion not reasonable, so we put the converted METS in the file system; access via PID (how would you store additional METS files within the DigiTool data model?)

9. September 1 st 2010 Igelu Ghent 2010 9 Next example: PDF download Let us look at the content again. We want to offer a "PDF download option. An additional manifestation (single PDF file for an IE) would need a significant amount of extra storage This is even true for caching these PDF files; besides that, each read/write-operation for such big files is expensive best performance when using the streams as they are in the repository, embedding them in PDF and give that as HTTP response further optimization: use fast PERL library and access Oracle DB directly For a 350 page book, 1 minute can be reached. Usually the internet connection of the end user is the bottleneck.

10. September 1 st 2010 Igelu Ghent 2010 10 PDF download dialogue

11. September 1 st 2010 Igelu Ghent 2010 11 PDF download result

12. September 1 st 2010 Igelu Ghent 2010 12 Next example: EuropeanaTravel

13. September 1 st 2010 Igelu Ghent 2010 13 EuropeanaTravel The overall objective of this project is to digitise content on the theme of travel and tourism to be made accessible via Europeana, the European digital library, museum and archive. Launched in November 2008, Europeana provides integrated access to digital treasures from museums, archives, audio-visual archives and libraries of Europe. Europeana EuropeanaTravel has officially started on 1st May 2009 and it will last for two years. Contribution of the University Library of Regensburg via DigiTool: 400 books 200maps 600 illustrations from books (+ detailed metadata)

14. September 1 st 2010 Igelu Ghent 2010 14 OAI Problem with transporting the metadata to Europeana via OAI: Europeana expects extended DC with special europeana: elements Yet another replica set? How often should we replicate information with just little differences in format? Replica sets demand 1.) database space 2.) significant java resources during replication

15. September 1 st 2010 Igelu Ghent 2010 15 Solution: build a set that contains all the information which is needed for different demands (granular bibliographic data, delivery URL, thumbnail URL, file type, object type, …) And then…

16. September 1 st 2010 Igelu Ghent 2010 16 filter OAI-XML through stylesheet on the fly!

17. September 1 st 2010 Igelu Ghent 2010 17 OAI for Europeana

18. September 1 st 2010 Igelu Ghent 2010 18 Travel … Facet for EuropeanaTravel Search for Reise (which means travel in German)

19. September 1 st 2010 Igelu Ghent 2010 19 … and get … link to DigiTool delivery

20. September 1 st 2010 Igelu Ghent 2010 20 … back home

21. September 1 st 2010 Igelu Ghent 2010 21 further application of this technique just 1 OAI record per object OAI for harvester 1 stylesheet 1 OAI for harvester 2 stylesheet 2 report list of objects HTML stylesheet 3 report list of objects Excel stylesheet 4 HTML page for starting services stylesheet 5

22. September 1 st 2010 Igelu Ghent 2010 22 example: thumbnail selec- tion service (post-ingest)

23. September 1 st 2010 Igelu Ghent 2010 23 Current / planned activities (1) secure PDF display option which prohibits download and printout for copyright material (at least makes that very wearisome, as you cant prevent screen snapshots) assume multi-page PDF as storage format normal PDF viewer (client-side plugin) has too many options which cant be disabled reliably We look at: -multi-page jpg (conversion via Ghostscript) -flash approach (browser plugin needed) -applet approach Observation: the images look somehow not so nice without using the Adobe stuff

24. September 1 st 2010 Igelu Ghent 2010 24 Current / planned activities (2) We will evaluate conversion to ePUB as alternative download format, possibly on the fly. Who has experience already?

25. September 1 st 2010 Igelu Ghent 2010 25 THE END … Thank you very much for your patience attention! gross@bsb-muenchen.de Special credits to Petra Schröder for most of the work, Joe Shubitowski @ Getty for thumbnail inspiration and to Wikimedia Commons for some pictures!