Developing an Ingest Service for Fedora Ryan Scherle Muzaffer Ozakca

IUDL infrastructure project 2-year project funded by University Information Technology Services to reengineer digital library infrastructure around Fedora Builds on experience with Fedora in context of EVIA Digital Archive (ethnomusicology video) 2 full-time staff, plus part-time from many others Dozens of legacy collections with roughly 100,000 objects New collections: some content-focused, some research-focused

Diversity Multiple media types Multiple brands Multiple tools

The goal Ingest Aajk fs jkflsf jkds s jfs sdkf Jkl id jid whi ahin inpa aialw hwiwl

Required features Ingest common content types: ▫ Images ▫ Paged documents ▫ Textual documents Allow for easy creation of new content types Must support several workflows ▫ Metadata or media may be primary ▫ Most objects include derived media ▫ Systematic changes to metadata may be desired ▫ May need to connect with external tools for metadata generation, validation, etc. ▫ A workflow engine may sit on top of the ingest system

Existing Ingest Tools

Criteria Ease of install Native content models Custom content models (e.g. paged) Workflow neutrality, including object modification Batch ingest Remember, we’re evaluating object ingest only, not object delivery!

But first, some disclaimers… This is not an objective evaluation, just our experiences We’re not experts in these systems We’re evaluating ingest only, not delivery! We’re evaluating ingest with a focus on our needs We believe in community

Fedora admin client Comes with Fedora Geared towards admins rather than end users No systematic way of entering data or attaching files Very flexible The only way to create disseminators Tedious

Fez End-to-End GUI system Highly customizable content models, workflow, security Customizable role and group based access control Growing community Originally developed as an Institutional Repository Many preset content models Can create “extension” metadata based on an XSD External MySQL database for workflow/vocabulary data GPL

Fez - ingest Single object ingest ▫ Through Web UI ▫ ImageMagick/JHOVE integration Bulk ingest: ▫ Upload files to a directory ▫ Also can import existing Fedora objects in bulks ▫ Templates for metadata common to all objects, manual updates for the rest ▫ Batches possible, but only one file per object No disseminators Custom metadata can be stored as a simple XML file Objects must use “compound” content model File Custom MD Fedora

Fez – object organization Content level Collection level Community level CommunityCollectionImage DOPaper DOCollection DO with Custom MD

Elated overview End to end complete system for digital collections Simple customizable metadata and a simple workflow supported GPL “Elated is a lightweight, general-purpose application for managing digital files. ELATED is built on top of the Fedora Repository System, and could be used as a digital assets management system, an institutional repository, or to meet other collection archiving, publishing and searching needs.”

Elated ingest Single object ingest ▫ Through Web UI ▫ Focused on DC metadata, custom fields can be added Multi object ingest via zipped folders and files ▫ Metadata template + manually ▫ Batches possible, but only one file per object Simple content model Manually-attached disseminators File DC + Custom MD Fedora

Elated object organization Level n Level 2 Level 1 Top level CollectionFolder Image DO PDF DO Image DO PDF DOFolder Image DO

Valet for ETDs A component of the VTLS VITAL product focused on ETD submission Allows submission of thesis and a simple workflow for approval Part of a larger framework Highly focused on ETDs

DirIngest overview Ingests objects from a structured ZIP file Highly flexible User must create METS structure by hand Doesn’t handle disseminators Can create some RELS-EXT data, but not fully flexible Cannot modify existing objects/collections Easy to use OhioLink Bulk Ingest

DirIngest CollectionImages Image File TextsText File Zip Archive METS.xml Fedora Crules.xml Content level Folder level Top level CollectionImages Image DO TextsText DO

Batch modify A method of controlling API-M with simple XML statements Can create “empty” objects and change them in systematic ways. Requires manual (or programmatic) creation of the modify scripts Can be used in conjunction with other tools…

Summary FezElatedValetDir Ingest Batch Modify Admin Client Ease of install Native CM Custom CM Workflow Neutrality Batch ingest

Indiana Ingest Tool

A structured interface between a workflow management or repository management GUI and the Fedora repository Focused on simple input formats for maximum flexibility Keeps the tools independent of the repository architecture Builds the FOXML, rather than requiring a full structure to be pre-built Binds disseminators Creates RELS-EXT relationships Can create and/or alter items in a collection Auto-generates technical metadata with JHOVE or XSLT.

Ingest Tool Fedora MODSEADPDF DatastreamsFOXML Image Cataloging ToolSheet Music Cataloging Tool JPGSIP

Performing an ingest Place source metadata in an accessible location (filesystem, website) Place media files (both master and derivative) in an accessible location Define the "collection configuration" Run the ingest process Receive report

Sample collection config file Hoagy Carmichael Correspondence paged hoagy iudl:6 {path to master images}.tif {path to dreivative images here} -thumb.jpg -screen.jpg -full.jpg {path to ead} Collection defn File defn Desc. metadata Tech. metadata What to do If item exists

Ingest Config MODSImages Link to Parent Ingest Tool Fedora Tech MD FOXML Datastreams: Images METS RELS-EXT Example – Sheet Music

Ingest Config AES31 Metadata Audio Link to Parent Ingest Tool Fedora Tech MD FOXML Datastreams: Images METS RELS-EXT Example – preservation package SIP

Summary FezElatedValetDir Ingest Batch Modify Admin Client Ease of install Native CM Custom CM Workflow Neutrality Batch ingest IU Tool

Major difficulties in any ingest tool Providing flexibility in “style” of content model Matching filenames with metadata records Indicating the sequence of files in complex objects Abstracting over differing local metadata standards (even in our own collections)

Topics for future discussion What is the best structure for an ingest tool? ▫ Is our tool of interest to others? ▫ Would it be better to combine our capabilities with an existing tool? Can we agree on some core content models?

Thank You! Infrastructure project wiki: ▫ Contact info: ▫ Ryan Scherle ▫ Muzaffer Ozakca