University of Michigan’s OAIster Lessons Learned Kat Hagedorn OAIster/Metadata Harvesting Librarian University of Michigan, DLPS October 7, 2002

Overview of Issues Use of tools (harvester, transformation script) Metadata foibles Communication with data providers What users want Questions we’d like to answer

Use of Tools Using the harvester was easy and very functional Transformation script evolved as system evolved Lessons learned:  Re-harvesting set from scratch involves getting into the SQL  Scheduling harvests can be time intensive; can’t harvest more than 10 medium-size repositories at a time  Normalization and filtering routines were perfected as we progressed (e.g., removal of super-long records so ranking algorithms worked appropriately)

Metadata Foibles Inconsistent use of tags  “oai-dc” used instead of “dc”  Namespace occurrences placed willy-nilly Widely varying data values  DC Subject uses thesaurus or doesn’t  Resource types either too general or too specific XML validation errors, assumedly due to incorrect UTF-8 formatting Repositories differ in levels of specificity

Communication With Data Providers Currently, about 10% of data providers have XML validation problems Solved a number of these by contacting the data provider directly Most more than willing to look at their data again Informal community that could be made more formal (one fix helps multiple service providers)

What Users Want It depends… We know what irritates them  No browse function  Not knowing exactly how to search in a new service  Metadata inconsistencies Continual improvements to service

Questions We’d Like to Answer Duplication Restrictions Normalization Relevancy ranking “Best” answers Portal offshoots…especially as data increases

Contact Info Kat Hagedorn UM Digital Library Production Service For technical info: Mike Burek,