Digital Libraries: From Theory to Applications in Education and Business ICADL 2000 – Seoul, Korea December 7, 2000 Edward A. Fox CS DLRL Internet TIC Virginia Tech, Blacksburg, VA, USA

Outline F Introduction (5S) F Education (CSTC, NDLTD) F OAI F MARIAN F Conclusions

Acknowledgements (Selected) F Conference Organizers and Sponsors F Mentors: JCR Licklider, Michael Kessler, Gerard Salton F Sponsors: Advance Auto Parts, CNI, DLF, IBM, NLM, NSF, OCLC, UNESCO, US Dept. of Ed. (FIPSE), … F VT Faculty/Staff: Tony Atkins, Debra Dudley, John Eaton, Jim Hicks, Lance Matheson, Gail McMillan, James Powell, … F VT Students: Fernando Das Neves, Robert France, Marcos Goncalves, Neill Kipp, Paul Mather, Ryan Richardson, Ohm Sornil, Hussein Suleman, Omar Vasnaik, Marc Vass, … F Visitors: Mann-Ho Lee (Korea), Byongsun Kim (Korea), Shalini Urs (India), Akira Maeda (Japan)

Internet Technology Innovation Center Supported by Virginia’s Center for Innovative Technology Statewide University Partners - Governing Board: F Christopher Newport University –William Winter, William Muir, Virginia Electronic Commerce Technology Center / Southeastern Virginia Network (VECTEC/SEVAnet) F George Mason University –Scott Martin, Internet Multimedia Center (ICM) –Steven Ruth, International Center for Applied Studies in IT (ICASIT) F University of Virginia –Alf Weaver, Internet Commerce Group (InterCom) –Jim French, Internet Digital Library F Virginia Tech –Edward Fox, Digital Library Research Laboratory (DLRL), CC, CS –Scott Midkiff, Center for Wireless Telecomm. (CWT), VTISC, ECpE

JCDL 2001  First Joint ACM/IEEE Conference on Digital Libraries (+ NSF DLI-2 PI mtg) F F June 24-28, 2001 in Roanoke, VA F Conference Committee: F General Chair: Edward A. Fox, Virginia Tech F Program Chair: Christine Borgman, UCLA F Treasurer: Neil Rowe, Naval Postgraduate School F Posters Chair: Craig Nevill-Manning, Rutgers U.

URLs   (DLRL)  (Courseware)  &  (CSTC and JERIC)  (OAI)  (JCDL’2001 – June 24-28)

Collaboration! U.S. – Korea Joint Workshop on Digital Libraries San Diego Supercomputer Center August 10 & 11, 2000 Sponsored by National Science Foundation, USA Ministry of Information & Communication, Korea Institute of Information Tech. Assessment, Korea San Diego Supercomputer Center University of Maryland Virginia Tech

Workshop Participants (1 of 3) Robert AllenUniversity of Dookwon BaikKorea Ching-Chih Chen Simmons College, Su-Shing ChenUniversity of Missouri - Jonghoon ChunMyongji Gregory CraneTufts Lois DelcambreOregon Graduate Edward FoxVirginia Michael GertzUniversity of California, Stephen Helmreich New Mexico State

Workshop Participants (2 of 3) Ulf HermjakobUSC Information Sciences Soon Joo HyunInformation & Communications University (ICU) Hyeon KimKorea Research & Development Information Center Sung-Hyuk KimSookmyung Women’s Yongchae KimMinistry of Information & Communication Ron LarsenUniversity of Sang-goo LeeSeoul National Sang Ho LeeSoongsil Young-Suk LeeMIT, Lincoln Karl LoUniversity of California, San

Workshop Participants (3 of 3) Bruce MillerUniversity of California, San Sung Been Moon Yonsei Reagan MooreSan Diego Supercomputer Sung Hyon Myaeng Chungnam National KR Gang-Tak OhNational Computerization Agency, Sam-Gyun OhSungKyunKwan Hae-Chang RimKorea Shalini UrsUniversity of Lee ZiaNational Science

Some Observations F So many conferences! Lots of R&D! F Exhibits: a DL industry is emerging. F But: we don’t cite each other’s works; F nobody is asking “Why”; F we are not connecting theory + projects; F nobody is talking about OAI. F So, I’ve redone my talk, since you can see: –paper in proceedings –demo tomorrow (p. 327) and online –see tutorial notes (in book) and online

DL = Users Direct (Organized Artifact Mediated Communication) Author Reader Digital Library EditorReviewer Teacher Learner Librarian Sponsor Publisher

DL = Users Direct (Organized Artifact Mediated Communication) Sales Agent Inventory Digital Library Sales Partners Parts Supplier Training Home Garages Shopper Store Repair Manuals B2B B2C Staff

CS 6604: Digital Libraries (Fall 2000) DL of Images of Birds for Virginia Tech Museum of Natural History Student Team Ameya Datey Aniket Sule Supriya Angle Balaprasuna Chennupati and the Eagle Scouts Under the guidance of Dr. Edward Fox Ms. Llyn Sharp (VT Museum of Natural History) Mr. Anthony Atkins (Digital Library and Archives) Plus, 3-D VTMNH minerals in UH3004

Libraries of the Future JCR Licklider, 1965, MIT Press: Unified Theory? F Not ready in 1960s F Analog – unified field theory in physics F “Mess” today – segmented field, specialities –Database Knowledge Content Mgmnt –Multimedia, Hypermedia, Hypertext –Logic, Algebra, Artificial Intelligence, … F Expensive, annoying for users –Don’t know where to look –Don’t know how to use services

5S Layers Societies Scenarios Spaces Structures Streams

Definition: Digital Libraries are complex systems that F help satisfy info needs of users (societies) F provide info services (scenarios) F organize info in usable ways (structures) F present info in usable ways (spaces) F communicate info with users (streams)

Definition: 5S Framework F Societies: interacting people (, computers) F Scenarios: services, functions, operations, methods F Spaces: domains + constraints (e.g., distance, adjacency): 2D, vector, probability F Structures: relations, trees, nodes and arcs F Streams: sequences of items (text, audio, video, network traffic) F (5 Element System: Fire, Wood, Earth, Metal, Water)

5S: Combinations F Societies + Scenarios = user model F Societies + Scenarios + Spaces = user interface F Streams + Structures = markup F Streams + Structures + Scenarios = object F Structures + Scenarios = DBMS

Outline F Introduction (5S) F Education (CSTC, NDLTD) F OAI F MARIAN F Conclusions

NSDL Spine full-service collections full-service collections NSDL Collections referenced items & collections referenced items & collections Referenced Items & Collections NSDL Services NSDL Services Other NSDL Services CI Services discussion CI Services personalization CI Services topic-map registry CI Services query transform Core Collection- Usage Services annotation Core Collection- Building Services protocol mediation Core Collection- Building Services persistence Core Collection- Building Services harvesting Portals & Clients Portals & Clients Portals & Clients (Slide from Dave Fulker, Bill Arms – 11/2/2000)

ARIADNE Screens (E. Duval)

CS Teaching Center (CSTC) F Instead of building large, expensive multimedia packages, that become obsolete and are difficult to re-use, concentrate on small knowledge units. F Learners benefit from having well-crafted modules that have been reviewed and tested. F Use digital libraries to build a powerful base of support for learners, upon which a variety of courses, self-study tutorials & reference resources can be built. F ACM Education Board and SIG support, new NSF grant with UNCW, Eduprise, TCNJ, … - iLumina Project F ACM J. of Educational Resources in Computing (JERIC)

Browsing (1)

Browsing (2)

A Digital Library Case Study F Domain: graduate education, research F Genre: ETDs = electronic theses & dissertations F Submission: F Collection: Project: Networked Digital Library of Theses & Dissertations (NDLTD – remember: ND LTD / NDL TD) (also, newer NUDL: Networked University Digital Library, with e-courseware, etc.)

ETD Initiative (and UMI) Students Learn about DL, EPub TDs become more expressive N. Amer. (T)Ds are accessible, archived Global TDs become more accessible, archived UMI Universities

Library Catalogs ETD, Access is Opened to the New Research WWW NDLTD

What are the long term goals? F Attract all TDs/yr: 50K D-US, 25K D-Germany, 10K TD-Canada, … F >200K/yr rich hypermedia ETDs that may turn into electronic portfolios (images, video, audio, …) F Dramatic increase in knowledge sharing: literature reviews, bibliographies, … F Services providing lifelong access for students: browse, search, prior searches, citation links F Hundreds/thousands of downloads / year / work

The Networked Digital Library of Theses and Dissertations Leader of the Worldwide ETD (Electronic Thesis and Dissertation) Initiative Training Authors Expanding Access Preserving Knowledge Improving Graduate Education Enhancing Scholarly Communication Empowering Students & Universities

Outline F Introduction (5S) F Education (CSTC, NDLTD) F OAI F MARIAN F Conclusions

Why do we need the Open Archives Initiative ? F Current standards are too complicated F Information wants to be free ! F We can decouple –Running an archive (DL content collection) –Running a service (DL system / operation) F So we can have more and better archives, that build on each other F So we can have better services, that work on multiple collections

OAI: Archives of Digital Objects Archive Access Protocol Handle (ID) Digital object terms and conditions

The Open Archives Initiative a technical introduction Hussein Suleman Virginia Tech DLRL December 2000

History F Santa Fe Convention (October 1999) –Electronic pre-print community F San Antonio (July 2000), Lisbon (Sept. 2000) –Broader interest from other parties F Ithaca Meeting (September 2000) –Formulation of general-purpose protocol F OAI Open Meetings (January –Feb. 2001) –Public release of specifications

Federation vs. OAI Harvesting F Federation –Sending out queries to remote sites and combining results F Harvesting –Gathering all metadata from remote sites into a central search system –Lightweight protocol –Robust –Less network traffic –Redundant servers

Black Box OAI-ETD Perspective ISTEC (Ibero America) PhysDisNSYSU (Taiwan) ADT (Australia) BN.PT (Portugal) (Francophone) VTDissert.Online (Germany) MITOhioLINKCBUC (Catalunya) NDC (Greece) SEALS (S.Africa) CICU. Bergen (Norway) … …

Splitting Data & Services F Data Provider –Implements the OAI protocol on archive to allow external access to data F Service Provider –Uses the OAI protocol to access external archives and provide services (such as searching or linking) on their metadata

The Big Picture DL Repository 1Repository 2Repository 3Repository 4

Requirements for OAI Protocol F Unique identifiers (URNs) for each record F Date-stamp for each record when last modified/created/deleted F HTTP server with scripting ability

OAI Harvesting Protocol v1 F Operates over HTTP F HTTP Requests and XML Responses F HTTP Error codes F 6 Service requests (verbs): –Identify, ListMetadataFormats, ListSets –ListIdentifiers, GetRecord, ListRecords

Identify - Response

ListMetadataFormats - Response

GetRecord - Response

Verb: ListRecords F Retrieves metadata for multiple records F Parameters –from – start date (O) –until – end date (O) –set – set to harvest from (O) –resumptionToken – flow control mechanism (X) –metadataPrefix – metadata format (R)

ListRecords - Response

Feature: Different Metadata

Feature: Date Ranges

Feature: Resumption Token

Repository Explorer

ODU Search Service

What Next ? F In General –Cross-archive searching –Cross-archive linking, de-duping, threading –Selective Filtering –Open-DL in a Box ? F VT –The VT Digital Library –NDLTD Union Catalog

[acknowledgements] Carl Lagoze the Open Archives Initiative Herbert Van de Sompel Cornell University -- Computer Science DLF FALL FORUM 2000 – Chicago – November 18th 2000

Actions herbert van de sompel establish organizational stability for the OAI: institutional backing from CNI & DLF steering committee: policy guidance technical committee: technical specifications executive group: day to day coordination workshops: public dissemination, feedback revise specifications to allow adoption beyond preprints

low-barrier interop umbrella herbert van de sompel metadata OPACimageFTXTA&Ie-print

low-barrier interop umbrella herbert van de sompel metadata OPACimageFTXTA&Ie-print Author Title Abstract Identifer

OAI harvesting tools herbert van de sompel service provider harvester data provider repository Datestamp Identifier Set Records repositoryrepository

publication of specifications: January 2001 US Open Day, January 23rd Washington DC EC Open Day, February 2001, Berlin freeze specifications for 1 year: stable for experimentation; not definitive minimize risk for early adopters maximize chances for future interoperability across communities revision of specifications herbert van de sompel

alpha test of specs (11/ /2001) herbert van de sompel data providers: arXiv -- Los Alamos NACA -- NASA CogPrints -- U Southampton ETD -- Virginia Tech Thesis & Dissertations from WorldCat -- OCLC

data providers: HeinOnline law journals -- Cornell U TEI-lite collection -- U Tennessee STM publisher metadata -- U Illinois Resource Disovery Network -- UKOLN Open Language Archives -- U Pennsylvania Open Video Project -- U North Carolina Museum info. -- CIMI alpha test of specs (11/ /2001) herbert van de sompel

software: OAI harvesting interface to Ex Libris Aleph 500 Integrated Library System -- Ex Libris OAI harverster – Cornell U OAI harverster – Virginia Tech Open-source software capable of creating a merged catalog of metadata harvested from OAI- servers -- OCLC alpha test of specs (11/ /2001) herbert van de sompel

service providers: Repository explorer -- Virginia Tech MARIAN DL -- Virginia Tech ARC service -- Old Dominion U alpha test of specs (11/ /2001) herbert van de sompel

The Open Archives Initiative develops and promotes interoperability standards that aim to facilitate the efficient dissemination of content. New OAI mission statement herbert van de sompel The Open Archives Initiative has its roots in an effort to enhance access to e-print archives as a means of increasing the availability of scholarly communication. Continued support of this work remains a cornerstone of the Open Archives program.

The fundamental technological framework and standards that are developing to support this work are, however, independent of the both the type of content offered and the economic mechanisms surrounding that content, and promise to have much broader relevance in opening up access to a range of digital materials. [...] New OAI mission statement herbert van de sompel

Harvesting Document Metadata for Federated Search CS6604 Fall 2000 Project Presented By Avnish Kumar Chhabra

Benefits of Harvesting F Limited storage requirement F Fast search F Consistently ranked results F Improved reliability F Distributed collections are transparent to user. F Efficient use of network resources.

Design of the Solution OAI wrapper Z39.50 Wrapper Update Scheduling Query Generation Digital Library collection Parser/Updater Queries Replies New Metadata MARIAN Metadata Database Boundary of System developed

Implementation Main scheduler thread : Server, Protocol, Update Frequency SiteInfo Schedule File OAI harvester class: OAIInterface Instantiated with URL of OAI site And scheduling frequency HarvestorMonitor: Monitor for arbitrating access to network resources DL Collection OAIHandler XML Document Event Handler class Auth Sub Abs

Features of the system developed F Per-collection execution thread F Schedules updates F Encapsulation of protocol specific details F Extensibility F Control over active execution threads F Fault tolerance –Server unreachable –Failure / timeout of individual connections F Time zones and date ambiguity considered

Outline F Introduction (5S) F Education (CSTC, NDLTD) F OAI F MARIAN F Conclusions

MARIAN Layers Database Layer Search Engine Layer User Information Layer User Interface Layer User

Part of Hierarchy of MARIAN Classes

Relevant Document Structure

MARIAN-Phronesis Interoperability CS6604 Fall 2000 Project Tracy Lewis Ryan Richardson Kim Woods

MARIAN-Phronesis V1 Architectural Diagram Phron Query CGI Script Search Page Display to user Create object instance MARIANPHRONESIS Marian Query CGI Script Phron Results

MARIAN-Phronesis Login Page

Query in Español

Outline F Introduction (5S) F Education (CSTC, NDLTD) F OAI F MARIAN F Conclusions

Conclusions F Education is an important application of DLs F Having a framework and theory may lead to better (more effective) systems and broader applicability –5S –MARIAN F Interoperability is part of the DL grand challenge –OAI