Federated Digital Library Architecture and Distributed Resource Discovery Carl Lagoze CS

Component-Based Digital Library Architecture Building digital libraries out of a toolkit: –modular services –protocol interface to each service –mechanisms to combine services into digital library collections Advantages: –ability to build digital libraries that conform to specific service and collection needs –ability to add new services that enhance functionality

7 Dienst is a protocol and reference implementation of a distributed digital library service where a network of services provide World Wide Web browser access, uniform search over distributed indexes, and access to structured documents.

Defining the services Repository – deposit, storage, and access to structured documents. Index – process queries on documents and returned handles Query Mediator – route queries to appropriate indexes Collection – define services and content in logical collections User Interface – human-oriented front- end for services.

Why a service based protocol? Expose the operational semantics of the services through an API, to permit flexible integration of the services, and use of the services by other clients/consumers/services.

Defining the protocol Structured messages –Service –Version –Verb –Arguments Template /Dienst/ / / [?/] Example /Dienst/Repository/4.0/Formats/ncstrl.cornell/TR

Why a Document Model? “Documents” in current web are both: –Unstructured (GET) –Chaotic (CGI) Different views and pieces of contents are needed for: –Bandwidth reduction –Rights management –Usability

Dienst Document Model Views – alternative expression or structural representation of the content encapsulated in the digital object Divs – hierarchically nested structure contained in a view Metadata – support for multiple descriptive formats

Expressing the document model in the protocol Structure – expose the views and structure for the digital object Disseminate – select the structural component (and packaging of it) to disseminate List-Meta-Formats – list available descriptive formats

Protocol Demonstration Contents?file-after= http://cs-tr.cs.cornell.edu/Dienst/Repository/4.0/List- Contents?file-after= tr.cs.cornell.edu/Dienst/Repository/1.0/Disseminate/ncstrl.cornell/ TR /%23oams/xmlhttp://cs- tr.cs.cornell.edu/Dienst/Repository/1.0/Disseminate/ncstrl.cornell/ TR /%23oams/xml tr.cs.cornell.edu/Dienst/Repository/2.0/Structure/ncstrl.cornell/T R http://cs- tr.cs.cornell.edu/Dienst/Repository/2.0/Structure/ncstrl.cornell/T R tr.cs.cornell.edu/Dienst/Repository/4.0/Formats/ncstrl.cornell/TR ?part=bodyhttp://cs- tr.cs.cornell.edu/Dienst/Repository/4.0/Formats/ncstrl.cornell/TR ?part=body tr.cs.cornell.edu/Dienst/Repository/1.0/Disseminate/ncstrl.cornell/ TR /body/inline?pageimage=3http://cs- tr.cs.cornell.edu/Dienst/Repository/1.0/Disseminate/ncstrl.cornell/ TR /body/inline?pageimage=3

11 Collection Service Periodically polled by each user interface server for –elements of the collection –index servers for the collection User Interface Servers Index Servers

Deploying Collection Globally Internet connectivity varies considerably Good connectivity between nodes often does not correspond to geographic proximity Connectivity Region - a group of nodes on the network that among them have good connectivity, relative to nodes outside of the region.

Connectivity Regions When possible route queries within region In case of failure, use an alternate either within the region or in a “nearby” region

Dienst Services WWW browser User Interface Repository Index Repository QM user query generic search request specific search request NS user document request URI document request Collection Collection metadata

NCSTRL Digital Library Demonstration

Distributed Searching - Motivation Information infrastructure with distributed overlapping indexes –scalability –connectivity –domain-specificity –intellectual property issues Research Issue: Search Request Routing

17 Broadcast Distributed Search

18 Backup Index server replicates all index servers used by user interface when primary is down backup index

19 Regional Structure central collection server regional collection server regional merged index server

Routing Problem Disjoint Indexes Hopcroft I1, I3 Hartmanis I3 Tarjan I1, I2 Wilensky I2 I1I2I3 I1,I3 doc8 doc1, doc2 Content Summary author=Hopcroft? Hopcroft doc8 Tarjan doc9 Tarjan doc6 Wilensky doc7 Hopcroft doc1, doc2 Hartmanis doc3, doc4

Routing Problem Replicated Distributed Indexes author=Hopcroft? Hopcroft doc8 Tarjan doc9 Tarjan doc6 Wilensky doc7 Hopcroft doc8 Tarjan doc9 Tarjan doc6 Wilensky doc7

Routing Issues Choice of primary?, secondary?, etc. Fault-tolerance Routing Factors –Performance-based (Cornell and Virginia) –Freshness-based (Stanford) –Cost-based (FORTH) –weighted mix based on user preference

Components of Replicated Routing Problem Metadata Issue: metadata made available by indexer to aid in routing Metadata Distribution Issue: topology of metadata repositories Decision Issue: routing decision algorithms Fault-tolerance: use of backup indexers

Distributed Metadata for Query Routing central metadata store

Performance-based Routing 8 present- T Average response time Timed low pass filter Predicted response time New = low pass filter(T, actual response time, old )