1. PSU/Villanova/VT Discussion Virginia Tech’s Digital Library Research Laboratory Jan. 10, 2005 -- PSU Edward A. Fox, fox@vt.edu Virginia Tech, Blacksburg, VA 24061 USA http://fox.cs.vt.edu/talks/ http://fox.cs.vt.edu/cv.htm

3. Acknowledgements (Selected) Sponsors: ACM, Adobe, AOL, CAPES, CNI, CONACyT, DFG, IBM, Microsoft, NASA, NDLTD, NLM, NSF (IIS-9986089, 0086227, 0080748, 0325579; ITR- 0325579; DUE-0121679, 0136690, 0121741, 0333601), OCLC, SOLINET, SUN, SURA, UNESCO, US Dept. Ed. (FIPSE), VTLS

4. Acknowledgements: Faculty, Staff Lillian Cassel, Debra Dudley, Roger Ehrich, Joanne Eustis, Weiguo Fan, James Flanagan, C. Lee Giles, Eberhard Hilf, John Impagliazzo, Filip Jagodzinski, Rohit Kelapure, Neill Kipp, Douglas Knight, Deborah Knox, Aaron Krowne, Alberto Laender, Gail McMillan, Claudia Medeiros, Manuel Perez, Naren Ramakrishnan, Layne Watson, …

5. Acknowledgements: Students Pavel Calado, Yuxin Chen, Fernando Das Neves, Shahrooz Feizabadi, Robert France, Marcos Goncalves, Nithiwat Kampanya, S.H. Kim, Aaron Krowne, Bing Liu, Ming Luo, Paul Mather, Saverio Perugini, Unni. Ravindranathan, Ryan Richardson, Rao Shen, Ohm Sornil, Hussein Suleman, Ricardo Torres, Wensi Xi, Xiaoyan Yu, Baoping Zhang, Qinwei Zhu, …

6. Stepping Stones & Pathways: Improving retrieval by chains of relationships between document topics Fernando Das-Neves, Virginia Tech DLRL

7. A Little Experiment (Compare a simple query with a longer version that explicitly includes stepping stones) “Literary Style in Sherlock Holmes stories” Note: Numbers are total relevant web pages in top 20 Google results for the query made up of terms on either end of the link. VS. No. of rel. docs.

8. Another Example “What is the Relationship between Data Mining and Recommender Systems?” Naïve Results: There are many matches that are possible answers. Discussion: But, many of the pages with co-occurrences give no real information about the requested relationship. Social Networks Collaborative Filtering Recommender Systems Data Mining Recommender Systems Data Mining Machine Learning VS. 7 10 911 15

9. An Alternative Interpretation of a Query in IR: A query represents two related, separable concepts. Objective: Retrieve a sequence of documents that support a valid set of chains of relationships between the two concepts. Input: a query representing two concepts. Output: two groups of documents + a set of stepping stones (document groups, i.e., clusters) connecting the topics by pathways (relations among clusters).

10. Type of Questions Matching Alternative Interpretation Ill-defined questions, with non-enumerated answers: –“How or why is X related to Y?” –“What is the X of Y?” Even if queries with form “ give me something about X ” lead to relevant docs, it is possible to increase the quantity and quality of information in the query result, when relations are explicit (as a result of our semi-automatic method).

11. Why is this useful? Questions of this type are common. –For example, such questions often occur during research studies. –These occur often in educational settings, e.g., for homework. –These occur often in workplace settings, requiring gathering and relating of information. Handling of this type of question by current systems often is inadequate.

12. How to Build Stepping Stones and Pathways? Our approach involves a belief network, to combine content+structure in document similarity calculation, including citation and co-citation similarities. Find two relevant document sets, each related to one of the two original sub-queries. Find a diverse set of strong candidates, each connecting the two subsets, but as different as possible from other candidates. Create stepping stones by finding similar documents to those candidates; keep the clusters that are heavily cited, or whose documents are highly correlated (in all aspects). Repeat the process, finding a new stepping stone in between each pair of clusters that are weakly related, until the pathway length is too long, or the similarity is sufficient.

13. Streams, Structures, Spaces, Scenarios, and Societies (5S): A Formal Digital Library Framework and Its Applications Marcos André Gonçalves Doctoral defense Virginia Tech, Blacksburg, VA 24061 USA

14. Informal 5S Definition: DLs are complex systems that help satisfy info needs of users (societies) provide info services (scenarios) organize info in usable ways (structures) present info in usable ways (spaces) communicate info with users (streams)

15. 5Ss SsExamplesObjectives Streams Text; video; audio; image Describes properties of the DL content such as encoding and language for textual material or particular forms of multimedia data Structures Collection; catalog; hypertext; document; metadata Specifies organizational aspects of the DL content Spaces Measure; measurable, topological, vector, probabilistic Defines logical and presentational views of several DL components Scenarios Searching, browsing, recommending Details the behavior of DL services Societies Service managers, learners, teachers, etc. Defines service managers, responsible for running DL services; actors, that use those services

16. Hypotheses A formal theory for DLs can be built based on 5S. The formalization can serve as a basis for modeling and building high- quality DLs.

17. 5S Framework and DL Development (Gonçalves)

18. 5SLGen: Automatic DL Generation

19. Research Questions 1. Can we formally elaborate 5S? 2. How can we use 5S to formally describe digital libraries? 3. What are the fundamental relationships among the Ss and high-level DL concepts? 4. How can we allow digital librarians to easily express those relationships? 5. Which are the fundamental quality properties of a DL? Can we use the formalized DL framework to characterize those properties? 6. Where in the life cycle of digital libraries can key aspects of quality be measured and how?

20. Outline Motivation: the problem –Hypotheses and research questions Part 1:Theory –5S: introduction, formal definitions –The formal ontology Part 2: Tools/Applications –Language –Visualization –Generation –Logging Part 3: Quality Conclusions, Future Work

21. 5S and DL formal definitions and compositions (April 2004 TOIS)

22. Digital Library Formal Ontology

23. Composition of key infrastructure services

24. Composition of additional services

25. Ontology: Taxonomy of Services Binding Browsing Customizing Disseminating Expanding(query) Filtering Recommending Requesting Searching Annotating Classifying Clustering Evaluating Extracting Indexing Linking Logging Measuring Rating Reviewing (peer) Surveying Training (classifier) Translating Visualizing Conserving Converting Copying/Replicating Translating (format) Acquiring Authoring Cataloging Crawling (focused) Describing Digitizing Harvesting Submitting PreservationalCreational Add Value Repository-Building Information Satisfaction Services Infrastructure Services

26. 5SL: a DL Modeling language Domain specific languages –Address a particular class of problems by offering specific abstractions and notations for the domain at hand –Advantages: domain-specific analysis, program management, visualization, testing, maintenance, modeling, and rapid prototyping. XML-based realization of 5S –Interoperability –Use of many standard sub-languages (e.g., MIME types, XML Schemas, UML notations)

27. Overview of 5SGraph Workspace (instance model) Structured toolbox (metamodel)

28. 5SGen – Version 2: ODL, Services, Scenarios 5SL-Societies Model (1) XPATH/JDOM Transform (2) XMI:Class Model (3) Xmi2Java (4) Java Classes Model (5) Deterministic FSM (10) SMC (11) Java Finite State Machine Class Controller (12) 5SL-Scenario Model (6) XPath/JDOM Transform (7) StateChart Model (8) Scenario Synthesis (9) ODL Search Java Wrapping import Component Pool ODL Browse Java Wrapping import... JSP User Interface View (13) Generated DL Services DL Designer DL Designer binds 5SL-Societies Model (1) XPATH/JDOM Transform (2) XMI:Class Model (3) Xmi2Java (4) Java Classes Model (5) Deterministic FSM (10) SMC (11) Java Finite State Machine Class Controller (12) 5SL-Scenario Model (6) XPath/JDOM Transform (7) StateChart Model (8) Scenario Synthesis (9) ODL Search Java Wrapping import Component Pool ODL Browse Java Wrapping import... ODL Search Java Wrapping import Component Pool ODL Browse Java Wrapping import... JSP User Interface View (13) Generated DL Services DL Designer DL Designer binds 5SGen

29. The XML Log Format Log SessionIdMachineInfo StatementTransactionTimestamp SessionInfoRegisterInfo EventErrorInfo Action SearchBrowse StoreSysInfoUpdate SearchBy QueryString CatalogCollection PresentationInfo StatusInfo Timeout

30. Quality and the Information Life Cycle

31. Rao Shen’s Preliminary Exam: Hypothesis and Research Questions The 5S framework provides effective solutions to DL integration. –Formally define the DL integration problem? –Guide integration of domain focused DLs? How to formally model such domain specific DLs? How to integrate formally defined DL models into a union DL model? How to use the union DL model to help design and implement high quality integrated DLs? –Assess the integration?

32. Related Work DL interoperability approach Intermediary-basedmapping-based Consists of mediatorwrapperagent use two architectures federationUnion Archiving used in Consists of hybrid mappercomposite mapper use schema mapping use SemInt has an example LSD has an example Interrelated with

33. DL interoperability approach Intermediary-basedmapping-based Consists of mediatorwrapperagent use two architectures federationUnion Archiving used in Consists of hybrid mappercomposite mapper use schema mapping use Interrelated with GA trained by DL integration formalization based on

34. Formal Definition of DL Integration DL i =(R i, DM i, Serv i, Soc i ), 1 i n –R i is a network accessible repository –DM i is a set of metadata catalogs for all collections –Serv i is a set of services –Soc i is a society UnionRep UnionCat UnionServices UnionSociety

35. Repository1 DL1 Repository2 Union Catalog Union Repository Catalog1Catalog2 Searching Union DLDL2 archaeologists Society General Public Society Archaeologists General Public Union Society Service Browsing Service Union Service Harvesting, Mapping, Searching, Browsing, Clustering, Visualization Architecture of a Union DL

36. Example of Union Service: CitiViz

37. CitiViz: A Visual User Interface to the CITIDEL System ECDL 2004, Bath, England, September 2004 Nithiwat Kampanya, Rao Shen, Seonho Kim, Chris North, and Edward A. Fox fox@vt.edu http://fox.cs.vt.edu

38. Digital Object Repository Collection Minimal DL Metadata Catalog Descriptive Metadata Specification A Minimal DL in the 5S Framework Structural Metadata Specification StreamsStructuresSpacesScenariosSocieties indexing browsing searching services hypertext Structured Stream

39. StreamsStructuresSpacesScenariosSocieties indexing browsing searching services hypertext Structured Stream Descriptive Metadata specification SpaTemOrg StraDia Arch Descriptive Metadata specification ArchDO ArchObj ArchColl Arch Metadata catalog ArchDColl ArchDR Minimal ArchDL A Minimal ArchDL in the 5S Framework

40. 5SGraph 5S Archaeology MetaModel ArchDL Expert ArchDL Designer Structure Sub-model ETANA-DL Union Services Descriptions Harvesting Mapping Searching Browsing … Scenario Sub-model VN Metadata Format ETANA-DL Metadata Format HD Metadata Format Mapping Tool Wrapper4VNWrapper4HD Inverted Files Services DB Index Browse Service Search Service Browse DB Other ETANA-DL Services Web Interface XOAI VN Catalog HD Catalog Union Catalog 5SGen Component Pool Browsing …

41. Computing and Information Technology Interactive Digital Educational Library (CITIDEL) Domain: computing / information technology Genre: one-stop-shopping for teachers & learners: courseware (CSTC, JERIC), leading DLs (ACM, IEEE-CS, DB&LP, CiteSeer), PlanetMath.org, NCSTRL (technical reports), … Submission & Collection: sub/partner collections  www.citidel.org

42. www.CITIDEL.org Led by Virginia Tech, with co-PIs: –Fox (director, DL systems) –Lee (history) –Perez (user interface, Spanish support) –Students: Ryan Richardson, Kate McDevitt, Jon Pryor, Baoping Zhang Partners –College of New Jersey (Knox) –Hofstra (Impagliazzo) –Villanova (Cassel) –Penn State (Giles)

43. Digital library architecture for local and interoperable CITIDEL services

46. CITIDEL Technology Features Component architecture (Open Digital Library) Re-use and compose re-deployable digital library components. Built Using Open Standards & Technologies OAI: Used to collect DL Resources and DL Interoperability XSL and XML: Interface rendering with multi-lingual community based translation of screens and content (Spanish, …) Perl: Component Integration ESSEX: Search Engine Functionality Very fast, utilizing in-memory processing Includes snap-shots for persistence Multi-scheming (Aaron Krowne, now at Emory U. Library) Integrates multiple classifications / views through maps, closure Extensions: clustering, visualization, personalization, …

47. Cluster Search Results from CITIDEL

48. Cluster NDLTD-Computing

49. CITIDEL + PIPE Adds Interaction Personalization to CITIDEL Automatically handles multi-modal conversion to Cell phone, PDA, Etc. Can be adopted to any digital data set, only requires XML file of content with hierarchy maintained. Naren Ramakrishnan and Saverio Perugini (U. Dayton)

50. OCKHAM Library Network (NSDL)

51. OCKHAM (Ming Luo) Simplicity (a la OCCAM’s razor) Support by Mellon and DLF Four main ideas: 1.Components 2.Lightweight protocols 3.Open reference models (e.g., 5S, OAIS) 4.Community perspective and involvement Funded by NSF in NSDL, with P2P, with Emory, Notre Dame, Oregon State, …

52. OCKHAM Proposed Services Alerting Browsing Cataloging Conversion OAI – Z39.50 Pathfinding Registry (plus others such as from adapted ODL)

53. A Digital Library Case Study Domain: graduate education, research Genre:ETDs=electronic theses & dissertations Submission: http://etd.vt.edu Collection: http://www.theses.org Project: Networked Digital Library of Theses & Dissertations (NDLTD) http://www.ndltd.org (supported by Ming Luo)

57. OCLC SRU Interface => Dr. A.K. Tyagi

60. ETD Union Search Mirror Site in China (CALIS) (http://ndltd.calis.edu.cn – popular site!)

61. LOCKSS Extensions: Bing Liu, Xiaoyu Zhang, Ji-Sun Kim Lots of copies keep stuff safe Stanford (Vicky Reich) Initial focus on lower levels, journals Shift to OAI, esp. for ETDs Collab with Emory (Martin Halbert) –NDIIP: AmericanSouth, MetaArchive –Help deploy and adapt, apply in other contexts Another registry Set of publisher manifests (information providers) Set of storage systems (archival storage)

62. 1010100101 0100101010 1001010101 0101010101 Program 1010100101 0100101010 1001010101 0101010101 Document 1010100101 0100101010 1001010101 0101010101 Document 1010100101 0100101010 1001010101 0101010101 Document 1010100101 0100101010 1001010101 0101010101 Program 1010100101 0100101010 1001010101 0101010101 Program 1010100101 0100101010 1001010101 0101010101 Image 1010100101 0100101010 1001010101 0101010101 Image 1010100101 0100101010 1001010101 0101010101 Image 1010100101 0100101010 1001010101 0101010101 Video 1010100101 0100101010 1001010101 0101010101 Video 1010100101 0100101010 1001010101 0101010101 Video open digital library OA PMH XPMH Hussein Suleman (Capetown, S. Africa)

63. 1010100101 0100101010 1001010101 0101010101 Program 1010100101 0100101010 1001010101 0101010101 Document 1010100101 0100101010 1001010101 0101010101 Document 1010100101 0100101010 1001010101 0101010101 ETD-1 1010100101 0100101010 1001010101 0101010101 Program 1010100101 0100101010 1001010101 0101010101 ETD-2 1010100101 0100101010 1001010101 0101010101 Image 1010100101 0100101010 1001010101 0101010101 Image 1010100101 0100101010 1001010101 0101010101 ETD-3 1010100101 0100101010 1001010101 0101010101 Video 1010100101 0100101010 1001010101 0101010101 Video 1010100101 0100101010 1001010101 0101010101 ETD-4 ETD DL for the Networked Digital Library of Theses and Dissertations (www.ndltd.org) Search Filter Union Recent Browse PMH ODLRecent ODLBrowse ODLUnion ODLSearch ODLUnion PMH USER INTERFACE Students and researchers ETD collections Example Open Digital Library

64. Open Digital Library Deployments NDLTD (www.ndltd.org) Computer Science Teaching Center (www.cstc.org) Computing and Information Technology Interactive Digital Educational Library (www.citidel.org) Open Archives Distributed (NSF, DFG) – enhancements to PhysNet OCKHAM Open to others through DL-in-a-box

67. Interest-based User Grouping Model for Collaborative Filtering in Digital Libraries 7 th ICADL 2004 Shanghai, P.R. China Dec. 15, 2004 Edward A. Fox, Seonho Kim Virginia Tech, Blacksburg, VA 24061 USA

68. Some Other Students/Projects Wensi Xi: Matrices, reinforcement, clusters (Microsoft) Paul Mather: mod/sim of large DLs on clusters; characterization: uses, files (NASA) Ming Luo: personalization aided by demographics Ryan Richarson: CLIR with concept maps Xiaoyan Yu: Stepping Stones and Pathways (NSF, Fernando Das Neves completed & returned to Argentina) Baoping Zhang: Physics and classification (NSF, DFG) Several: TREC with GP New projects: –Superimposed information w. PSU (NSF NSDL) –Quality and metasearch and structure w. Emory (IMLS) …

69. Conclusion Many DL/IR: areas, projects, students Theory Architecture Modeling and simulation Systems development and testing to: validate above, demonstrate innovations Users, interfaces, visualization, usability