1. 1 Digging into Digital Libraries: From Archaeology to Formalism Edward A. Fox Virginia Tech, Dept. of CS fox@vt.edu CSC Spring Colloquium Villanova – February 20, 2006

2. Acknowledgements (selected) 5S Helpers: Weiguo Fan, Marcos Gonçalves, Doug Gorton, Rohit Kelapure, Neill Kipp, Uma Murthy, Ananth Raghavan, Rao Shen, Hussein Suleman, Srinivas, Vemuri, Layne Watson, … Sponsors: ACM, AOL, CAPES, DFG, IBM, Microsoft, NSF (IIS-9986089, 0086227, 0080748, 0325579, 0535057, 0535060; ITR- 0325579; DUE-0121679, 0136690, 0121741, 0333601), SUN

3. 3 Outline WWW and Digital Libraries (DLs) Minimal DLs Powerful DLs Why How Summary and Conclusions

4. 4 WWW and DLs Both emerged in early 1990s. Convergence began around 1994. Example: Google spun off from Stanford DL. Crawling WWW is one way to build DLs. WWW support many portals to DLs. Parts of WWW that have catalogs (e.g., Yahoo categories) are close to DLs. Web Services help move WWW toward DLs, as the Semantic Web emerges.

5. 5 Degree of Structure Chaotic OrganizedStructured WebDLsDBs

6. 6 NSDL Information Architecture Essentially as developed by the Technical Infrastructure Workgroup referenced items & collections referenced items & collections Special Databases NSDL Services NSDL Services Other NSDL Services CI Services annotation CI Services discussion CI Services personalization CI Services authentication CI Services browsing Core Services: information retrieval Core Collection- Building Services harvesting Core Collection- Building Services protocols Core Services: metadata gathering Portals & Clients Portals & Clients Portals & Clients Usage Enhancement Collection Building User Interfaces NSDL Collections NSDL Collections NSDL Collections Core NSDL “Bus”

7. 7

8. 8

9. 9

10. 10 Outline WWW and Digital Libraries (DLs) Minimal DLs –Definitions –ETANA example Powerful DLs Why How Summary and Conclusions

11. 11 Minimal Digital Libraries Key concepts, core ideas Minimalist perspective Underlying concepts: 5S (ETANA example) Higher DL constructs Bases: –Literature –Informal explanations –Formal definitions

12. 12 Informal 5S & DL Definitions DLs are complex systems that help satisfy info needs of users (societies) provide info services (scenarios) organize info in usable ways (structures) present info in usable ways (spaces) communicate info with users (streams)

13. 13 5Ss SsExamplesObjectives Streams Text; video; audio; image Describes properties of the DL content such as encoding and language for textual material or particular forms of multimedia data Structures Collection; catalog; hypertext; document; metadata Specifies organizational aspects of the DL content Spaces Measure; measurable, topological, vector, probabilistic Defines logical and presentational views of several DL components Scenarios Searching, browsing, recommending Details the behavior of DL services Societies Service managers, learners, teachers, etc. Defines managers, responsible for running DL services; actors, that use those services; and relationships among them

14. 14 Example of 5Ss: ETANA-DL Archaeological DL (Electronic Tools for Ancient Near Eastern Archaeology Digital Library) Integrated DL –Heterogeneous data handling Applies and extends the OAI-PMH –Open Archives Initiative Protocol for Metadata Handling Design considerations –Componentized –Extensible –Portable –Work based on 5S framework

15. 15

16. 16 ETANA Societies 1.Historic and pre-historic societies (being studied) 2.Archaeologists (in academic institutes, fieldwork settings, or local and national governmental bodies) 3.Project directors 4.Technical staff (consisting of photographers, technical illustrators, and their assistants) 5.Field staff (responsible for the actual work of excavation) 6.Camp staff (e.g., camp managers, registrars, tool stewards) 7.General public (e.g., educators, learners, citizens)

17. 17 ETANA Societies – cont’d Social issues 1.Who owns the finds? 2.Where should they be preserved? 3.What nationality and ethnicity do they represent? 4.Who has publication rights? 5.What interactions took place between those at the site studied, and others? What theories are proposed by whom about this?

18. 18 ETANA Scenarios 1.Life in the site in former times 2.Digital recording: the planning stage and the excavation stage 3.Planning stage: remote sensing, fieldwalking, field surveys, building surveys, consulting historical and other documentary sources, and managing the sites and monuments 4.Excavation 1.Detailed information is recorded, including for each layer of soil, and for features such as pole holes, pits, and ditches. 2.Data about each artifact is recorded together with information about its exact find spot. 3.Numerous environmental and other samples are taken for laboratory analysis, and the location and purpose of each is carefully recorded. 4.Large numbers of photographs are taken, both general views of the progress of excavation and detailed shots showing the contexts of finds. 5.Organization and storage of material 6.Analysis and hypotheses generation and testing 7.Publications, museum displays 8.Information services for the general public

19. 19 ETANA Spaces 1.Geographic distribution of found artifacts 2.Temporal dimension (as inferred by archaeologists) 3.Metric or vector spaces 1.used to support retrieval operations, and to calculate distance (and similarity) 2.used to browse / constrain searches spatially 4.3D models of the past, used to reconstruct and visualize archaeological ruins 5.2D interfaces for human-computer interaction

20. 20 ETANA Structures 1.Site Organization 1.Region, site, partition, sub-partition, locus, … 2.Temporal orderings (ages, periods) 3.Taxonomies 1.for bones, seeds, building materials, … 4.Stratigraphic relationships 1.above, beneath, coexistent

21. 21 ETANA Streams 1.successive photos and drawings of excavation sites, loci, unearthed artifacts 2.audio and video recordings of excavation activities and discussions 3.textual reports 4.3D models used to reconstruct and visualize archaeological ruins.

22. 22 5S and DL formal definitions and compositions (April 2004 TOIS)

23. 23 Digital Object Repository Collection Minimal DL Metadata Catalog Descriptive Metadata Specification A Minimal DL in the 5S Framework Structural Metadata Specification StreamsStructuresSpacesScenariosSocieties indexing browsing searching services hypertext Structured Stream

24. 24

25. 25 Outline WWW and Digital Libraries (DLs) Minimal DLs Powerful DLs –Services –Ontology Why How Summary and Conclusions

26. 26

27. 27 Ontology: Applications

28. 28 Ontology: Applications Expand definition of minimal DL by characterizing –typical DL services –in the context of “employs” and “produces” relationships Use characterization to: –Reason about how DL services can be built from other DL components –As well as be composed with other services through extension or reuse

29. 29 Composition of key fundamental / infrastructure services

30. 30

31. 31 Outline WWW and Digital Libraries (DLs) Minimal DLs Powerful DLs Why –Support DL education –Practical systems –Institutional repositories (DSpace) –Personal DLs (SenseCam -> Memex) –Support archaeology How Summary and Conclusions

32. 32 DL Curriculum Framework

33. 33 Foundations for Information Systems: Digital Libraries and the 5S Framework Ch. 1. Introduction (Motivation, Synopsis) Part 1 – The “Ss” Part 2 – Higher DL Constructs Part 3 – Advanced Topics Appendix

34. 34 Book Parts and Chapters - 1 Ch. 1. Introduction (Motivation, Synopsis) Part 1 – The “Ss” –Ch. 2: Streams –Ch. 3: Structures –Ch. 4: Spaces –Ch. 5: Scenarios –Ch. 6: Societies

35. 35 Book Parts and Chapters - 2 Part 2 – Higher DL Constructs –Ch. 7: Collections –Ch. 8: Catalogs –Ch. 9: Repositories and Archives –Ch. 10: Services –Ch. 11: Systems –Ch. 12: Case Studies

36. 36 Book Parts and Chapters - 3 Part 3 – Advanced Topics –Ch. 13: Quality –Ch. 14: Integration –Ch. 15: How to build a digital library –Ch. 16: Research Challenges, Future Perspectives Appendix –A: Mathematical preliminaries –B: Formal Definitions: Ss –C: Formal Definitions: DL terms, Minimal DL –D: Formal Definitions: Archeological DL –E: Glossary of terms, mappings

37. 37 Practical Systems Commercial: IBM, VTLS, … Open Source –Greenstone –CWIS (for NSDL) –Institutional repositories DSpace Fedora

38. 38 Institutional Repositories “A university-based institutional repository is a set of services that a university offers to the members of its community for the management and dissemination of digital materials created by the institution and its community members. It is most essentially an organizational commitment to the stewardship of these digital materials, including long-term preservation where appropriate, as well as organization and access or distribution.” Lynch, C.A. In ARL Bimonthly Report 226, pp. 1-7, Feb. 2003, www.arl.org/newsltr/226/ir.html

39. 39

40. 40 ETANA-DL Global Architecture DigBase and DigKit Lahav Nimrin Umayri Hisban Megiddo Jalul New Sites DATABASEWRAPPERSDATABASEWRAPPERS ETANA-DL UNION CATALOG Search USERINTERFACEUSERINTERFACE Browse Recommend Note Personalize Review Visualizations Archaeology Specific Work in progress …

41. 41 Megiddo Opening Screen

42. 42 Locus Screen: Pictures View all

43. 43 Area Screen

44. 44 Repository1 DL1 Repository2 Union Catalog Union Repository Catalog1Catalog2 Searching Union DLDL2 archaeologists Society General Public Society Archaeologists General Public Union Society Service Browsing Service Union Service Harvesting, Mapping, Searching, Browsing, Clustering, Visualization Global DL: Architecture of a Union DL

45. 45 Outline WWW and Digital Libraries (DLs) Minimal DLs Powerful DLs Why How –Components –Metamodels, Models –Graphical model building aids –DL generators –Integration –Quality Summary and Conclusions

46. 46 1010100101 0100101010 1001010101 0101010101 Program 1010100101 0100101010 1001010101 0101010101 Document 1010100101 0100101010 1001010101 0101010101 Document 1010100101 0100101010 1001010101 0101010101 Document 1010100101 0100101010 1001010101 0101010101 Program 1010100101 0100101010 1001010101 0101010101 Program 1010100101 0100101010 1001010101 0101010101 Image 1010100101 0100101010 1001010101 0101010101 Image 1010100101 0100101010 1001010101 0101010101 Image 1010100101 0100101010 1001010101 0101010101 Video 1010100101 0100101010 1001010101 0101010101 Video 1010100101 0100101010 1001010101 0101010101 Video componentized digital library ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

47. 47 Discovery Current Awareness Preservation Service Providers Data Providers Metadata harvesting The World According to OAI: Open Archives Initiative – Protocol for Metadata Harvesting

48. 48

49. 49

50. 50 Metamodels Completed –Minimal –Archaeological Planned –Practical –System oriented Doug Gorton’s thesis, so people can build models for their systems, and have them generated to work with a particular DL system

51. 51 Digital Object Repository Collection Minimal DL Metadata Catalog Descriptive Metadata Specification A Minimal DL in the 5S Framework Structural Metadata Specification StreamsStructuresSpacesScenariosSocieties indexing browsing searching services hypertext Structured Stream

52. 52 5SL – The Minimal DL Metamodel

53. 53 StreamsStructuresSpacesScenariosSocieties indexing browsing searching services hypertext Structured Stream Descriptive Metadata specification SpaTemOrg StraDia Arch Descriptive Metadata specification ArchDO ArchObj ArchColl Arch Metadata catalog ArchDColl ArchDR Minimal ArchDL A Minimal ArchDL in the 5S Framework

54. 54

55. 55 Overview of 5SGraph Workspace (instance model) Structured toolbox (metamodel)

56. 56 Tools/Applications

57. 57 5SGen – Version 2: ODL, Services, Scenarios

58. 58 XML-based DL Log Standard Log analysis –is a source of information on: How patrons really use DL services How systems behave while supporting user information seeking activities Used to: –Evaluate and enhance services –Guide allocation of resources Common practice in the web setting –Supported by web servers, proxy caches DL Logging can be more detailed

59. 59 The XML Log Format Log SessionIdMachineInfo StatementTransactionTimestamp SessionInfoRegisterInfo StatementEventTimestamp Action SearchBrowse StoreSysInfoUpdate SearchBy QueryString CatalogCollection PresentationInfo StatusInfo Timeout

60. 60 DL Integration What is “DL Integration” –Hide distribution –Hide heterogeneity –Enable autonomy of individual component Why Integration –island-DLs –inability to seamlessly and transparently access knowledge across DLs Utilize various autonomous DLs in concert

61. 61 Formal Definition of DL Integration DL i =(R i, DM i, Serv i, Soc i ), 1 i n –R i is a network accessible repository –DM i is a set of metadata catalogs for all collections –Serv i is a set of services –Soc i is a society UnionRep UnionCat UnionServices UnionSociety

62. 62 Formal Definition of DL Integration (Cont.) DL integration problem definition: Given n individual libraries, integrate the n DLs to create a UnionDL.

63. 63 ETANA-DL Approach Applying and extending Digital Library (DL) techniques to solve key problems: making primary data available, data preservation, and interoperability Modeling archaeological information systems using 5S to better understand the domain and design the system and the supporting services Rapidly prototyping DLs that handle heterogeneous archaeological data using componentized frameworks: –eliciting requirements –refining metamodel and union schema –modeling sites –mapping –harvesting –providing useful services

64. 64 Example of Union Service: CitiViz

65. 65 Union Catalog Integration VN Metadata Format Global Metadata Format VN Catalog HD Catalog Union Catalog Mapping Tool Wrapper Mapping Tool Wrapper HD Metadata Format Virtual Nimrin (VN) Halif DigMaster (HD) Union ArchDL

66. 66 local schemaglobal schema

67. 67 Describing Quality in Digital Libraries What’s a “good” digital Library? –Central Concept: Quality! –Hypotheses of this work: Formal theory can help to define “what’s a good digital library” by: New formalizations of quality indicators for DLs within our 5S framework Contextualizing these measures within the Information Life Cycle

68. 68 Quality Dimensions

69. 69 Quality and the Information Life Cycle

70. 70 Summary and Conclusions WWW and Digital Libraries (DLs) Minimal DLs Powerful DLs Why How -> Theory-based discipline and high quality DL management systems (DLMS)

71. 71 Selected Links - http://fox.cs.vt.edu CITIDEL (computing education resources) –www.citidel.org NCSTRL (computing technical reports) –www.ncstrl.org NDLTD (electronic theses and dissertations worldwide) –www.ndltd.org and etdguide.org NSDL (National Science Digital Library) –www.nsdl.org OAI (Open Archives Initiative) –www.openarchives.org Virginia Tech Digital Library Research Laboratory (DLRL, www.dlib.vt.edu) –5S, AmericanSouth.Org, CSTC, DL-in-a-box, ENVISION, ETANA, MARIAN, NDLTD, NSDL, OAD, ODL, …)

72. 72 Questions? Discussion? Thank You!