1. 1 William Y. Arms Cornell University October 25, 2002 The National Science Digital Library (NSDL) as an Example of Information Science Research

2. 2 Some Light Reading William Y. Arms, "Economic models for open-access publishing." iMP, March 2000. http://www.cisp.org/imp/march_2000/03_00arms.htm William Y. Arms, "Automated digital libraries." D-Lib Magazine, July/August 2000. http://www.dlib.org/dlib/july20/07contents.html William Y. Arms, "What are the alternatives to peer review? Quality control in scholarly publishing on the web." Journal of Electronic Publishing, 8(1), August 2002. http://www.press.umich.edu/jep/08-01/arms.html William Y. Arms, et al., "A Spectrum of Interoperability: The Site for Science Prototype for the NSDL." D-Lib Magazine, 8(1), January 2002. http://www.dlib.org/dlib/january02/arms/01arms.html

3. 3 A Scenario A faculty member wished to find a paper for students to read in a class. He began by asking an expert. She suggested the original research paper as suitable. Later, he typed a few terms into Google, browsed the hits, selected one that led to ResearchIndex, found the paper, and downloaded a PDF version from the author's web site.

4. 4 Society Cognitive Studies HCI Viewpoints Computer Science

5. 5 HCI: Eye Tracking

6. 6

7. 7 Society Cognitive Studies HCI Computer Science Applications Information Science

8. 8 Open Access to Scientific, Scholarly and Professional Information

9. 9 Before the Web Access to Scientific, Medical, Legal Information In the United States: excellent if you belonged to a rich organization (e.g, a major university) very poor otherwise (e.g., most K-12 schools) In many countries of the world: very poor for everybody

10. 10 Research Libraries are Expensive library materials buildings & facilities staff

11. 11 Baumol's Cost Disease Year Price 1900 19502000 Bundle of goods and services Labor-intensive services Manufactured goods 2050

12. 12 Baumol's Cost Disease Year Price 1900 19502000 Bundle of goods and services Labor-intensive services Manufactured goods 2050 Moore's Law

13. 13 Brute Force Computing Few people really understand Moore's Law Computing power doubles every 18 months Increases 100 times in 10 years Increases 10,000 times in 20 years Simple algorithms plus immense computing power can outperform human intelligence

14. 14 Example: Catalogs and Indexes Cost disease: catalogs and indexes Catalog, index and abstracting records are very expensive when created by skilled professionals Moore's Law: automatic indexing of full text Retrieval effectiveness using automatic indexing can be at least as effective as manual indexing with controlled vocabularies (Cleverdon 1967, reporting on experiments by Salton)

15. 15 Brute Force Computing: Substitutes for Human Intelligence Automated algorithms for information discovery Similarity of two documents Vector space and statistical methods (Salton, Sparc Jones, et al.) Importance of digital object Rank importance of web pages by analysis of the graph of web links (Kleinberg, Page, et al.)

16. 16 Information Discovery: 1992 and 2002 19922002 Contentprintdigital Computingexpensiveinexpensive Choice of contentselectivecomprehensive Index creationhumanautomatic Frequencyone timemonthly Vocabularycontrollednot controlled Query Booleanranked retrieval Userstraineduntrained

17. 17 Brute Force Computing: Automated Metadata Extraction Informedia (Carnegie Mellon) Automatic processing of segments of video, e.g., television news. Algorithms for: dividing raw video into discrete items generating short summaries indexing the sound track using speech recognition recognizing faces (Wactlar, et al.)

18. 18

19. 19 Simple algorithms plus immense computing power plus the intelligence of the user can replace labor-intensive services Cognitive Studies HCI Brute Force Computing + Intelligence of the User Computer Science

20. 20 The National Science Foundation's National Science Digital Library (NSDL) http://www.nsdl.org

21. 21 Scope All digital information relevant to any level of education in any branch of science. Scientific and technical information Materials used in education Materials tailored to education

22. 22 All branches of science, all levels of education, very broadly defined: Five year targets 1,000,000 different users 10,000,000 digital objects 10,000 to 100,000 independent sites How Big might the NSDL be?

23. 23... to provide a coherent set of collections and services across great diversity The Integration Task...

24. 24 Resources Integration team Budget $4-6 million Staff 25 - 30 Management Diffuse How can a small team, without direct management control, create a very large-scale digital library?

25. 25 It is possible to build a very large digital library with a small staff. But... Every aspect of the library must be planned with scalability in mind. Some compromises will be made. Philosophy

26. 26 Example 1: The Mortal behind the Portal [This space left intentionally blank.]

27. 27 Example 2: Interoperability The Problem Conventional approaches require partners to support agreements (technical, content, and business) But NSDL needs thousands of very different partners... most of whom are not directly part of the NSDL program The challenge is to create incentives for independent digital libraries to adopt agreements

28. 28 Function Versus Cost of Acceptance Function Cost of acceptance Many adopters Few adopters

29. 29 Example: Textual Mark-up Function Cost of acceptance SGML ASCII HTML XML

30. 30 The Spectrum of Interoperability LevelAgreementsExample FederationStrict use of standardsAACR, MARC (syntax, semantic, Z 39.50 and business) HarvestingDigital libraries exposeOpen Archives metadata; simplemetadata harvesting protocol and registry GatheringDigital libraries do not Web crawlers cooperate; services mustand search engines seek out information

31. 31 Example 3: Searching Basic Assumptions The integration team will not manage any collections The integration team will not create any metadata

32. 32 Effective Information Retrieval Comprehensive metadata with Boolean retrieval (e.g., monograph catalog). Can be excellent for well-understood categories of material, but requires expensive metadata, which is rarely available. Full text indexing with ranked retrieval (e.g., news articles). Excellent for relatively homogeneous material, but requires available full text. Full text indexing with contextual information and ranked retrieval (e.g., Google). Excellent for mixed textual information with rich structure. Contextual information without non-textual materials and ranked retrieval (e.g., Google image retrieval). Promising, but still experimental.

33. 33 Full Text or Metadata? Full text indexing is excellent, but is not possible for all materials (non-textual, no access for indexing). Comprehensive metadata is available for very few of the materials. What Architecture to Use? Few collections support an established search protocol (e.g., Z39.50). The NSDL Search Service

34. 34 Broadcast Searching does not Scale User interface server User Collections

35. 35 Users Collections Metadata repository The Metadata Repository Services The metadata repository is a resource for service providers. It holds information about every collection and item known to the NSDL, including contextual information.

36. 36 The Metadata Repository as a Resource Records are exposed through Open Archives Initiative protocol for metadata harvesting. Core Integration team provides some services based on the metadata repository. The architecture encourages others to build services. Support for Service Providers

37. 37 Search Service Portal Search and Discovery Services Collections SDLIP OAI http Metadata repository James Allan, Bruce Croft (University of Massachusetts, Amherst)

38. 38 Where is the Center of the Universe? NSDL Alexandria Elsevier Informedia Library of Congress Joe's Pictures Math DL

39. 39 Where is the Center of the Universe? NSDL British Library Elsevier OCLC Library of Congress Internet Archive Harvard

40. 40 Where is the Center of the Universe? NSDL email Course web sites News and weather Bill Arms Office Technical documentation Google Directories

41. 41 The NSDL is a program of the National Science Foundation's Directorate for Education and Human Resources, Division of Undergraduate Education. The NSDL Core Integration is a collaboration between the University Center for Atmospheric Research (Dave Fulker), Columbia University (Kate Wittenberg) and Cornell University (Bill Arms). The Technical Director is Carl Lagoze (Cornell University). Acknowledgement