1 The NSDL Program Stephen Griffin National Science Foundation

2 The NSDL Program 1996 Vision articulated by NSF's Division of Undergraduate Education 1997 National Research Council workshop 1998 Preliminary grants through Digital Libraries Initiative SMETE-Lib workshop 1999 NSDL Solicitation Core Integration demonstration projects + 23 others funded large Core Integration System project funded 2002 More than 60 independent projects funded

3 NSF-funded Research Programs NSF Solicitation Proposals Research New ideas

4 The NSDL Program NSF's objective Build a comprehensive digital library for all aspects of science education NSF's approach Solicitation encouraged wide diversity of proposals divided into general categories Best 60+ proposals funded -- more to follow Grants allow projects flexibility Result A splendid set of projects A challenge in interoperability!

5 The NSDL: The Challenge of Scale William Y. Arms Cornell University

6 Core Integration Philosophy Scientific and technical information Materials used in education Materials tailored to education

7 It is possible to build a very large digital library with a small staff. But... Every aspect of the library must be planned with scalability in mind. Some compromises will be made. Core Integration Philosophy

8 All branches of science, all levels of education, very broadly defined: Five year targets 1,000,000 different users 10,000,000 digital objects 10,000 to 100,000 independent sites How Big might the NSDL be?

9 Resources for Core Integration Core Integration Budget $4-6 million Staff Management Diffuse How can a small team, without direct management control, create a very large-scale digital library?

10 Collections: the Basic Assumption The Core Integration team will not manage any collections

11 Collections The NSDL program funds only a fraction of the relevant collections.

12 Every Collection is Different

13... to provide a coherent set of collections and services across great diversity. The Core Integration Task...

14 Interoperability The Problem Conventional approaches to interoperability require partners to support agreements (technical, content, and business But NSDL needs thousands of very different partners... most of whom are not directly part of the NSDL program The Approach A spectrum of interoperability

15 Levels of interoperability LevelAgreementsExample FederationStrict use of standardsAACR, MARC (syntax, semantic, Z and business) HarvestingDigital libraries exposeOpen Archives metadata; simplemetadata harvesting protocol and registry GatheringDigital libraries do not Web crawlers cooperate; services mustand search engines seek out information

16 What to Index? When possible, full text indexing is excellent, but full text indexing is not possible for all materials (non-textual, no access for indexing). Comprehensive metadata is an alternative, but available for very few of the materials. What Architecture to Use? Few collections support an established search protocol (e.g., Z39.50) Searching

17 Broadcast Searching does not Scale User interface server User Collections

18 Users Collections Metadata repository The Metadata Repository Services The metadata repository is a resource for service providers. It holds information about every collection and item known to the NSDL.

19 Search Architecture Portal Search and Discovery Services Collections SDLIP OAI http Metadata repository James Allan, Bruce Croft (University of Massachusetts, Amherst)

20 The Metadata Repository as a Resource Records are exposed through Open Archives Initiative harvesting protocol. Core Integration team will provide some services based on the metadata repository. The architecture encourages others to build services. Support for Service Providers

21 Metadata Strategy Metadata is expensive The NSDL cannot afford to create it manually

22 Metadata Strategy Support eight standard formats Collect all existing metadata in these formats Provide crosswalks to Dublin Core Expose records in the metadata repository for others to harvest Concentrate on collection-level metadata Use automatic generation to augment item-level metadata

23 Collection-level Operations Material in the NSDL is selected and managed as collections: Alexandria Digital Library Cornell course web sites JISC Resource Discovery Network Joe's web page Human effort will be used to select and integrate major collections. Automated methods (e.g., web crawling) will be used to identify and integrate additional collections.

24 The Problem Material in the NSDL should be relevant. But we cannot select each item individually. The Approach Most selection and quality control decisions are made at a collection level, not at an item level. Information about quality will be maintained in a collection-level metadata record, which is stored in a central metadata repository. This metadata is made available to NSDL service providers. User interfaces can display quality information. Quality Control

25 User Interfaces The Problem Cannot handcraft every web page Must be usable on a very wide range of equipment and with a very diverse group of users The Solution Data driven portals using a channel architecture. Interfaces guide the user to understand the library. One library, many portals.

26

27

28 The Mortal behind the Portal [This space left intentionally blank.]

29 Where is the Center of the Universe? NSDL Alexandria Elsevier Informedia Library of Congress Joe's Pictures Math DL

30 Where is the Center of the Universe? NSDL British Library Elsevier OCLC Library of Congress Internet Archive Harvard

31 Where is the Center of the Universe? NSDL Course web sites News and weather Bill Arms Office Technical documentation Google Directories

32 The NSDL is a program of the National Science Foundation's Directorate for Education and Human Resources, Division of Undergraduate Education. The NSDL Core Integration is a collaboration between the University Center for Atmospheric Research (Dave Fulker), Columbia University (Kate Wittenberg) and Cornell University (Bill Arms). The Technical Director is Carl Lagoze (Cornell University). Acknowledgement and Disclaimer

33 The NSDL: The Challenge of Scale William Y. Arms Cornell University