1 Strategies for Collecting and Preserving Open Access Materials on the Web William Y. Arms Cornell University Federal Library and Information Center Committee

2 Open Access Materials on the Web

3 The Library of Congress: the Web Preservation Project Library of Congress collects cultural and intellectual output of today for the benefit of future generations. An ever-increasing amount of this material is born digital. The library has: privileged legal position generous public funding... but cannot do everything! Step 1: Open Access Materials on the Web

4

5

6

7

8 Partnership with publishers Publishers and libraries as partners Selective collection of open access web Librarianship in a new domain Bulk collection of open access web Automated processes Approaches to Preservation of the Web OPEN ACCESS CLOSED ACCESS

9 Example: Web Preservation Project Pilot Small number of web sites nominated by selection officers. Three chosen for close study Copies downloaded using HTTrack mirroring program. Inspected for errors, anomalies, etc. Catalog records created using OCLC's CORC software Loaded into Library of Congress's ILS system. Trial web site developed to evaluate user interfaces.

10 Example: The Internet Archive

11 Example: National Library of Australia

12 Example: National Library of Sweden

13 Selection and Collection

14 Collecting: Making a Snapshot Web site Snapshot Download Archive A web site is downloaded, using a mirroring program. A snapshot is stored in an archive.

15 Collecting: Periodic Snapshots Web site Snapshot 1 Archive At scheduled time intervals additional snapshots are made. Snapshot 2 Snapshot 3

16 Selection Decisions Which sites to collect Bulk -- collect all within a certain category Selective -- collect sites selected by a librarian How often to make snapshots Monthly, weekly, or depending on circumstances Which content to collect HTML pages only Text and images only Everything

17 Examples of Selection Decisions SelectionFrequencyContent Internet ArchivebulkmonthlyHTML + images Pandoraselectivevariesall Kulturarw 3 bulksweepsall Web Preservationselectiveirregularall

18 Legal Issues Legal position of archives that download open access materials is unclear Preservation is in the national interest See the discussion in The Digital Dilemma Crucial factor is economic impact on copyright owners Library of Congress has no special position except via copyright deposit

19 Legal Issues: Thoughts and Actions Presumption is that downloading open access materials is permitted by the publisher unless other indication given, e.g., robot exclusion using robots.txt file Different parties to consider => Library of Congress => other national libraries => partners of the Library of Congress and national libraries => independent archives U.S. Copyright Office has offered to help clarification

20 Access to Collections

21 Access: Analysis by Computer Snapshot 1 Archive Snapshot 2 Snapshot 3 Analysis by computer

22 Access: Analysis by Patron Web site Snapshot 1 Archive Snapshot 2 Snapshot 3 Access 1 Access 2 Access 3 Analysis by patron Analysis by computer

23 Access Decisions Style of access Analysis of snapshot files by computer Analysis of Web access version by patron Editing Minimal editing to make access version Fuller editing to maintain experience Automatic or by hand Policy Who has access to the collections?

24 Examples of Access Decisions StyleEditing Internet Archivecomputernone Pandoraresearchersome Kulturarw 3 ?? Web Preservationresearchersome

25 Information Discovery

26 Options for Information Discovery Very large numbers of Web sites will be collected and preserved. Some form of index or catalog is required. Options List of sites (e.g., Internet Archive) => Access by URL + date Automatic index (e.g., Web search engines) Catalog (e.g., Web Preservation Project) => Record for individual site or group of sites => Access through library catalog

27 Information Discovery: Web Preservation Project Procedure MARC catalog records created using OCLC's CORC system. Loaded into Library of Congress's ILS. Observations Catalog effort similar to other electronic files Continual changes between snapshots Some similarities to serials No significant workflow difficulties

28 Storage

29 Storage: Preservation Versions Snapshot 1Access 1 Snapshot 1Access 1 Snapshot 1Access 1 Over time, other versions of a snapshot will be made for preservation.

30 Storage Decisions: Size Each Web site will be stored many times Repeated snapshots Access versions Preservation versions Saving space Many files are repeated (e.g., video clips) Storing a single copy saves space, but leads to more complex computer systems Compressing files save space, but leads to more complex computer systems

31 Very Rough Estimates of Size and Cost Public web sites (OCLC, February 2000) 2,900,000 Library of Congress collects 1%30,000 Average size of site 60 Mbytes Size of 30,000 sites 1.8 terabytes Storage requirements/year (monthly snapshot) 21.6 terabytes Storage requirements (no duplicates) 5.0 terabytes Cost per year ($25,000 per terabyte) $125,000

32 Storage Decisions: Identification Identification of Web site URL, but Web sites may change their URL URN (e.g., Handle or PURL) Identification and provenance of versions Web site identifier Collection information (date, time, etc.) History of changes

33 Archive Accession Control Web Crawler Process Catalog External Access Workflow snapshot Analysis by patron Analysis by computer Web site

34 Preservation

35 Objective Objective is to preserve the digital collections in a manner that makes them usable for scholarship and research in the future. What is preserved? Preservation of bits Preservation of content Preservation of experience How is it used? Analysis by computer program Analysis by human researcher Viewed by human researcher

36 Process of Preservation Version 1 Version 2 Version 3 Time 0 Time 1 Time 2 This process may be applied to either the snapshot or the access version.

37 Preservation: Refreshing Each version is created from the previous by exactly copying the bits. Keeps the exact files for all time Preserves bits, and content but not always in an accessible form Later computers and software are unlikely to support today's protocols, formats, languages, etc. Keeping the unedited snapshot files by repeated refreshing should be a basic part of any preservation strategy.

38 Preservation: Automatic Migration of Individual Files As protocols, formats, languages, etc. become obsolete, convert individual files to new standards. Can be carried out automatically Preserves content and helps toward preservation of experience Effectiveness depends on availability of conversion tools and the complexity and quality of original source Migrated versions will steadily diverge from original Web sites will eventually cease to function Automated migration of individual files is the basic technique for keeping web sites functional at moderate cost.

39 Preservation: Automatic Migration with Manual Editing In conjunction with automatic migration, web sites are reviewed by a librarian and edited as necessary to preserve functionality The only method that can be expected to preserve the experience of using web sites Migrated versions will steadily diverge from original Some web sites will be impossible to edit without changing the experience Manual editing is very expensive and is therefore suitable for only a small number of particularly important sites.

40 Acknowledgements The members of the Web Preservation Project are: Roger Adkin Cassy Ammen William Arms Allene Hayes Melissa Levine Diane Kresh Barbara Tillett