Presentation is loading. Please wait.

Presentation is loading. Please wait.

Tools for a Preservation-Ready Web Joan A. Smith & Michael L. Nelson Old Dominion University Department of Computer Science {jsmit, NDIIPP.

Published by Modified over 9 years ago

Similar presentations

Presentation on theme: "Tools for a Preservation-Ready Web Joan A. Smith & Michael L. Nelson Old Dominion University Department of Computer Science {jsmit, NDIIPP."— Presentation transcript:

1 Tools for a Preservation-Ready Web Joan A. Smith & Michael L. Nelson Old Dominion University Department of Computer Science {jsmit, mln}@cs.odu.edu NDIIPP Digital Preservation Partners Meeting July 9, 2008

2 NDIIPP Partners Meeting Slide # 2 What is Preservation? We will define preservation of a web site W to be: –refreshing copying the bits from place to place R(W) = W r –migrating converting the bits from format f 1 to format f 2 M(W) = W m –emulation simulating the original context for the bits E(W) = W e –putting it all together: E(M(R(W))) = W rme

3 July 9, 2008NDIIPP Partners Meeting Slide # 3 Preservation Function P We define a preservation function P P(W) = W p Intuition is that P makes other functions easier: M(W p ) is easier to implement than M(W) E(W p ) is easier to implement than E(W) R(W p ) is probably easier to implement than R(W)

4 July 9, 2008NDIIPP Partners Meeting Slide # 4 Web Site Preservation: 2 Problems The counting problem What are the members of W? The representation problem How do we define P(W)?

5 July 9, 2008NDIIPP Partners Meeting Slide # 5 Preservation & the Counting Problem To preserve a site, we need to enumerate the full set of a web site’s resources: W = {w 1,w 2,w 3, w 4 … w n } For non trivial web sites: –The membership of W depends on who is asking –W is unknown (unknowable?) –W can only be approximated There is no HTTP mechanism to define W Sitemaps are a method to convey locally-held knowledge about W to web crawlers

6 July 9, 2008NDIIPP Partners Meeting Slide # 6 % telnet foo.edu 80 Trying 82.165.199.160... Connected to foo.edu. Escape character is '^]'. GET /jackJill.jpg HTTP/1.1 Host: foo.edu HTTP/1.1 200 OK Date: Mon, 11 Jun 2007 16:49:25 GMT Server: Apache/1.3.33 (Unix) Last-Modified: Mon, 29 Aug 2005 12:01:40 GMT ETag: "5800535-3e72-4312f924" Accept-Ranges: bytes Content-Length: 15986 Content-Type: image/jpeg ÿØÿà "#2s¡35Rq‘±³ÁÂ$%Ccruƒ“¢ÃÒÿÄ HTML metadata HTTP metadata Preservation & the Representation Problem

7 July 9, 2008NDIIPP Partners Meeting Slide # 7 MD5 Hash: 58a54e8638db432f4515eedf89f44505 EXIF: File Name103_0315.JPG Camera Model NameCanon EOS DIGITAL REBEL Date/Time Original2003:09:30 13:37:51 Shooting ModeSports Shutter Speed1/2000 Aperture7.1 Metering ModeEvaluative Exposure Compensation0 ISO400 Lens75.0 - 300.0mm Focal Length300.0mm Image Size3072x2048 QualityNormal FlashOff White BalanceAuto Focus ModeAI Servo AF Contrast+1 Sharpness+1 Saturation+1 Color ToneNormal File Size1606 kB File Number103-0315 Standard HTTP Headers -- Last-Modified: Mon, 29 Aug 2005 12:01:40 GMT ETag: "5800535-3e72-4312f924" Content-Length: 15986 Content-Type: image/jpeg JHOVE: Date: 2007-06-18 14:35:50 EDT RepresentationInformation: /home/crate/apache/htdocs/jackJill.jpg ReportingModule: JPEG-hul, Rel. 1.2 (2005-08-22) LastModified: 2007-01-16 23:09:07 EST Size: 27750 Format: JPEG Version: 1.00 Status: Well-Formed and valid SignatureMatches: JPEG-hul MIMEtype: image/jpeg Profile: JFIF JPEGMetadata: CompressionType: Huffman coding, Baseline DCT Images: Number: 1 Image: NisoImageMetadata: MIMEType: image/jpeg ByteOrder: big-endian CompressionScheme: JPEG ColorSpace: YCbCr SamplingFrequencyUnit: inch XSamplingFrequency: 33 YSamplingFrequency: 26 ImageWidth: 172 ImageLength: 146 BitsPerSample: 8, 8, 8 SamplesPerPixel: 3 Scans: 1 QuantizationTables: QuantizationTable: Precision: 8-bit DestinationIdentifier: 0 Comments: LEAD Technologies Inc. V1.01 ApplicationSegments: APP0 File/Magic: JPEG image data JFIF standard 1.00 resolution (DPI) "LEAD Technologies Inc. V1.01“ 33 x 26 P(W) Involves the Output of Forensic Metadata Utilities

8 July 9, 2008NDIIPP Partners Meeting Slide # 8 Experiments & Evaluation Counting problem –Web crawler behavior on decaying web sites (D-Lib 2006) –Web crawler behavior on deep and wide web sites (D-Lib 2008) –Defining W on a departmental web site (unpublished) Representation problem –Dissemination time preservation metadata (JCDL 2007, IWAW 2007, D-Lib 2008) –Performance evaluation of metadata utilities (ECDL 2008) Reference implementation: mod_oai, an Apache module –uses Sitemaps, OAI-PMH resource harvesting for counting problem –uses “CRATE” -- base64’d resource + metadata output as the OAI-PMH metadataPrefix for representation problem

9 July 9, 2008NDIIPP Partners Meeting Slide # 9 Lots of pages die in between crawler visits IA never came in 3+ months Decaying Web Sites (D-Lib 2006)D-Lib 2006

10 July 9, 2008NDIIPP Partners Meeting Slide # 10 “Buffet” = Level 1 links to levels 2, 3, 4 … n “Bread Crumb” = Level 1 links to level 2, level 2 links to level 3, etc. Deep & Wide Web Sites (D-Lib 2008)D-Lib 2008

11 July 9, 2008NDIIPP Partners Meeting Slide # 11 Coverage of www.cs.odu.edu SourceFilesURLs Self-Crawl406538 External Crawl406761 File System2,0522,052* Notes: Departmental snapshot (no ~user URLs; CGI files removed; spotty http logs) Google Python Sitemap script crashed on ill-formed log data 100% defined in terms of file system count Results written in a Sitemap file for mod_oai processing (more later)

12 July 9, 2008NDIIPP Partners Meeting Slide # 12 Integrate OAI-PMH functionality into the web server itself… 1.Use mod_oai an Apache 2.0 module automatically answers OAI-PMH requests for an http server written in C respects values in.htaccess, httpd.conf 2.Install mod_oai on http://www.foo.edu/http://www.foo.edu/ 3.Define baseURL: http://www.foo.edu/modoai →Result: web harvesting with OAI-PMH semantics (e.g., from, until, sets) mod_oai implementation (JCDL 2007, IWAW 2007, D-Lib 2008)JCDL 2007IWAW 2007D-Lib 2008 Using OAI-PMH http://www.foo.edu/modoai?verb=ListRecords&metdataPrefix=oai_crate&from=2007-09-21&set=mime:video:mpeg Give me all resources And their preservation metadata From site foo, dating from 9/21/2007 through today that are MIME type video-MPEG Uses a public (or private) Sitemap for the definition of W (used to be dynamic file system inspection); create the Sitemap as the union of multiple tools

13 July 9, 2008NDIIPP Partners Meeting Slide # 13 CRATE JVBERi0xLjIKJcfsj6IKNiAwIG9iago8PC9MZW5ndGgg…. Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 Accept-Language: en-us,en;q=0.5 … Content-Length: 101702 Content-Type: image/jpeg … If you like to look at lots of XML, go to: http://www.cratemodel.org/modoai/?verb=List Records&metadataPrefix=oai_crate http://www.cratemodel.org/modoai/?verb=List Records&metadataPrefix=oai_crate Intuition: now is the best time to describe a file, and the local user has a good idea as to what should be said (does not preclude further analysis by harvester)

14 July 9, 2008NDIIPP Partners Meeting Slide # 14 CRATE and the OAIS Information Model Base64- encoded resource CRATE Metadata from plug-ins: Summary, index, format analysis… OAI-PMH MPEG-21 DIDL Metadata Format MIME / GDFR Type Copyright Originator Description SIP: original web resource as it exists on the web site AIP: resource processed by mod_oai for metadata & DIP: disseminated to crawler; to other repositories; to an information archeologist for research/extraction

15 July 9, 2008NDIIPP Partners Meeting Slide # 15 CRATE: Apache Configuration File

16 July 9, 2008NDIIPP Partners Meeting Slide # 16 Tested CRATE Plug-Ins for mod_oai NameDescription ExifImage/video metadata extractor JhoveImage analysis DCdcTag html extactor DroidPronom registry info MetaXMeta-extractor OTSOpen Text Summarizer wcunix word count utility fileunix file utility (magic cookie) md5, shaunix md5sum, shasum utilities

17 July 9, 2008NDIIPP Partners Meeting Slide # 17 Server response time to other web requests: < 2% throughput delta Time Required to CRATE Web Site (ECDL 2008)

18 July 9, 2008NDIIPP Partners Meeting Slide # 18 Future Work OAI-ORE support –CRATEs as Resource Maps Defining CRATEs as an http encoding format –like gzip, zip, etc. –can return a CRATE in response to a regular http request with appropriate q values (not just OAI-PMH harvest request) Third party metadata –how can my web server use your installation of Jhove? Tighter http log / Sitemap integration: –“Sitemap strict” -- don’t serve a file unless it appears in a Sitemap –“Sitemap synch” -- in real-time, add/delete entries in Sitemap based on 200 / 404 responses

19 July 9, 2008NDIIPP Partners Meeting Slide # 19 For more information More info, code: http://www.modoai.org/ http://code.google.com/p/modoai/ A joint research project between: –Old Dominion University and –LANL Digital Library Research & Prototyping Team Research supported by the Andrew Mellon Foundation & the Library of Congress

Download ppt "Tools for a Preservation-Ready Web Joan A. Smith & Michael L. Nelson Old Dominion University Department of Computer Science {jsmit, NDIIPP."

Similar presentations

TCP/IP Protocol Suite 1 Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Chapter 22 World Wide Web and HTTP.

TCP/IP Protocol Suite 1 Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Chapter 22 World Wide Web and HTTP.
The Open Archives Initiative DRIADE Workshop, Durham NC, May 16-17, 2007 Michael L. Nelson The Open Archives Initiative Michael L. Nelson Computer Science,

The Open Archives Initiative DRIADE Workshop, Durham NC, May 16-17, 2007 Michael L. Nelson The Open Archives Initiative Michael L. Nelson Computer Science,
Where museums, libraries, and archives intersect NISO Z39.87 Developments Robin L. Dale RLG.

Where museums, libraries, and archives intersect NISO Z39.87 Developments Robin L. Dale RLG.
HTTP HyperText Transfer Protocol. HTTP Uses TCP as its underlying transport protocol Uses port 80 Stateless protocol (i.e. HTTP Server maintains no information.

HTTP HyperText Transfer Protocol. HTTP Uses TCP as its underlying transport protocol Uses port 80 Stateless protocol (i.e. HTTP Server maintains no information.
Chapter 2 Application Layer Computer Networking: A Top Down Approach Featuring the Internet, 3 rd edition. Jim Kurose, Keith Ross Addison-Wesley, July.

Chapter 2 Application Layer Computer Networking: A Top Down Approach Featuring the Internet, 3 rd edition. Jim Kurose, Keith Ross Addison-Wesley, July.
1 The HyperText Transfer Protocol: HTTP Nick Smith Stuart Alley Tara Tjaden.

1 The HyperText Transfer Protocol: HTTP Nick Smith Stuart Alley Tara Tjaden.
2/9/2004 Web and HTTP February 9, 2004. 2/9/2004 Assignments Due – Reading and Warmup Work on Message of the Day.

2/9/2004 Web and HTTP February 9, /9/2004 Assignments Due – Reading and Warmup Work on Message of the Day.
PL-IV- Group A HTTP Request & Response Header

PL-IV- Group A HTTP Request & Response Header
July 25, 2012 Arlington, Virginia Digital Preservation 2012warcreate.com WARCreate Create Wayback-Consumable WARC Files from Any Webpage Mat Kelly, Michele.

July 25, 2012 Arlington, Virginia Digital Preservation 2012warcreate.com WARCreate Create Wayback-Consumable WARC Files from Any Webpage Mat Kelly, Michele.
Web Server Design Week 5 Old Dominion University Department of Computer Science CS 495/595 Spring 2010 Martin Klein 2/10/10.

Web Server Design Week 5 Old Dominion University Department of Computer Science CS 495/595 Spring 2010 Martin Klein 2/10/10.
1 HTML and CGI Scripting CSC8304 – Computing Environments for Bioinformatics - Lecture 10.

1 HTML and CGI Scripting CSC8304 – Computing Environments for Bioinformatics - Lecture 10.
Krerk Piromsopa. Web Caching Krerk Piromsopa. Department of Computer Engineering. Chulalongkorn University.

Krerk Piromsopa. Web Caching Krerk Piromsopa. Department of Computer Engineering. Chulalongkorn University.
Using OAI-PMH Resource Harvesting & MPEG-21 DIDL for Digital Preservation Joan A. Smith & Michael L. Nelson Old Dominion University Department of Computer.

Using OAI-PMH Resource Harvesting & MPEG-21 DIDL for Digital Preservation Joan A. Smith & Michael L. Nelson Old Dominion University Department of Computer.
CSC 2720 Building Web Applications Getting and Setting HTTP Headers (With PHP Examples)

CSC 2720 Building Web Applications Getting and Setting HTTP Headers (With PHP Examples)
CP476 Internet Computing Lecture 5 : HTTP, WWW and URL 1 Lecture 5. WWW, HTTP and URL Objective: to review the concepts of WWW to understand how HTTP works.

CP476 Internet Computing Lecture 5 : HTTP, WWW and URL 1 Lecture 5. WWW, HTTP and URL Objective: to review the concepts of WWW to understand how HTTP works.
TCP/IP Protocol Suite 1 Chapter 22 Upon completion you will be able to: World Wide Web: HTTP Understand the components of a browser and a server Understand.

TCP/IP Protocol Suite 1 Chapter 22 Upon completion you will be able to: World Wide Web: HTTP Understand the components of a browser and a server Understand.
OAI-PMH for Resource Harvesting Tutorial OAI4, October 20 th 2005, CERN, Geneva, Switzerland OAIResource Software Her This work supported in part by the.

OAI-PMH for Resource Harvesting Tutorial OAI4, October 20 th 2005, CERN, Geneva, Switzerland OAIResource Software Her This work supported in part by the.
An Apache Module for Generating Self-Describing Web Resources Joan A. Smith Michael L. Nelson Alliance for Information Science and Technology Innovation.

An Apache Module for Generating Self-Describing Web Resources Joan A. Smith Michael L. Nelson Alliance for Information Science and Technology Innovation.
Generating Best Effort Preservation Metadata for Web Resources at Time of Dissemination Joan A. Smith & Michael L. Nelson Old Dominion University Department.

Generating Best Effort Preservation Metadata for Web Resources at Time of Dissemination Joan A. Smith & Michael L. Nelson Old Dominion University Department.
Lis512 lecture 4 XML: documents and records. up until now Relational databases can store information that is internal to an organization. But a lot of.

Lis512 lecture 4 XML: documents and records. up until now Relational databases can store information that is internal to an organization. But a lot of.

Similar presentations

Ads by Google