Presentation on theme: "Keeping Digital Documents Usable: Managing Data Formats John Mark Ockerbloom Carnegie Mellon University April 27, 1999."— Presentation transcript:
Keeping Digital Documents Usable: Managing Data Formats John Mark Ockerbloom Carnegie Mellon University April 27, 1999
What I’ll be talking about A data model and architecture that supports definition, use, and conversion of an arbitrarily large number of data formats How this model helps digital libraries and archives keep electronic documents accessible and usable over long term
The problem of electronic document preservation We can digitize lots of information, but it can become inaccessible very quickly –(150 year old book vs. 5 year old 5 1/4” floppy) Electronic preservation problems differ sharply from print preservation problems Preserving the bits is easy: just replicate them –(Remove the hardware dependency first if you can) –Internet allows very wide replication, avoiding single- archive failures The problem is understanding the bits, so that you can continue to use them...
Data format mismatch In a large, diverse, digital archive, information comes in a variety of formats Most clients only understand a few formats They therefore cannot effectively use many materials –data may be in incomprehensible form –data may be in form not easily worked with Particularly problematic: –formats that have complex (but useful) structure –legacy data and programs (obsolete format assumptions) In a long-lived library, most information IS “legacy”
Standards are a partial solution Standards allow common understandings… –Data: SGML/XML, Word processor formats, HTML, PDF, Quark, specialized scientific formats, page image formats…. –Metadata: USMARC, Dublin Core, RDF... …But no one standard fits all –different uses may require different data choices –“lowest common denonimator” often not good enough And standards change over time –needs and applications change (sometimes quickly) –standardization process lags –even established standards become obsolete »Who supports EBCDIC now? »Who will support 1999 standards in 2049?
Techologies for usefully preserving unfamiliar formats Emulation –don’t change the data; maintain programs to deal with it –Essentially data abstraction, since the “emulation” just needs to provide same functionality, and may be implemented very differently from original –But: May be costly to maintain infrastructure; may unnecessarily lock user into old interaction styles Migration –Periodically convert data to more “up-to-date” formats; then use your everyday programs on it –But: How do you control information loss?
An archived document
Its raw electronic form From: Sherry T Haddock To: Subject: CAETI Community Meeting Info Date: Thu, 15 Feb :12: (CST) Mime-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY=" =:20798" Cc: Sherry T Haddock This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. Send mail to for more info =:20798 Content-Type: TEXT/PLAIN; charset=US-ASCII Here are maps detailing the March CAETI Community Meeting Location.... Thanks again, Sherry =:20798 Content-Type: TEXT/PLAIN; charset=US-ASCII; name="CaetiMap.hqx" Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: KFRoaXMgZmlsZSBtdXN0IGJlIGNvbnZlcnRlZCB3aXRoIEJpbkhleCA0LjAp DQoNCjojODBLQ0E0VCklZUtGISI2NiUzYzgmIjgtYCMzIiQpIU4hLSlEMyVt ZC1tNGkrJ2EnWiUhTiIhbCEhLSFyW20NCg0KKiEhQiFOIVgiISohJCEzIzMj IiEhISFtIU4hLSIhKiEkcltxMyFgIzMjMnEzcnJxM1hJaHJOIS1BISohJCFg Iw0KDQozIWAzIU4hLSYhKiEkIkojMyFgRiFOIS0pISohJCMzIzMhYFMhTiEt LCEqISQkISMzIWBkIU4hLTEhKiEkcltxDQoNCjMhcmxyTiEtNCEqISQlSiMz IWEtIU4hLTghKiEkJjMjMyFhQiFOITJxcmohJHJbcTNycnEzVCEiNSEqIXE ( ed, MIME-attached, base64, binhexed, Powerpoint 3)
TOM Basic Data Concepts An object is a (non-mutable) typed value –objects can come from anywhere, be passed about (cf. MIME) A type specifies abstractly what can be done with an object –includes attributes, methods, with slots for specs... An encoding is a mapping from one type to another type that represents the original type –usually maps to a simpler type –``lowest-level’’ type: byte sequence A format is a representation of a type as a byte sequence. –As a type plus a sequence of encodings –Conversions can be defined between formats
Example: A simple mail message as a TOM object Value is the content of the message Type is simple ``mail message’’ type –attributes: sender, recipients, header, body... –methods: get_attachment (num)... This ``mail message’’ type has encodings: –``standard’’ encoding is RFC822 encoding (as ASCII byte sequence) Format is just a type with encodings: –``mail message’’ type in ``standard’’ encoding –could also have further encodings (e.g. mail header) TOM ships objects around with format tags
Part of the TOM type hierarchy Object PackageReference URL Powerpoint Mail messageBinhex Communication Subtypes are substitutable for supertypes (cf. Liskov & Wing)
Substitutable types? What are they? –In a ``substitutable’’ subtyping model, objects of type T behave exactly like objects of T’s supertypes when used through supertype interfaces –Conceptually, there’s an ``abstraction mapping’’ where each object in a subtype has a corresponding object in the supertype that can “substitute” for it Why are they important? –They allow unfamiliar types to be used through familiar supertype interfaces, with information and behavior guaranteed to be consistent with the supertype. –(Most other OO systems don’t require this) –The guarantee of the information preserved by a type is also useful in characterizing conversions
What can you do with an object in an unfamiliar format? If you know its type, you can use it through its interface (attributes, methods) –e.g. getting “sender”, “recipients” attributes of mail If you know one of its encodings, you can use it through its encoded type –e.g. displaying the text encoding of the mail If you don’t know its type, but know one of its supertypes, use its supertype’s interface –e.g. using “element”, “describe” methods of package There may be a conversion to a known format –e.g. Powerpoint slide format to GIF image format
The architecture supporting TOM (simplified) Type Broker ServerClient Server Type Broker Clients get info on formats, request operations (e.g. conversions) Servers implement operations Brokers maintain info on formats, invoke servers for operations Brokers can trade info, consult other brokers Clients can also register new formats, operations, server information...
TOM in action Brokers, apps deployed at Carnegie Mellon –TOM Conversion Service, TOM Frame Service –Uses off-the-shelf technology (converters from the Net, Web browsers) –Try them yourself! Demos at It should work even better when scaled up –Broker software to be released later in 1999 –Anyone can add new types, methods, formats, services –The more sites get involved, the more services will be available, and the more expertise brokers will have
Brokers enable smart conversions This doesn’t scaleThis may lose too much information If formats are nodes, and conversions are directed edges... This gives greatest flexibility But how do we find best conversion path? Breadth-first search is a good starting point We can label edges with conversion characteristics Example: Conversions can be said to respect a type (preserve the information given in that type)
Respectful conversions? A conversion c respects type T if for all inputs i in c’s domain, i and c(i) are indistinguishable when viewed through the interface of type T –strong form: abstraction mapping to T of i and c(i) map to the same “substitute” object in T –weak form: there exists no method m of type T for which m(i) can return a value that m(c(i)) cannot return, other parameters and context being equal. (And vice versa. With a similar rule for attributes of T.) Why is this important? –It says that a conversion preserves all the information defined in type T –It lets one specify which information must be preserved
Choices in converting our mail message Object PackageReference URL Powerpoint Mail messageBinhex Communication Respecting Communication: Sender, recipient, etc., guaranteed preserved Respecting Package: Attachment structure guaranteed preserved Want different guarantees? You can always declare a new supertype
What’s good about TOM’s design? It’s simple (and therefore flexible): –Minimal, basic, well understood standards It’s accommodating: –Describes past, present and future data formats with good breadth and depth of expressiveness –It can be composed with a wide variety of programs and databases (including the Web, off-the-shelf programs) –Benefits start with very low investment, then increase It’s scalable (largely by taking advantage of distributed, interactive nature of Net): –Anyone can define new formats and services –Brokers coordinate contributions from Net community, allowing efficient sharing of work
Sharing the work: Key for successful digital libraries, archives Internet gives new opportunities for collaboration, resource sharing, e.g. –Dealing with diverse data formats (TOM broker architecture) –Coordinated cataloging, acquisition (some now, can be done more openly and broadly) –Sharing crucial metadata (Catalog of Copyright Entries) Many long-term payoffs for openly sharing collections –Not just the benefits of public access for citizenry, but also... –Replication minimizes risks of information loss –Value-added services (e.g. indexing, xrefs, search engines) –Allows whole community to adapt, migrate, augment information for new needs and situations “Giving it away” lets you amortize cost of improvement & maintenance over wide constituency