1. © 2010 Microsoft Corporation. All rights reserved. Quality Assurance: Towards Tools for Characterizing and Comparing Digital Documents Natasa Milic-Frayling Microsoft Research Cambridge UK www.planets-project.eu

2. © 2010 Microsoft Corporation. All rights reserved. What is the problem? Digital is a victim of its own success i.e., the advances in digital technologies that make digital media broadly used and adopted Document formats, software and hardware are becoming obsolete faster than we can ensure the forward compatibility of the content.

3. © 2010 Microsoft Corporation. All rights reserved. What are technical solutions? We have two main strategies: emulation and simulation – Create emulators of hardware and simulators of software systems to enable old programmes to run and old data to be used. content migration – Migration to standards that are likely to be supported in the future.

4. © 2010 Microsoft Corporation. All rights reserved. Preservation and Long-term Access through NETworked Services Ensure long-term access to Europe’s cultural and scientific heritage Improve decision-making about long term preservation Ensure long-term access to valued digital content Control the costs through automation, scalable infrastructure Ensure wide adoption across the user community Establish market place for preservation services and tools Build practical solutions Integrate existing expertise, designs and tools Share and build

5. © 2010 Microsoft Corporation. All rights reserved. The British Library National Library, Netherlands Austrian National Library State and University Library, Denmark Royal Library, Denmark National Archives, UK Swiss Federal Archives National Archives, Netherlands Hatii at University of Glasgow University of Freiburg Technical University of Vienna University at Cologne Tessella Plc IBM Netherlands Microsoft Research, Cambridge ARC Seibersdorf research PLANETS Partners

6. © 2010 Microsoft Corporation. All rights reserved. PLANETS Sub Projects

7. © 2010 Microsoft Corporation. All rights reserved. CONVERSION TOOLS preserving office documents

8. © 2010 Microsoft Corporation. All rights reserved. Microsoft & PLANETS: Preserving Office Documents Microsoft Research role within PLANETS: – Conversion of binary Microsoft Office Documents into Office Open XML File Format (OpenXML) We extended the effort to include other formats – More legacy formats, e.g. WordPerfect – Other open standards, e.g. Open Document Format. Binary MS Office OpenXML WordPerfect ODF Binary MS Office OpenXML DOS Word UOF

9. © 2010 Microsoft Corporation. All rights reserved. Document Conversion Tools – Our Approach Three-step approach, resulting in a modular and extendible infrastructure – Identify existing conversion tools and libraries – Wrap these tools and libraries into re-usable components – Integrate these components into PLANETS and other systems. If possible, do not use the office applications (e.g., Microsoft Office or OpenOffice.org) – They are designed as interactive applications – Message boxes might pop up (“Do you want …”) – Unclear license question when running on a server.

10. © 2010 Microsoft Corporation. All rights reserved. Reusable Components Transformer Box (Wrapper) “Binary  OpenXML” TB Interface TB Interface Watch Folder Tool Web Service ToooXML (GUI)

11. © 2010 Microsoft Corporation. All rights reserved. Extendible Architecture Transformer Box (Wrapper) “ODF  OpenXML” Transformer Box (Wrapper) “WP  OpenXML” Transformer Box (Wrapper) “Binary  OpenXML” TB Interface TB Interface Watch Folder Tool Web Service ToooXML (GUI)

12. © 2010 Microsoft Corporation. All rights reserved. More Technical Details (1) Currently two types of wrappers for – Command-line tools (stand-alone executables) OpenXML/ODF Translator (OpenXML  ODF) OpenXML Document Viewer (OpenXML  HTML) – Microsoft conversion libraries (CNV libraries) WordPerfect  RTF RTF  OpenXML … We allow wrappers to be chained – WordPerfect  RTF  OpenXML  ODF.

13. © 2010 Microsoft Corporation. All rights reserved. Microsoft Word More Technical Details (2) Microsoft conversion libraries (CNV libraries) – Originally designed to import/export “foreign” document formats into/from Microsoft Word – Based on the Microsoft Conversion API Foreign2RTF RTF2Foreign – Transformer Box CNV Wrapper follows this API. Transformer Box CNV Wrapper CNV Library RTF2Foreign Foreign2RTF

14. © 2010 Microsoft Corporation. All rights reserved. Supported Formats  Source formats  WordPerfect 5  WordPerfect 6  DOS Word  Word 2, 6, 95  Word 97-2003  RTF  ODF  OpenXML  Target formats  OpenXML  ODF  UOF  HTML  XCDL (format defined in PLANETS/PC)

15. © 2010 Microsoft Corporation. All rights reserved. CONVERSION SERVICES preserving office documents

16. © 2010 Microsoft Corporation. All rights reserved. Conversion applications and service

17. © 2010 Microsoft Corporation. All rights reserved. Conversion applications and service

18. © 2010 Microsoft Corporation. All rights reserved. Conversion applications and service

19. © 2010 Microsoft Corporation. All rights reserved. Conversion applications and service

20. © 2010 Microsoft Corporation. All rights reserved. Conversion applications and service

21. © 2010 Microsoft Corporation. All rights reserved. Conversion applications and service

22. © 2010 Microsoft Corporation. All rights reserved.

23. SIMILARITY ASSESSMENT understanding the quality criteria

24. © 2010 Microsoft Corporation. All rights reserved. How do we explore and compare digital artefacts Perceptive aspects of the digital object – In the past printed version of the document and screen display Interactive aspects of the digital objects – Dynamic content includes both individual artefacts and the `stream characteristics‘. Non-perceptive aspects of the digital objects – Document object model, cashed data, action generated metadata, hidden formulas, etc.

25. © 2010 Microsoft Corporation. All rights reserved.

26. EXAMPLE: Perceptive features for Word Documents Two objects in different formats are mapped onto the normalize form – E.g., a WP file converted into.docx. For both we create an XPS representation of the document Feature extraction and comparison – For each feature develop a `digital object probe‘ that extract the feaeture and measure a property of the feature – E.g., pass XPS through OCR package and extract various layout features.

27. © 2010 Microsoft Corporation. All rights reserved. Conversion applications and service

28. © 2010 Microsoft Corporation. All rights reserved. Conversion applications and service

29. © 2010 Microsoft Corporation. All rights reserved. What is ahead of us? Research – What is the relationship between the human criteria and automated measurements? What usage scenarios do we aim for? Technology – What ‘instruments’ do we need to extract and measure properties of the digital content? – How do we automate the process of inspection and quality assurance? Legal – How do we run legacy software as services? We need updated licensing agreements. – How to provide services that combine open source and non-open source software?

30. © 2010 Microsoft Corporation. All rights reserved. THANK YOU Contact: Natasa Milic-Frayling Microsoft Research Cambridge natasamf@microsoft.com