1. Introduction to Electronic Records and Digital Preservation CARYN WOJCIK, MILS, CA GOVERNMENT RECORDS ARCHIVIST RECORDS MANAGEMENT SERVICES STATE OF MICHIGAN

2. Objectives  Define the scope of the challenge  Describe the diversity of electronic records  Introduce potential solutions  Provide resources  Explain what the State of Michigan is doing

3. BUILDING THE FOUNDATION

4. Records are Records are Records  Definitions  Records: recorded information on a fixed storage medium  Compared to transitory memory or speech  Records: evidence of activities that are set aside in an official recordkeeping system  Note - not all records are managed this effectively  Electronic Records: bits that are used to re-constitute a record

5. Record Characteristics  Authentic: Can be proven to be what it professes to be, to have been created or sent by the person claiming to have created or sent it, and to have been created or sent at that time.  Reliable: Content can be trusted to be a full and accurate representation of the transactions, activities or facts to which they attest and can be depended upon in the course of subsequent transactions or activities.  Integrity: Content is complete and unaltered.  Useable: Can be located, retrieved, presented and interpreted.

6. Authentic Records Attributes  Content: Information about an idea or concept, an event, a person, an organization or similar phenomenon that is essential to the activity that created the record  Context: Circumstances of creation, control, use and storage in a recordkeeping system  Who, what, when, where, why, how?  Structure: Logical and physical attributes of a record  Logical: how the information is organized, how the parts of a record interact to convey content and meaning  Physical: file format, software version, font, line spacing, colors, margins

7. How do we know that a record is authentic?  It is in its original format  It is stored in a filing system (paper or electronic) that was properly maintained  It was created/used according to written procedures  It was created in the normal course of business by a responsible person  Custody over time is documented  Modifications/reproductions are documented

8. Authentic Paper Records  Physical characteristics of the record document the activities of creation and use  Type/condition of paper  Signatures  Dates  Modifications

9. Authentic Electronic Records  Metadata documents the activities of creation and use  System logs  Audit trails  System security and user permissions  Electronic signatures  Encryption  File format  Checksum

10. THE WORLD OF ELECTRONIC RECORDS

11. Electronic Records are Pervasive  1971: E-mail is developed to support communication  1980s: PC becomes a staple in the modern office (microchip allowed computers to evolve from mainframes to PCs)  1990s: Internet becomes a omnipresent tool for exchanging information  2000s: Portable devices, wireless networks and social media have a transformative affect  Today: most employees cannot function effectively at work without a device, because most business processes are electronic

12. Why do we like electronic records?  ER eliminate physical storage  ER are easy to  Analyze, manipulate, report  Modify and duplicate  Search, retrieve, transport and share

13. http://www.domo.com/blog/2012/06/how-much-data-is-created-every-minute/

14. http://www.techrepublic.com/blog/big-data-analytics/the-landscape-of-big-data/

16. HOW CAN WE MANAGE ALL OF THIS STUFF?

17. Electronic Records Challenges  Employees are not given instructions about how to manage the records that are created by new tools  Unlimited growth of electronic records makes them harder to manage  Electronic storage is not unlimited and free  Back-ups are required to protect against failure and corruption  Retention is not implemented consistently, if at all

18. Electronic Records Reality  We can’t ignore ER  We can’t prevent ER creation  We have a responsibility to manage and preserve  We need:  Strategy  Support  Partners

19. LOST Stuff Can’t Share Stuff Stuff is Everywhere Duplicate Records Disorganized Records FOIA Timelines Email/Shared Drives Full Loose Security Accidental Deletion DIFFICULT File Cleanup Retirements and Departures Version Control Litigation Holds No Collaboration Tools Common Recordkeeping Problems

20. Desktop Records  ER created by desktop applications  Biggest challenge for archivists and records managers  Modern office distributes/de-centralizes records among large number of creators  Records document both business processes and general administrative activities  Creators store records in a variety of places  E-mail accounts, hard drives, shared/network drives, cloud storage, social media accounts, external drives, paper files

21. Desktop Records Challenges - IT  Manages records according to creator, not business process  Files are not separated by business process  Files are not “owned” by the business process owner  Focus on format  Ignore the value of the content  Some tools prevent different formats from being stored together

22. Desktop Records Challenges - Creators  Don’t organize files by record series  Keep a lot of duplicates  Don’t manage versions  Rarely purge older records  Forget which older records they have  Forget where older records are stored

23. Solutions - Supervisors  Accountability  Designating filing systems  Assigning responsibility for recordkeeping  Implementing record retention routinely  Training employees about recordkeeping  Managing records of departing employees

24. Solutions - Standard Operating Procedures  Define  Roles and responsibilities  Quality controls  Detailed task instructions (consistency)  Business rules  Naming conventions  File plans

25. Solutions - Document Management “… consists of technologies, tools, and methods used to capture, manage, store, preserve, and deliver information, content, and documents related to organizational processes.” --AIIM “… consists of technologies, tools, and methods used to capture, manage, store, preserve, and deliver information, content, and documents related to organizational processes.” --AIIM

26. Benefits of Document Management

27. Records Management Benefits  Records are owned by business processes (instead of users)  File plans can organize the records  Security and metadata document authenticity of creation and use  Records can be linked retention schedules  Automate routine disposition  Identify long-term and archival records that have preservation issues

28. Michigan Records Management Services  Provides EDM solutions  IRMA (since 2004) and HP Records Manager (since 2009)  State government  Needs analysis, consulting services, EDM implementation

29. TYPES OF ELECTRONIC RECORDS

30. Born Electronic Records  Originally created on a computer  May never be printed to paper  Wide variety  Structured Data  Databases, GIS, CAD, document management systems, etc.  Unstructured Data  Word, Excel, PowerPoint, e-mail, messaging, social media posts, web pages, digital audio, digital photos, etc.

31. Paper Is Not Dead  15% of an organization’s revenues are spent creating, managing and distributing documents  60% of employee time is spent working with documents  85% of business documents are in paper form  The average document is printed five times  90% of a business’s information is in documents  At $30 an hour, knowledge workers waste $4,500 a year working with paper Source: The Paperless Project 2013The Paperless Project

32. Converted to Electronic  Paper or microfilm records that are scanned and converted into digital formats  Digital images may be flat or text-searchable

33. Legality of Imaging for Government  Michigan’s Records Reproduction Act states that “a record reproduced under this act shall have the same force and effect as a true paper copy of a record.”Records Reproduction Act  Reproductions under the act “shall have the same force and effect as an original for all legal purposes and is admissible in court, administrative proceedings, and elsewhere as evidence in the same manner as an original.”  Records must be reproduced in compliance standards adopted by the State of Michigan  Government agencies can use SOM contract for compliant imaging services

34. Technology Change = Technology Obsolescence  ER are changing in unpredictable ways  This will continue into the future  Change is the cornerstone of the technology creator’s business model  Generates new revenue  Retains customers’ interest and dependency  Little incentive for vendors to support older technology over time

35. Technological Obsolescence = Preservation Challenge  ER are hardware and software dependent  ER may need to be retained longer than the technology originally used to create them  ER need to be migrated approximately every 5-10 years to remain accessible  Each migration has the potential to change the record, and the record’s integrity

36. Obsolescence Factors  Hardware: computers, disk drives  Applications: operating system, driver software  Interface: ports, connecting wires/cables  Software: file formats, versions

37. File Formats and Standards  File Formats: standard way that information is encoded for storage in a computer file  Specify how bits are used to encode information in a digital storage medium  May be either proprietary or shared  May be either unpublished or open  Some are designed for very particular types of data  Standards: requirements that specify uniform criteria, methods, processes and practices concerning how a file format is used

38. File Format Versions  File formats can change over time  File format versions are not always compatible with each other

39. ER Storage Media Aging ProductIntroducedObsolete Punch Card1930~1975 8” disk1970~1980 5 ¼” disk1975~1985 3 ½” disk1982~2005 CD1988~2010 DVD1998TBD USB Flash Drive2001TBD Next???TBD

40. Vulnerabilities  ER could be purged/abandoned when a system is migrated  Expensive to migrate data to new system  New and old systems may be incompatible  ER could be purged when storage and maintenance costs get too high  Get rid of the older records, regardless of content  Technology obsolescence is ignored until it is too late  Failure to maintain storage media or file format  ER are forgotten and lost  Employee turnover, programmatic changes, time

41. Collaborating with IT “Seventy years in IT – that’s like 1000 years in regular life.”  IT maintains the technology  IT is the physical custodian of ER  IT does not own the data  Creator is the legal custodian of ER  IT focuses on implementing and maintaining the technology  IT does not consider long-term access/preservation of the content  Data owner must define its need for retention and communicate with IT

42. Be Proactive - Get Involved in System Design  Questions for IT and Data Owner  Does all data have the same retention period?  How is data collected/organized?  How will retention be implemented?  Triggers (date closed, birth date, graduation date, etc.)  Can expired data be found and segregated from active data?  Can data be purged or exported?  How will purged data and metadata (indexes, audit trails, etc.) be removed so it is unrecoverable?  What procedures and quality controls will be used to document the destruction process?

43. Responsibility  When record creators choose to use ER, they accept* certain responsibilities  Good decisions will help maintain usable ER  We cannot anticipate most of the technology that will exist in the future  There is no guarantee of permanent preservation for ER  Must focus on preserving records for the next generation of technology that is coming

44. Threats to Preservation  Records in all formats need to be actively protected  Traditional Records  Environment, elements, media longevity, pests, etc.  Electronic Records  Obsolescence, viruses, breaches, corruption, etc.

45. Preservation Example  Your office is now using Microsoft Word, but still has a lot of WordPerfect records  WordPerfect is no longer available, but Word currently can open those records  Next version of Word will not open WordPerfect records  WordPerfect records need to be deleted (if no longer needed) or preserved  What should you do?

46. ER PRESERVATION OPTIONS

47. Conversion  Converting electronic files into human-readable media  Paper, microfilm  Not feasible for all types of ER  Practical for some, like word processed documents  Meeting minutes, press releases, reports, forms  Not practical for others, like databases, websites, etc.  Human-readable formats do not facilitate interactivity, accessibility

48. Migration  Periodic transformation of the bits/bytes to run directly on current technology  New version of the old file format  New file formats

49. Normalization  All digital objects of a particular type are converted into a single chosen file format that is thought to embody the best overall compromise amongst characteristics  Reduces the number of formats that need to be monitored for migration  Repeatable (understood) migration procedures  Need to select the acceptable formats, and versions

50. File Format Migration Resources  Conversion Software Registry: finds software capable of converting files from a source format to a target format (NARA/U of Illinois)  NARA File Analyzer and Metadata Harvester: identifies file format, file size and checksum (NARA)  PRONOM: registry defines the characteristics of known file formats (UK)  DROID: tool allows the search and discovery of file formats within a collection, supports monitoring of file format changes and migration (UK)  GDFR: Global Digital Format Registry (Harvard)  JHOVE2: tool characterizes known file types (Harvard)  UDFR: open source platform for unifying use of PRONOM and GDFR (California Digital Archives)  PLANETS: registry of preservation recommendations and actions (EU)  Stanford Digital Repository Format Scoring Matrix: sustainability scores for preservation risk of a file format (Stanford U)

51. File Information Tool Set (FITS)  http://fitstool.org/ http://fitstool.org/  FITS identifies, validates, and extracts technical metadata for various file formats  FITS wraps several third-party open source tools, normalizes and consolidates their output, and reports any errors  FITS was created by the Harvard University Library Office for Information Systems Tools currently bundled in: ADL Tool Apache Tika DROID Exiftool File Utility (windows port) File Utilitywindows port Jhove MediaInfo National Library of New Zealand Metadata Extractor National Library of New Zealand Metadata Extractor OIS Audio Information OIS Audio Information OIS File Information OIS File Information OIS XML Information OIS XML Information Tools currently bundled in: ADL Tool Apache Tika DROID Exiftool File Utility (windows port) File Utilitywindows port Jhove MediaInfo National Library of New Zealand Metadata Extractor National Library of New Zealand Metadata Extractor OIS Audio Information OIS Audio Information OIS File Information OIS File Information OIS XML Information OIS XML Information

52. Emulation  Keep ER in their original format  Emulation tools re-create the functionality of the original software and operating system by imitating obsolete technology on new technology  Used when:  Form and appearance are important  Video games, GIS, CAD  No migration pathway is available  Custom built systems, very old formats  Resources  Open Preservation Foundation  http://openpreservation.org/blog/2013/01/08/emulation-service/ http://openpreservation.org/blog/2013/01/08/emulation-service/  Preservica  http://www.youtube.com/watch?v=OJeKXSPPF1I http://www.youtube.com/watch?v=OJeKXSPPF1I

53. File Viewing  Avantstar, QuickView Plus 13 Standard  Allows you to view files without the native application on the desktop  Supports over 300 file formats  View, copy, print and search files  File is not opened, so viruses are not activated  Costs only $49  http://www.avantstar.com/metro/home/products/quickviewplusstandardedition http://www.avantstar.com/metro/home/products/quickviewplusstandardedition

54. Digital Preservation Tools  Community Owned digital Preservation Tool Registry (COPTR)  http://coptr.digipres.org/Main_Page http://coptr.digipres.org/Main_Page  Describes tools useful for long term digital preservationlong term digital preservation  Helps practitioners find the tools they need to preserve digital data  404 different tools listed in the registry

55. Are you overwhelmed? OR… Are you ready to take on the responsibility of preserving electronic records? ARE YOU READY?

56.  Digital Preservation Management: Implementing short- term strategies for long-term problems  http://www.dpworkshop.org/dpm-eng/eng_index.html http://www.dpworkshop.org/dpm-eng/eng_index.html  Cornell University and ICPSR, University of Michigan Digital Preservation Tutorials

57. CoSA Digital Preservation Capability Maturity Model © http://www.statearchivists.org/seri/phase_one_report.htm

58. Capability Maturity Model Stages ©  Five progressively advanced capability levels  Enables high level assessment of an organization’s current state digital preservation capabilities against good practice standards  Facilitates development of a strategic roadmap for incremental levels of improvement based on the organization’s:  risk exposure  business requirements and  available resources  Adaptable to virtually any organization CoSA/Charles Dollar/Lori Ashley

59. NDSA Levels of Digital Preservation http://blogs.loc.gov/digitalpreservation/files/2012/09/Levels-of-Digital-Preservation-draft-handout-v3.pdf

60. Building a Preservation Program  ER preservation requires new and different tools than those used for paper records  Storage space  Preservation tools  Staff have to acquire new knowledge and skills  Monitor and preserve ER  Additional expenses often are not covered by a traditional office budget  Office may need to build a business case for new resources

61. Business Case and Risk Tools  Making the Business Case for Digital Preservation  Preservica, 2013  http://preservica.com/resources/knowledge-centre/ http://preservica.com/resources/knowledge-centre/  Guide and template for writing a business case  Includes options for building a case that will be the most compelling for your organization  Digital Value at Risk Calculator  Tool allows you to describe your ER or e-collections  Calculates risk and cost for digital preservation  White Papers

62. Digital Preservation Standards  Reference Model for Open Archival Information Systems (OAIS)  Published January 2002  Consultative Committee for Space and Data Systems  NASA, NARA, many others  ISO 14721 (2003)  Maps workflow of ER from creator to reference user  http://public.ccsds.org/publications/archive/650x0m2.pdf http://public.ccsds.org/publications/archive/650x0m2.pdf

63. OAIS Model

64. OAIS Processes  Ingest  Accessioning: taking custody of ER and responsibility for preservation  SIP: Submission Information Package  Archival Storage  Ensuring records are safe and secure  AIP: Archival Information Package  Data Management  Administration  Preservation Planning  Keeping records accessible over time  Access  Helping people use the records  DIP: Dissemination Information Package

65. Michigan’s SERI Self-Assessment 2012

66. Michigan’s SERI Self-Assessment 2015

67. How did Michigan do it?  Michigan needed:  System for storing and preserving electronic records  Employees with the skills required to use and manage the system  Funding  Michigan’s challenge is not unique  Michigan did not want to work independently to solve this problem  Michigan collaborated with others and monitored international research

68. Michigan’s ER Preservation Projects  Records Management Application Pilot Project (RMA)  1999: NHPRC ($190,000)  Installed and evaluated RMA software on user desktops  Preserving Electronic Records Stored in a RMA (PERM)  2001: NHPRC ($160,000)  Developed and tested functional requirements for preservation  Persistent Archives Testbed (PAT)  2003: NHPRC ($242,500)  Conducted a case study to preserve 30 years worth of elections databases  Distributed Custodial Archival Preservation Environments (DCAPE)  2008: NHPRC ($257,800)  Initiated design of a trusted digital repository service for preserving electronic records

69. Old Storage Media  2005: Archives of Michigan needed to recover data from old magnetic tapes  State of Michigan no longer owned working machines that could read the tapes  Muller Media, http://www.mullermedia.com/http://www.mullermedia.com/  Experts in rescuing data from obsolete storage media  Reasonable pricing  Recommended by NARA

70. Michigan’s Functional Requirements  Compliant with ISO standards  Affordable  Can be implemented and maintained without internal IT support, and can be used by staff who do not have technical expertise  No hardware purchase, installation or maintenance: cloud storage preferred  No software installation and maintenance  No programming or coding: must function out of the box with an intuitive user interface  Secure storage with backup/disaster recovery

71. Michigan’s Functional Requirements  Automated ingestion processes  File format identification  Virus check  Fixity check  Automated preservation processes  File format monitoring  On-going fixity checks  Metadata maintenance  Automated dissemination processes  Search  Retrieval of a duplicate  Exit strategy for records and metadata

72. Michigan Gets Preservica  Archives of Michigan became the first US state archives to purchase Preservica on October 10, 2012 (Electronic Records Day)  Michigan ER Collections  Born electronic  163 GB > 19,300 Files  Accessioned from all 3 branches of Michigan government  30 records series  Formats: databases, digital images, word processed documents, digital audio and video, etc.

73. Preservica  “World leader in digital preservation solutions, technology, consulting and research.”  Customers across 4 continents  Preservica Standard Edition software was first developed in 2003  Launched Preservica Cloud Edition in June 2012  www.preservica.com www.preservica.com

74. Preservica Cloud Edition  Software as a service  Software is accessed via internet, and is not installed locally  Client does not install upgrades  Runs on Amazon Web Services (AWS)  Cloud computing  Uploads are encrypted while in transit and at rest  Multiple redundant copies  6 replications are continually checked for accuracy to 99.99999999999% and will self-repair if errors are found  Security is maintained by Preservica  No hardware to install and maintain  Fees for service  Licensing  Storage

75. Preservica Key Features  Ingest  Data Management  Storage  Access  Preservation Planning and Action  Administration

76. Preservica Technology  Designed for archivists, to address electronic records preservation  Company partners with international research projects addressing various aspects of ER preservation  Company has the resources to monitor the state of the art  Many open source tools were developed via these projects  Provides a unified interface and workflow for using these tools  Preservica will evolve as new tools are developed and user needs are defined  New public access tool was released in May 2014

77. Preservica Dashboard

78. Future Objectives  Acquire more collections for preservation  Offer a fee-based preservation service for government agencies with long-term and permanent records that do not belong to the Archives of Michigan  Potential service models:  Preserve and return  Preserve and store  Preserve, store, provide access

79. More Information… Records Management Services 3400 N. Grand River Ave. Lansing, Michigan 48909 517-335-9132 wojcikc@michigan.gov http://www.michigan.gov/recordsmanagement/ http://inside.michigan.gov/wr/records/ Archives of Michigan 702 W. Kalamazoo Street Lansing, MI 48913 (517) 373-3559 archives@michigan.gov archives@michigan.gov www.michigan.gov/archivesofmi www.seekingmichigan.org