Download presentation
Presentation is loading. Please wait.
Published byAlberta Reed Modified over 7 years ago
1
Introduction to Electronic Records and Digital Preservation CARYN WOJCIK, MILS, CA GOVERNMENT RECORDS ARCHIVIST RECORDS MANAGEMENT SERVICES STATE OF MICHIGAN
2
Objectives Define the scope of the challenge Describe the diversity of electronic records Introduce potential solutions Provide resources Explain what the State of Michigan is doing
3
BUILDING THE FOUNDATION
4
Records are Records are Records Definitions Records: recorded information on a fixed storage medium Compared to transitory memory or speech Records: evidence of activities that are set aside in an official recordkeeping system Note - not all records are managed this effectively Electronic Records: bits that are used to re-constitute a record
5
Record Characteristics Authentic: Can be proven to be what it professes to be, to have been created or sent by the person claiming to have created or sent it, and to have been created or sent at that time. Reliable: Content can be trusted to be a full and accurate representation of the transactions, activities or facts to which they attest and can be depended upon in the course of subsequent transactions or activities. Integrity: Content is complete and unaltered. Useable: Can be located, retrieved, presented and interpreted.
6
Authentic Records Attributes Content: Information about an idea or concept, an event, a person, an organization or similar phenomenon that is essential to the activity that created the record Context: Circumstances of creation, control, use and storage in a recordkeeping system Who, what, when, where, why, how? Structure: Logical and physical attributes of a record Logical: how the information is organized, how the parts of a record interact to convey content and meaning Physical: file format, software version, font, line spacing, colors, margins
7
How do we know that a record is authentic? It is in its original format It is stored in a filing system (paper or electronic) that was properly maintained It was created/used according to written procedures It was created in the normal course of business by a responsible person Custody over time is documented Modifications/reproductions are documented
8
Authentic Paper Records Physical characteristics of the record document the activities of creation and use Type/condition of paper Signatures Dates Modifications
9
Authentic Electronic Records Metadata documents the activities of creation and use System logs Audit trails System security and user permissions Electronic signatures Encryption File format Checksum
10
THE WORLD OF ELECTRONIC RECORDS
11
Electronic Records are Pervasive 1971: E-mail is developed to support communication 1980s: PC becomes a staple in the modern office (microchip allowed computers to evolve from mainframes to PCs) 1990s: Internet becomes a omnipresent tool for exchanging information 2000s: Portable devices, wireless networks and social media have a transformative affect Today: most employees cannot function effectively at work without a device, because most business processes are electronic
12
Why do we like electronic records? ER eliminate physical storage ER are easy to Analyze, manipulate, report Modify and duplicate Search, retrieve, transport and share
13
http://www.domo.com/blog/2012/06/how-much-data-is-created-every-minute/
14
http://www.techrepublic.com/blog/big-data-analytics/the-landscape-of-big-data/
16
HOW CAN WE MANAGE ALL OF THIS STUFF?
17
Electronic Records Challenges Employees are not given instructions about how to manage the records that are created by new tools Unlimited growth of electronic records makes them harder to manage Electronic storage is not unlimited and free Back-ups are required to protect against failure and corruption Retention is not implemented consistently, if at all
18
Electronic Records Reality We can’t ignore ER We can’t prevent ER creation We have a responsibility to manage and preserve We need: Strategy Support Partners
19
LOST Stuff Can’t Share Stuff Stuff is Everywhere Duplicate Records Disorganized Records FOIA Timelines Email/Shared Drives Full Loose Security Accidental Deletion DIFFICULT File Cleanup Retirements and Departures Version Control Litigation Holds No Collaboration Tools Common Recordkeeping Problems
20
Desktop Records ER created by desktop applications Biggest challenge for archivists and records managers Modern office distributes/de-centralizes records among large number of creators Records document both business processes and general administrative activities Creators store records in a variety of places E-mail accounts, hard drives, shared/network drives, cloud storage, social media accounts, external drives, paper files
21
Desktop Records Challenges - IT Manages records according to creator, not business process Files are not separated by business process Files are not “owned” by the business process owner Focus on format Ignore the value of the content Some tools prevent different formats from being stored together
22
Desktop Records Challenges - Creators Don’t organize files by record series Keep a lot of duplicates Don’t manage versions Rarely purge older records Forget which older records they have Forget where older records are stored
23
Solutions - Supervisors Accountability Designating filing systems Assigning responsibility for recordkeeping Implementing record retention routinely Training employees about recordkeeping Managing records of departing employees
24
Solutions - Standard Operating Procedures Define Roles and responsibilities Quality controls Detailed task instructions (consistency) Business rules Naming conventions File plans
25
Solutions - Document Management “… consists of technologies, tools, and methods used to capture, manage, store, preserve, and deliver information, content, and documents related to organizational processes.” --AIIM “… consists of technologies, tools, and methods used to capture, manage, store, preserve, and deliver information, content, and documents related to organizational processes.” --AIIM
26
Benefits of Document Management
27
Records Management Benefits Records are owned by business processes (instead of users) File plans can organize the records Security and metadata document authenticity of creation and use Records can be linked retention schedules Automate routine disposition Identify long-term and archival records that have preservation issues
28
Michigan Records Management Services Provides EDM solutions IRMA (since 2004) and HP Records Manager (since 2009) State government Needs analysis, consulting services, EDM implementation
29
TYPES OF ELECTRONIC RECORDS
30
Born Electronic Records Originally created on a computer May never be printed to paper Wide variety Structured Data Databases, GIS, CAD, document management systems, etc. Unstructured Data Word, Excel, PowerPoint, e-mail, messaging, social media posts, web pages, digital audio, digital photos, etc.
31
Paper Is Not Dead 15% of an organization’s revenues are spent creating, managing and distributing documents 60% of employee time is spent working with documents 85% of business documents are in paper form The average document is printed five times 90% of a business’s information is in documents At $30 an hour, knowledge workers waste $4,500 a year working with paper Source: The Paperless Project 2013The Paperless Project
32
Converted to Electronic Paper or microfilm records that are scanned and converted into digital formats Digital images may be flat or text-searchable
33
Legality of Imaging for Government Michigan’s Records Reproduction Act states that “a record reproduced under this act shall have the same force and effect as a true paper copy of a record.”Records Reproduction Act Reproductions under the act “shall have the same force and effect as an original for all legal purposes and is admissible in court, administrative proceedings, and elsewhere as evidence in the same manner as an original.” Records must be reproduced in compliance standards adopted by the State of Michigan Government agencies can use SOM contract for compliant imaging services
34
Technology Change = Technology Obsolescence ER are changing in unpredictable ways This will continue into the future Change is the cornerstone of the technology creator’s business model Generates new revenue Retains customers’ interest and dependency Little incentive for vendors to support older technology over time
35
Technological Obsolescence = Preservation Challenge ER are hardware and software dependent ER may need to be retained longer than the technology originally used to create them ER need to be migrated approximately every 5-10 years to remain accessible Each migration has the potential to change the record, and the record’s integrity
36
Obsolescence Factors Hardware: computers, disk drives Applications: operating system, driver software Interface: ports, connecting wires/cables Software: file formats, versions
37
File Formats and Standards File Formats: standard way that information is encoded for storage in a computer file Specify how bits are used to encode information in a digital storage medium May be either proprietary or shared May be either unpublished or open Some are designed for very particular types of data Standards: requirements that specify uniform criteria, methods, processes and practices concerning how a file format is used
38
File Format Versions File formats can change over time File format versions are not always compatible with each other
39
ER Storage Media Aging ProductIntroducedObsolete Punch Card1930~1975 8” disk1970~1980 5 ¼” disk1975~1985 3 ½” disk1982~2005 CD1988~2010 DVD1998TBD USB Flash Drive2001TBD Next???TBD
40
Vulnerabilities ER could be purged/abandoned when a system is migrated Expensive to migrate data to new system New and old systems may be incompatible ER could be purged when storage and maintenance costs get too high Get rid of the older records, regardless of content Technology obsolescence is ignored until it is too late Failure to maintain storage media or file format ER are forgotten and lost Employee turnover, programmatic changes, time
41
Collaborating with IT “Seventy years in IT – that’s like 1000 years in regular life.” IT maintains the technology IT is the physical custodian of ER IT does not own the data Creator is the legal custodian of ER IT focuses on implementing and maintaining the technology IT does not consider long-term access/preservation of the content Data owner must define its need for retention and communicate with IT
42
Be Proactive - Get Involved in System Design Questions for IT and Data Owner Does all data have the same retention period? How is data collected/organized? How will retention be implemented? Triggers (date closed, birth date, graduation date, etc.) Can expired data be found and segregated from active data? Can data be purged or exported? How will purged data and metadata (indexes, audit trails, etc.) be removed so it is unrecoverable? What procedures and quality controls will be used to document the destruction process?
43
Responsibility When record creators choose to use ER, they accept* certain responsibilities Good decisions will help maintain usable ER We cannot anticipate most of the technology that will exist in the future There is no guarantee of permanent preservation for ER Must focus on preserving records for the next generation of technology that is coming
44
Threats to Preservation Records in all formats need to be actively protected Traditional Records Environment, elements, media longevity, pests, etc. Electronic Records Obsolescence, viruses, breaches, corruption, etc.
45
Preservation Example Your office is now using Microsoft Word, but still has a lot of WordPerfect records WordPerfect is no longer available, but Word currently can open those records Next version of Word will not open WordPerfect records WordPerfect records need to be deleted (if no longer needed) or preserved What should you do?
46
ER PRESERVATION OPTIONS
47
Conversion Converting electronic files into human-readable media Paper, microfilm Not feasible for all types of ER Practical for some, like word processed documents Meeting minutes, press releases, reports, forms Not practical for others, like databases, websites, etc. Human-readable formats do not facilitate interactivity, accessibility
48
Migration Periodic transformation of the bits/bytes to run directly on current technology New version of the old file format New file formats
49
Normalization All digital objects of a particular type are converted into a single chosen file format that is thought to embody the best overall compromise amongst characteristics Reduces the number of formats that need to be monitored for migration Repeatable (understood) migration procedures Need to select the acceptable formats, and versions
50
File Format Migration Resources Conversion Software Registry: finds software capable of converting files from a source format to a target format (NARA/U of Illinois) NARA File Analyzer and Metadata Harvester: identifies file format, file size and checksum (NARA) PRONOM: registry defines the characteristics of known file formats (UK) DROID: tool allows the search and discovery of file formats within a collection, supports monitoring of file format changes and migration (UK) GDFR: Global Digital Format Registry (Harvard) JHOVE2: tool characterizes known file types (Harvard) UDFR: open source platform for unifying use of PRONOM and GDFR (California Digital Archives) PLANETS: registry of preservation recommendations and actions (EU) Stanford Digital Repository Format Scoring Matrix: sustainability scores for preservation risk of a file format (Stanford U)
51
File Information Tool Set (FITS) http://fitstool.org/ http://fitstool.org/ FITS identifies, validates, and extracts technical metadata for various file formats FITS wraps several third-party open source tools, normalizes and consolidates their output, and reports any errors FITS was created by the Harvard University Library Office for Information Systems Tools currently bundled in: ADL Tool Apache Tika DROID Exiftool File Utility (windows port) File Utilitywindows port Jhove MediaInfo National Library of New Zealand Metadata Extractor National Library of New Zealand Metadata Extractor OIS Audio Information OIS Audio Information OIS File Information OIS File Information OIS XML Information OIS XML Information Tools currently bundled in: ADL Tool Apache Tika DROID Exiftool File Utility (windows port) File Utilitywindows port Jhove MediaInfo National Library of New Zealand Metadata Extractor National Library of New Zealand Metadata Extractor OIS Audio Information OIS Audio Information OIS File Information OIS File Information OIS XML Information OIS XML Information
52
Emulation Keep ER in their original format Emulation tools re-create the functionality of the original software and operating system by imitating obsolete technology on new technology Used when: Form and appearance are important Video games, GIS, CAD No migration pathway is available Custom built systems, very old formats Resources Open Preservation Foundation http://openpreservation.org/blog/2013/01/08/emulation-service/ http://openpreservation.org/blog/2013/01/08/emulation-service/ Preservica http://www.youtube.com/watch?v=OJeKXSPPF1I http://www.youtube.com/watch?v=OJeKXSPPF1I
53
File Viewing Avantstar, QuickView Plus 13 Standard Allows you to view files without the native application on the desktop Supports over 300 file formats View, copy, print and search files File is not opened, so viruses are not activated Costs only $49 http://www.avantstar.com/metro/home/products/quickviewplusstandardedition http://www.avantstar.com/metro/home/products/quickviewplusstandardedition
54
Digital Preservation Tools Community Owned digital Preservation Tool Registry (COPTR) http://coptr.digipres.org/Main_Page http://coptr.digipres.org/Main_Page Describes tools useful for long term digital preservationlong term digital preservation Helps practitioners find the tools they need to preserve digital data 404 different tools listed in the registry
55
Are you overwhelmed? OR… Are you ready to take on the responsibility of preserving electronic records? ARE YOU READY?
56
Digital Preservation Management: Implementing short- term strategies for long-term problems http://www.dpworkshop.org/dpm-eng/eng_index.html http://www.dpworkshop.org/dpm-eng/eng_index.html Cornell University and ICPSR, University of Michigan Digital Preservation Tutorials
57
CoSA Digital Preservation Capability Maturity Model © http://www.statearchivists.org/seri/phase_one_report.htm
58
Capability Maturity Model Stages © Five progressively advanced capability levels Enables high level assessment of an organization’s current state digital preservation capabilities against good practice standards Facilitates development of a strategic roadmap for incremental levels of improvement based on the organization’s: risk exposure business requirements and available resources Adaptable to virtually any organization CoSA/Charles Dollar/Lori Ashley
59
NDSA Levels of Digital Preservation http://blogs.loc.gov/digitalpreservation/files/2012/09/Levels-of-Digital-Preservation-draft-handout-v3.pdf
60
Building a Preservation Program ER preservation requires new and different tools than those used for paper records Storage space Preservation tools Staff have to acquire new knowledge and skills Monitor and preserve ER Additional expenses often are not covered by a traditional office budget Office may need to build a business case for new resources
61
Business Case and Risk Tools Making the Business Case for Digital Preservation Preservica, 2013 http://preservica.com/resources/knowledge-centre/ http://preservica.com/resources/knowledge-centre/ Guide and template for writing a business case Includes options for building a case that will be the most compelling for your organization Digital Value at Risk Calculator Tool allows you to describe your ER or e-collections Calculates risk and cost for digital preservation White Papers
62
Digital Preservation Standards Reference Model for Open Archival Information Systems (OAIS) Published January 2002 Consultative Committee for Space and Data Systems NASA, NARA, many others ISO 14721 (2003) Maps workflow of ER from creator to reference user http://public.ccsds.org/publications/archive/650x0m2.pdf http://public.ccsds.org/publications/archive/650x0m2.pdf
63
OAIS Model
64
OAIS Processes Ingest Accessioning: taking custody of ER and responsibility for preservation SIP: Submission Information Package Archival Storage Ensuring records are safe and secure AIP: Archival Information Package Data Management Administration Preservation Planning Keeping records accessible over time Access Helping people use the records DIP: Dissemination Information Package
65
Michigan’s SERI Self-Assessment 2012
66
Michigan’s SERI Self-Assessment 2015
67
How did Michigan do it? Michigan needed: System for storing and preserving electronic records Employees with the skills required to use and manage the system Funding Michigan’s challenge is not unique Michigan did not want to work independently to solve this problem Michigan collaborated with others and monitored international research
68
Michigan’s ER Preservation Projects Records Management Application Pilot Project (RMA) 1999: NHPRC ($190,000) Installed and evaluated RMA software on user desktops Preserving Electronic Records Stored in a RMA (PERM) 2001: NHPRC ($160,000) Developed and tested functional requirements for preservation Persistent Archives Testbed (PAT) 2003: NHPRC ($242,500) Conducted a case study to preserve 30 years worth of elections databases Distributed Custodial Archival Preservation Environments (DCAPE) 2008: NHPRC ($257,800) Initiated design of a trusted digital repository service for preserving electronic records
69
Old Storage Media 2005: Archives of Michigan needed to recover data from old magnetic tapes State of Michigan no longer owned working machines that could read the tapes Muller Media, http://www.mullermedia.com/http://www.mullermedia.com/ Experts in rescuing data from obsolete storage media Reasonable pricing Recommended by NARA
70
Michigan’s Functional Requirements Compliant with ISO standards Affordable Can be implemented and maintained without internal IT support, and can be used by staff who do not have technical expertise No hardware purchase, installation or maintenance: cloud storage preferred No software installation and maintenance No programming or coding: must function out of the box with an intuitive user interface Secure storage with backup/disaster recovery
71
Michigan’s Functional Requirements Automated ingestion processes File format identification Virus check Fixity check Automated preservation processes File format monitoring On-going fixity checks Metadata maintenance Automated dissemination processes Search Retrieval of a duplicate Exit strategy for records and metadata
72
Michigan Gets Preservica Archives of Michigan became the first US state archives to purchase Preservica on October 10, 2012 (Electronic Records Day) Michigan ER Collections Born electronic 163 GB > 19,300 Files Accessioned from all 3 branches of Michigan government 30 records series Formats: databases, digital images, word processed documents, digital audio and video, etc.
73
Preservica “World leader in digital preservation solutions, technology, consulting and research.” Customers across 4 continents Preservica Standard Edition software was first developed in 2003 Launched Preservica Cloud Edition in June 2012 www.preservica.com www.preservica.com
74
Preservica Cloud Edition Software as a service Software is accessed via internet, and is not installed locally Client does not install upgrades Runs on Amazon Web Services (AWS) Cloud computing Uploads are encrypted while in transit and at rest Multiple redundant copies 6 replications are continually checked for accuracy to 99.99999999999% and will self-repair if errors are found Security is maintained by Preservica No hardware to install and maintain Fees for service Licensing Storage
75
Preservica Key Features Ingest Data Management Storage Access Preservation Planning and Action Administration
76
Preservica Technology Designed for archivists, to address electronic records preservation Company partners with international research projects addressing various aspects of ER preservation Company has the resources to monitor the state of the art Many open source tools were developed via these projects Provides a unified interface and workflow for using these tools Preservica will evolve as new tools are developed and user needs are defined New public access tool was released in May 2014
77
Preservica Dashboard
78
Future Objectives Acquire more collections for preservation Offer a fee-based preservation service for government agencies with long-term and permanent records that do not belong to the Archives of Michigan Potential service models: Preserve and return Preserve and store Preserve, store, provide access
79
More Information… Records Management Services 3400 N. Grand River Ave. Lansing, Michigan 48909 517-335-9132 wojcikc@michigan.gov http://www.michigan.gov/recordsmanagement/ http://inside.michigan.gov/wr/records/ Archives of Michigan 702 W. Kalamazoo Street Lansing, MI 48913 (517) 373-3559 archives@michigan.gov archives@michigan.gov www.michigan.gov/archivesofmi www.seekingmichigan.org
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.