1. 1 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Cross-Industry Preservation Architectures Michael Peterson May 2011

2. 2 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Introductions ● Michael Peterson Founder, past President, and Chief Strategy Advocate for the SNIA Founder, past President, and Chief Strategy Advocate for the SNIA Currently working on Cloud Archive and Long-term Retention standards, best practices, market education Currently working on Cloud Archive and Long-term Retention standards, best practices, market education Information Services Architect – consulting in long-term retention and digital preservation system design and implementation Information Services Architect – consulting in long-term retention and digital preservation system design and implementation Currently driving the LTDPRM.org & ILM2.0.org Communities Currently driving the LTDPRM.org & ILM2.0.org Communities Author: “ 100 Year Archive Requirements Study,” 2008, “Building a Terminology Bridge: Guidelines for Digital Information Retention and Preservation Practices in the Datacenter,” Sept. 2009 Author: “ 100 Year Archive Requirements Study,” 2008, “Building a Terminology Bridge: Guidelines for Digital Information Retention and Preservation Practices in the Datacenter,” Sept. 2009

3. 3 “Cross-Industry Preservation Architectures” – PASIG May, 2011 www.ltdprm.org

4. 4 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Agenda ● LTDP Paradoxes (Laws designed to be broken) ● New Digital Preservation Models That the cloud empowers That the cloud empowers ● Using the Cloud for Digital Archives (Digital Preservation)

5. 5 “Cross-Industry Preservation Architectures” – PASIG May, 2011 4 Paradoxes of Digital Preservation ● Data will be lost ● Migration does not scale ● Access & use models keep changing ● Cost overwhelms everything complexity does not

6. 6 “Cross-Industry Preservation Architectures” – PASIG May, 2011 How do we break them? “Old” Laws of Digital Preservation

7. 7 “Cross-Industry Preservation Architectures” – PASIG May, 2011 100 Year Complexity Barrier ● Overwhelming growth, cost, change Constant Physical and Logical migration Constant Physical and Logical migration Power, cooling, space, people, resources, maintenance,… Power, cooling, space, people, resources, maintenance,… Always adding & migrating systems, networking, storage Always adding & migrating systems, networking, storage Managing thousands of formats Managing thousands of formats Constant Auditing and recovery of damaged or lost data Constant Auditing and recovery of damaged or lost data Thousands of moving parts Thousands of moving parts Complex systems and architectures Complex systems and architectures Changing software platforms Changing software platforms

8. 8 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Aha! Move from “physical” preservation architectures and design as in physical media or a physical repository (a 2002 ‘OAIS’) to “virtualized” Preservation Services based on Service Management principles (Sounds like the Cloud…)

9. 9 “Cross-Industry Preservation Architectures” – PASIG May, 2011 New Digital Preservation Models Using the Cloud

10. 10 “Cross-Industry Preservation Architectures” – PASIG May, 2011 “Physical” Doesn’t Scale ● Old Architecture “Storing digital images effectively requires standards related to the storage media, such as CD- ROMs, and the file formats, such as TIFF.” Source: “A Resource List for Standards Related to Digital Imaging” Dec. 2010 “Storing digital images effectively requires standards related to the storage media, such as CD- ROMs, and the file formats, such as TIFF.” Source: “A Resource List for Standards Related to Digital Imaging” Dec. 2010 Physical Application & Storage infrastructure Physical Application & Storage infrastructure ● Physical Standards Architecture: OAIS 2002 Architecture: OAIS 2002 Metadata: Dublin Core: ISO 15836:2009 Metadata: Dublin Core: ISO 15836:2009 Storage Media: ISO 18921:2008, ISO 18925:2008, Digitization: ISO/IEC 10918- 1/Cor1:2005, ISO/IEC 10918- 3/Amd1:1999 Storage Media: ISO 18921:2008, ISO 18925:2008, Digitization: ISO/IEC 10918- 1/Cor1:2005, ISO/IEC 10918- 3/Amd1:1999 File Formats: ISO 19005- 1:2005, Adobe TIFF Specification, V6, 1992 File Formats: ISO 19005- 1:2005, Adobe TIFF Specification, V6, 1992 Transfer Protocols: ISO 15740:2008 Transfer Protocols: ISO 15740:2008

11. 11 “Cross-Industry Preservation Architectures” – PASIG May, 2011 “Infrastructure Virtualization” ● New Architecture Media independent Media independent System Architecture virtualized, self-protecting, cloud based, and self healing System Architecture virtualized, self-protecting, cloud based, and self healing Integrated migration & transformation services Integrated migration & transformation services Virtualized historical applications hosted in the cloud in specialized containers running in virtual machines Virtualized historical applications hosted in the cloud in specialized containers running in virtual machines ● New Standards Architecture: OAIS 2010 Architecture: OAIS 2010 Metadata: FCIS, PREMIS, IETF Metadata: FCIS, PREMIS, IETF Cloud: SNIA-CDMI Cloud: SNIA-CDMI Interoperability: NIST Smart Grid Framework, Cloud and Interoperability workgroups Interoperability: NIST Smart Grid Framework, Cloud and Interoperability workgroups Object Containerization: SNIA-SIRF Object Containerization: SNIA-SIRF

12. 12 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Add “Information Virtualization” ● Portable Information Objects Extensible Preservation Objects Extensible Preservation Objects Location, media independent Location, media independent Secure, auditable, authentic, portable Secure, auditable, authentic, portable Self-healing Self-healing ● On-demand, virtual emulation “Jumpbox” hosted emulators “Jumpbox” hosted emulators Populations of legacy ‘readers’ Populations of legacy ‘readers’ Web-based delivery and access Web-based delivery and access ● New Standards Architecture: OAIS 2010 Architecture: OAIS 2010 Metadata: FCIS, PREMIS, IETF Metadata: FCIS, PREMIS, IETF Cloud: SNIA-CDMI Cloud: SNIA-CDMI Interoperability: NIST Smart Grid Framework, Cloud and Interoperability workgroups Interoperability: NIST Smart Grid Framework, Cloud and Interoperability workgroups Object Containerization: SNIA-SIRF and CDMI Object Containerization: SNIA-SIRF and CDMI

13. 13 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Move to “Managed” Content Management Content Management Service Management Service Management ITIL, ITSM, ILM2.0, Information Governance ITIL, ITSM, ILM2.0, Information Governance Litigation ‘Ready’ Litigation ‘Ready’ Preservation begins at “Creation” Preservation begins at “Creation” Preservation is a new Datacenter Practice Operating Practices: Operating Practices: ITSM - IT Service Mgmt. ITIL-IT Infrastructure Library ILM2.0 - Service mgmt. based approach to information mgmt. and automation Regulatory Compliance

14. 14 “Cross-Industry Preservation Architectures” – PASIG May, 2011 And to “Virtual Services” in the Cloud ● Platform as a Service ● Infrastructure as a Service ● Storage as a Service ● Evolving Web access and use models ● Private, Hybrid, Public Clouds Multiple clouds, multiple providers Multiple clouds, multiple providers

15. 15 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Impact of Virtualization ● Physical Migration Self-healing systems Self-healing systems High Availability, failover High Availability, failover Self-healing storage Self-healing storage File systems, “chunking” vs. RAID File systems, “chunking” vs. RAID Digital Auditing Digital Auditing Thin provisioning Thin provisioning Snapshots Snapshots Self-healing information Self-healing information Portable Information Objects and digital auditing Portable Information Objects and digital auditing

16. 16 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Impact of Virtualization ● Logical Migration Content Management automates and controls Content Management automates and controls Transformation a background process Transformation a background process Preservation objects contain original + transforms, audit trail, links Preservation objects contain original + transforms, audit trail, links ● Virtualized Emulation changes everything Web-based delivery normalizes presentation layer Web-based delivery normalizes presentation layer Virtual Appliance Libraries Virtual Appliance Libraries Contain legacy apps, configurations, and OS Contain legacy apps, configurations, and OS Launch ‘Jumpboxes’ on VMs on demand Launch ‘Jumpboxes’ on VMs on demand

17. 17 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Impact of Virtualization ● Cost - #1 Eliminate physical migration Eliminate physical migration Digital Audits and Self Healing Digital Audits and Self Healing Exception – physically moving systems Exception – physically moving systems Minimize logical migration Minimize logical migration Virtual emulation, web delivery vs. old formats Virtual emulation, web delivery vs. old formats Instrument and Automate transformations Instrument and Automate transformations

18. 18 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Impact of Virtualization ● Cost - #2 Minimize Storage and associated management Costs Minimize Storage and associated management Costs Redundancy reduced via ‘chunking’ vs. RAID Redundancy reduced via ‘chunking’ vs. RAID Capacity Optimization reduces storage 75% Capacity Optimization reduces storage 75% (delta-snapshot, dedupe, compression, thin provisioning) (delta-snapshot, dedupe, compression, thin provisioning) Eliminate “backup” – use data protection methods via delta- snapshots, chunking, replication Eliminate “backup” – use data protection methods via delta- snapshots, chunking, replication Self-healing systems Self-healing systems Virtualized infrastructure – scale on demand, more efficiency Virtualized infrastructure – scale on demand, more efficiency

19. 19 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Impact of Virtualization ● Cost - #3 Scale and Adjust infrastructure use dynamically Scale and Adjust infrastructure use dynamically On-demand, Virtual infrastructure On-demand, Virtual infrastructure Reduce power and cooling costs Reduce power and cooling costs Shift the cost burden Shift the cost burden Reduce staffing Reduce staffing

20. 20 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Impact of Virtualization ● Information Virtualization Information becomes self-describing, self-contained, and portable - Breaking the silos Information becomes self-describing, self-contained, and portable - Breaking the silos

21. 21 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Using the “Cloud” in Preservation ● Most likely use-cases: Private and Hybrid clouds Private and Hybrid clouds Virtualize infrastructure Virtualize infrastructure Virtualize delivery and access Virtualize delivery and access Virtualize emulation Virtualize emulation Virtualize information Virtualize information Providing portability Providing portability ● Examples Web-access models Web-access models Web-drop boxes Web-drop boxes Agile, Scalable, cost effective compute and storage resources (on demand) Agile, Scalable, cost effective compute and storage resources (on demand) Virtual emulation Virtual emulation Demand spikes Demand spikes Disaster recovery Disaster recovery Distributed data sets Distributed data sets Infrastructure extensions Infrastructure extensions

22. 22 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Emerging Cloud Standards ● Cloud Data Management Interface, CDMI SNIA to ISO: storage-to-cloud, cloud-to-cloud interchange format SNIA to ISO: storage-to-cloud, cloud-to-cloud interchange format ● Self-contained Information Retention Format, SIRF SNIA to ISO: extensible preservation object format SNIA to ISO: extensible preservation object format ● Interoperability ISO project: Data Preservation Interchange Framework, DPIF ISO project: Data Preservation Interchange Framework, DPIF

23. 23 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Cloud Data Management Interface ● Data Portability Standard with an Object Storage Interface Move data and metadata in standard portable containers in and out of the cloud and between clouds Move data and metadata in standard portable containers in and out of the cloud and between clouds Simple XML container of objects plus metadata Simple XML container of objects plus metadata ● A data and information services management interface and control path Operate services through CDMI Operate services through CDMI Rules and Policies in metadata Rules and Policies in metadata Cloud Peering – cloud to cloud communications Cloud Peering – cloud to cloud communications

24. 24 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Design for the Cloud ● Considerations Establish Service Objectives Establish Service Objectives Include verification of recovery, authenticity, availability, digital audit, etc. Include verification of recovery, authenticity, availability, digital audit, etc. Consider using multiple cloud destinations or local and remote copies for increased reliability and availability Consider using multiple cloud destinations or local and remote copies for increased reliability and availability Beware of excessive moving of data across the WAN due to high I/O and bandwidth costs Beware of excessive moving of data across the WAN due to high I/O and bandwidth costs ● Evaluate Cloud providers Establish strong contracts Establish strong contracts ● Test and Audit All required services All required services ● Use CDMI !

25. 25 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Cloud Contract Considerations Costs Costs Retention Management Retention Management Preservation/Integrity/ Authentication Preservation/Integrity/ Authentication Return and Secure Disposal – Subpoenas, Control Return and Secure Disposal – Subpoenas, Control Legal Hold Legal Hold Digital Audits & Verification Digital Audits & Verification Physical and logical migration practices and authenticity verifications Physical and logical migration practices and authenticity verifications Access Access Availability, Protection, Security & Confidentiality Availability, Protection, Security & Confidentiality Search/Discovery Search/Discovery Multi ‐ Cloud Provider Relationships Multi ‐ Cloud Provider Relationships Right to Conduct Forensic Exams Right to Conduct Forensic Exams Cross ‐ Border Data Transfers Cross ‐ Border Data Transfers

26. 26 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Summary Thoughts Preservation Architectures: Virtualization and Cloud

27. 27 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Move to Virtual Preservation ● Shift thinking from “Physical” Preservation to “Virtual” ● Virtualization Applies in many ways System, storage, application, infrastructure System, storage, application, infrastructure Information Information Migration – both physical and logical Migration – both physical and logical Cost reduction Cost reduction ● Conclusion: ‘Cloud’ has a positive role

28. 28 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Using the Cloud ● Start out Private, Move to Hybrid ● Apply Service Management Principles Classify, Requirements, SLAs, Design, Audit, Improve Classify, Requirements, SLAs, Design, Audit, Improve ● Design for the Cloud Create strong and measureable SLA style contracts Create strong and measureable SLA style contracts Test, Audit, Verify Test, Audit, Verify ● Use and Promote CDMI Need cloud interface, management, and information portability standards Need cloud interface, management, and information portability standards

29. 29 “Cross-Industry Preservation Architectures” – PASIG May, 2011 Contact Information ● Michael Peterson IMERGE consulting and LTDPRM.org IMERGE consulting and LTDPRM.org mpeterson@ltdprm.org mpeterson@ltdprm.org (805)201-3178 (805)201-3178