1. Getting Started Using Database Archiving Toronto DAMA Chapter Meeting 16 September, 2009 Jack E. Olson jack.olson@SvalTech.com www.svaltech.com SvalTech “Database Archiving: How to Keep Lots of Data for a Long Time” Jack E. Olson, Morgan Kaufmann, 2008 Copyright SvalTech, Inc., 2009

2. 2 Why This Presentation SvalTech A common position of many IT shops is –We know we should be doing database archiving –We know it will be valuable to us –But we don’t know how to get started Database archiving is an enterprise technology: it can be used in many applications Not all database applications are suitable for database archiving Suitable applications have widely differing return-on-investment potential Copyright SvalTech, Inc., 2009

3. 3 The Database Archiving Survey SvalTech organize survey team application enumeration first-cut feasibility data-life-cycle analysis operational analysis risk analysis metric gathering evaluate implementation options business case development prioritization Copyright SvalTech, Inc., 2009

4. 4 The Survey Organization SvalTech Mandate People Inputs Mandate A management directive that creates the database archiving survey task force and gives them the scope and objectives of the study. Scope : business units to include, organizational units (divisions, companies, campuses) Objectives: find best candidates for cost reduction, fixing operational problems, risk reduction Copyright SvalTech, Inc., 2009

5. 5 The Survey Team SvalTech Mandate People Inputs Chair Fulltime members IT/enterprise architect storage administration records retention Subject matter members database architect data management business unit data analyst database administration Incidental members legal department IT compliance data governance security administration data analyst (BI type) Copyright SvalTech, Inc., 2009

6. 6 Starting Materials SvalTech Mandate People Inputs Enterprise data model Data classification results SLA’s IT storage strategy Regulations/compliance rules Data governance mandates Copyright SvalTech, Inc., 2009

7. 7 Application Enumeration SvalTech Limit search to those within mandate business unit location enterprise Identify Operational Applications classify as transactional vs. static data include those already archiving to any extent Identify Retired Applications still retaining data Identify applications about to change consolidations planned planned or recent acquisitions replacements/ conversions/ reengineering identify any strategies for application retirement Copyright SvalTech, Inc., 2009

8. 8 Application Enumeration SvalTech For Applications with potential, Capture application data model Identify business records within the data model Connect business records to records retention and legal categories Identify database information: system/dbms/file/metadata Create a Database Topology chart Identify parallel applications within the corporation (even if out of scope) Identify operational replicates Identify backup/disaster recovery stores and strategies Identify recurring data extracts for BI, etc. Get rough idea of db size and transaction rates Copyright SvalTech, Inc., 2009

9. 9 Database Topology Chart SvalTech create data operational replicate operational BI stores CRM archive backup disaster recovery offline storage Copyright SvalTech, Inc., 2009

10. 10 First-Cut Feasibility SvalTech Factors for continuing to consider, important data lots of data lots of individual business records simple data structures relatively stable data structures (little change) long retention requirement long inactive period within retention requirement low frequency access requirement in inactive period low performance requirement in inactive period simple access requirements in inactive period Apply criteria after each subsequent step to further eliminate bad candidates Copyright SvalTech, Inc., 2009

11. 11 Examples SvalTech GoodNot Good Bank deposits and withdrawalsCustomer master data Stock tradesAirplane manufacturing records Credit card transactionsHR records Ticketmaster transactionsFelony records Medical claim dataHome sales Casualty claim data (auto, home) Retail sales inventory transactions Package tracking Passenger flight data Driver license records Sales tax records Property tax records Telephone call transactions Nuclear reactor monitoring records Auto warrantee records Copyright SvalTech, Inc., 2009

12. 12 Data Life Cycle Analysis SvalTech Create a database archiving DLCA for each business record type Data Retention Chart Business Record Process Chart to determine inactive period Business Record SLA chart by age of record Copyright SvalTech, Inc., 2009

13. 13 Data Retention Chart SvalTech The requirement to keep data for a business object for a specified period of time. The object cannot be destroyed until after the time for all such requirements applicable to it has past. Business Requirements Regulatory Requirements The Data Retention requirement is the longest of all requirement lines. Copyright SvalTech, Inc., 2009

14. 14 Business Record Process Chart SvalTech for a single instance of a data object Create PO Update PO Create Invoice Backorder Create Financial Record Update on Ship Update on Ack Weekly Sales Report Quarterly Sales report Extract for data warehouse Extract for bus analysis Common customer queries Common bus queries Ad hoc requests Law suit e-Discovery requests Investigation data gathering Retention requirement operationalreferenceinactive time Copyright SvalTech, Inc., 2009

15. 15 Business Record SLA Chart by Age SvalTech for a single instance of a data object Query response time Transaction volume create/update Security (no users) read Retention requirement operationalreferenceinactive time Copyright SvalTech, Inc., 2009

16. 16 Operational Analysis SvalTech Don’t assume there are no problems. Talk to DBAs and users. Look for trends Look for escalating operational costs. Get numbers. Copyright SvalTech, Inc., 2009

17. 17 Operational Analysis SvalTech Performance Issues –Not meeting response time SLA –Longer time to run extracts –Longer time to run backups –Longer time to run database reorganizations –Running reorganizations more frequently –More difficult to tune Risk Issues –Longer estimated time to run recovery –Longer estimated time to run disaster recovery Cost Issues –Higher annual hardware costs –Higher annual MIP-based software cost –Adding expensive DASD to support database and backups Copyright SvalTech, Inc., 2009

18. 18 Risk Analysis SvalTech Data Loss Risk –Isolation from internet hackers –Prevent ANY updates or deletes –Preserve data through multi-site backups and periodic pings Data Quality Risk –Changing data structures and column semantics –Changing reference data Unauthorized Access Risk –Reduced (or different) user set –Audit trail of access Legal Risk –Preserve authenticity of data in archive –Reduce cost and time to produce data for discovery requests Copyright SvalTech, Inc., 2009

19. 19 Metric Gathering SvalTech Data bytes stored per business object new transactions created per day bytes for backups, replicates growth in transactions rates any sudden expected additions past history plus future projections Storage Costs cost per byte: operational cost per byte: backup cost per byte: archive archive compression ratios System Costs mips required to process software license fees staff for operational Copyright SvalTech, Inc., 2009

20. 20 Metric Gathering SvalTech For retired applications concentrate on displaced system cost displaced software cost displaced staff cost IBM mainframe IMS DBMS CICS DBA/SYSPROG LINUX server Archive software JDBC Archive admin NOT shared Shared Copyright SvalTech, Inc., 2009

21. 21 Evaluate Implementation Options SvalTech Software –Vendor provided software –Custom built solution Access tools –Original application –Generic report generation/ query tools –Custom built Storage for archive –Storage subsystem –Hosted storage Copyright SvalTech, Inc., 2009

22. 22 Architecture of Database Archiving Archive Server Operational System archive catalog archive storage OP DB Archive Administrator Archive Designer Archive Data Manager Archive Access Manager SvalTech Archive Extractor Application program Archive extractor Copyright SvalTech, Inc., 2009

23. 23 Estimate Implementation Time and Cost SvalTech Archiving systems required –Servers –Storage systems (hosted storage?) –Licensed software Application Design Implementation Test Deployment Ongoing operation and administration Copyright SvalTech, Inc., 2009

24. 24 Business Case Development SvalTech –Lower IT costs –Improved operational efficiency –Risk reduction Copyright SvalTech, Inc., 2009

25. 25 Lower IT Costs SvalTech Systems –Reduce size/cost of operational systems –Put off or eliminate need for system upgrades Software –Eliminate or reduce cost of expensive system software DBMS Transaction system –Eliminate or reduce cost of application software Storage costs –Switch to lower cost storage –Impact on backups/ disaster recovery stores –Reduction in byte count stored Staff –Eliminate or reduce legacy system staff Copyright SvalTech, Inc., 2009

26. 26 Chart it operational operational archive All data in operational db most expensive system most expensive storage most expensive software Inactive data in archive db least expensive system least expensive storage least expensive software In a typical op db 60-80% of data is inactive This percentage is growing SvalTech Size Today Copyright SvalTech, Inc., 2009

27. 27 Lower IT Costs SvalTech First year impact Time to recover project costs Chart cost savings over time –Plot data growth over time for operational –Plot data growth over time of archive Copyright SvalTech, Inc., 2009

28. 28 Operational Improvements SvalTech Itemize improvements expected –Performance of operations –Reduction of utility times –Reduction of recovery times –Reduction of disaster recovery times –Reduction of DBA workload Provide cost savings where appropriate Copyright SvalTech, Inc., 2009

29. 29 Risk Reduction SvalTech Itemize improvements expected –Less risk of failing e-Discovery request –Enhanced data quality of older data –Less exposure to loss of data authenticity –Better access control –Better compliance –Better data governance –Less dependence on legacy systems Provide cost savings where appropriate Copyright SvalTech, Inc., 2009

30. 30 Prioritization SvalTech –Determine Prioritization Criteria Cost is most common primary factor –First archiving project may have other goals Lower risk of failure Faster implementation Faster return on investment Usually a retired application project –Risk may over-ride other factors Preserve data authenticity Copyright SvalTech, Inc., 2009

31. 31 Final Thought SvalTech Always do a survey to find the best applications to start with Always do a survey to identify those that make sense to proceed with versus those that do not: don’t waste time on apps that are too hard to implement or that will have little value. A good database archive application can save millions of dollars per year, increase performance of operational systems and reduce risk all at the same time. The trick is identifying them and proving it. Repeat the Database Archiving Survey from time to time in the future. Copyright SvalTech, Inc., 2009