1. Partition, Compress, Archive and Purge Keep Your System on the Road David Kurtz Go-Faster Consultancy Ltd. david.kurtz@go-faster.co.uk www.go-faster.co.uk

2. Who Am I? Oracle Database Specialist –Independent consultant Performance tuning –PeopleSoft ERP –Oracle RDBMS Book –www.psftdba.comwww.psftdba.com OakTable UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 2

3. It depends… The answer to all questions should be deemed to be implicitly prefixed it depends if they are not already explicitly so prefixed. The reasons for this include, but are not limited to: –There may be exceptions where the answer is either false or not completely true –There may be exceptions to the exceptions and so on ad infinitum. –The question may not explicitly scope all the conditions upon which the answer depends ©2013 www.go-faster.co.uk 3 UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge

4. This happens to be a story about... An ERP System –But it could be about any on-line transaction processing system. A PeopleSoft system –But it could be about any ERP PeopleSoft HR/Time & Labour/Payroll –But it could be about any product UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 4

5. PeopleSoft HR, T&L, GP @ Morrisons 130,000 employees –100,000 in stores Scheduled hours –Clock in/out –T&L matches reported hours with schedules to determine payable time –Legacy T&L feed from Manufacturing and Distribution. –Payable time passed to Global Payroll. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 5

6. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 6 0.91Tb/yr

7. Two Challenges Performance –Big Tables (for OLTP) Eg payroll accumulator table 2.4 billion rows! –Poor/Variable Performance Various processes and reports suffering Critical Processes Database Size –>4TB –Growth 1TB/year –Backup Size/Contention –Cloning Dev/Test environments UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 7

8. Throw Away When we say the database is slow, often: –It is fetching data from disk, reading it, and deciding it isnt something it wants and throwing it away. –It would be better if we had not spent time and resource fetching it in the first place The fastest way to do anything is not to do it at all. –Cary Millsap UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 8

9. 4 Techniques Partition –Breaking tables and indexes into managable pieces Compress –Squeezing more data into the same space Archive –Get the data out of the live application tables and put it somewhere else. Purge –Expunge the data from the database UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 9

10. PARTITIONING UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 10

11. Partitioning UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 11

12. Partitioning decisions driven by Application Global Payroll –streamed processing –RANGE on EMPLID to match streaming Financials General Ledger –FISCAL_YEAR & ACCOUNTING_PERIOD –Monthly/Quarterly/Annual partitions. Other Financials Modules? –Less Obvious UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 12

13. Partitioning is a Pre-requisite Links logical data value to physical location of data in the database. Time-based partitioning –Future / Current / Historical Partition-wise operations –Compress historical partitions –Archive by partition exchange –Purge by dropping whole partitions. –Less Redo. Faster. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 13

14. Delete rows –v- Drop Partition? Free space is left in data block. Data block remain part of table/index Still scanned by queries. Still backed up by RMAN No longer in a database object. So not referenced. Free space is available in tablespace. –Still backed up If the space at end of the datafile it can be trimmed off by resize. –No longer backed up UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 14

15. Managing Partitioning in PeopleSoft Partitioning strategies –Described own meta-data tables PL/SQL Utility package –generate DDL scripts. –Driven by PeopleTools tables and our own meta-data Application Designer from PT8.51 will preserve some existing partitioning –See Managing Oracle Table Partitioning in PeopleSoft Applications with GFC_PSPART Package http://www.go-faster.co.uk/gfc_pspart.manual.pdf UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 15

16. Partitioning PeopleSoft at Morrisons Global Payroll –46 tables range on EMPLID, matching 32 streams –List sub-partition largest tables by pay period (CAL_RUN_ID). –24 archive tables, range partitioned on pay period into tax-year partitions. –2 custom GP reporting tables Schedules & Time and Labor –10 tables in weekly range partitions –of which Reported Time table hash sub-partitioned Audit Tables –28 monthly range partitions on audit timestamp Security Tables –Separate partitions for different business units UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 16

17. Partitioning Trick Timezone Table –Small, but accessed VERY frequently by T&L –1 partition for GMT only Only 1 row –1 partition for CET only Only 1 row –1 partition for the rest –Less CPU to read first row in block than second etc. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 17

18. Licencing of Partitioning Oracle RDBMS Enterprise Edition only –And then separately licenced Also a cost option on other database platforms. No licence implication for PeopleSoft. –Partitioning does not invalidate any aspect of your support. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 18

19. COMPRESSION UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 19

20. Oracle RDBMS Compression Basic Compression –Not separately licenced –Only option in 10g Advanced Compression Out of scope of this presentation –Licenced option –Available in 11g Hybrid Columnar Compression (HCC) Also out of scope of this presentation. –Exadata and ZFS filesystem only. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 20

21. Simple Compression Block Dictionary Encoding –Data values stored only stored once in array in data block. –Pointer from row of data to array of values. Shorter Rows, more rows per block, fewer Blocks, less I/O More CPU to read row. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 21

22. Table Compression Specified at table or partition level with DDL –Sub-partition level inherits from partition level Compression only occurs –On creation of the object ALTER TABLE … COMPRESS –Direct path insert into object –CPU overhead during compression and read Decompression occurs –during any other DML UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 22

23. Index Compression Apply to all or none of index Index remains compressed –DML does not decompress –Performance overhead We have not compressed many live application indexes. We have compressed all archive tables. Table compression does not affect index size. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 23

24. What do I compress? Depends on application, business, and partitioning strategy. –Audit data partitions: as soon as month ends –Payroll result data: when pay period is closed for 1 month –T&L: after 6 months UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 24

25. Compression Gotcha! During –ALTER TABLE PARTITION … MOVE … COMPRESS. Local index partitions can momentarily become unusable before they go back to being usable. This can cause application errors –Choose when to compress carefully, or do it in an outage! UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 25

26. When can I compress my data? When it is no longer updated by application. –Because DML uncompresses it –But that may be well be you can archive/purge. Written custom PL/SQL utility package –Meta-data rules to determine when partition no longer updated. –Needs knowledge of business/application. –Incrementally compress static partitions. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 26

27. How much compression? It depends on your data! We have see a factor 2x – 5x on tables –(50%-80% reduction in size) Indexes are less compressible –Indexes with more columns can be more compressible UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 27

28. PL/SQL Utility Package GFC_ARCH_MGMT –DBA run management utility –Moves partitions to correct tablespace –Compresses/Decompresses partitions per specification in meta-data. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 28

29. ARCHIVE & PURGE UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 29

30. Archive & Purge Archive: –Move data from live application table to another table. –In this case another to table in the same database. –Read-only access provided to archive tables. Purge: –Expunge data from the database. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 30

31. Archive/Purge For each table or set of tables –Define a policy that specifies when data is archived or purged. Compression too, but that has no functional impact. Discuss with business –Check legal requirements PAYE 3 years, Company Accounts 6 years UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 31

32. But my data will disappear That may be a good thing –If you keep data you have to look after it responsibly. –Pay for the disk to store it/back it up. Read-only access via PS/Query tool to archive data. A very few components and reports access archive tables –Development cost of customisation UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 32

33. Example Archive/Purge Policies –GP: archive 2 rolling years, purge after 8 tax years –Schedules, purge at 56 weeks –T&L, purge at 108 weeks –Audit: various: Archive after 3 months, 1, 3 year Purge 6 months, 1 year, 2 years, never UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 33

34. Archive Method Partition Management –Exchange/Drop whole Partitions Time based partitioning –Avoid rewriting data –Avoids redo Faster –Compression must match Some archive table indexes not compressed. Row-by-Row –Non-time based partitioning GP –Can be done on-line –Can compress data during rewrite –Different partitioning strategy in archive and live. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 34

35. PeopleTools Archive Utility Row-by-Row processing only –Slow and cant reclaim space –Cant compress data during copy Additional key columns in archive table –Enlarges archive tables –Causes performance problems with index access Still need to drop empty partitions. Generic –Meta-data driven Sets of tables to archive Corresponding archive tables GP set delivered UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 35

36. PL/SQL Archive/Purge Utility Custom PL/SQL Package –Driven by same meta-data as partitioning utility –Archive by partition exchange –Purge by dropping partition. –Called from custom Application Engine So archiving can be done via application. –Writes To PeopleSoft Message Log Various Oracle trace/log files UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 36

37. No-Archive Rules UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 37

38. Custom Archive Utility Based on delivered PeopleSoft Archive Utility We only use it for GP archiving Copies data in direct path –Faster –Can compress during copy Calls PL/SQL utility to drop partitions to save deleting data. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 38

39. PSARCH Schema Design 2 nd schema to hold archive tables –Only tables archived by partition exchange –Same name as live tables –Same local indexes as live Benefits –SYSADM cannot update/delete from these tables. Drawbacks –Views to make them accessible to PS/Query. –Cant manage indexes through Application Designer. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 39

40. Tablespace Design No impact on performance –Because all on same tier of disk Do not keep objects with dissimilar purge policies in same tablespaces –You can drop empty tablespaces and recover space. Periodic tablespaces make sense without purge policy With a purge policy might just have one tablespace. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 40

41. Wrong Tablespace Design Before archive policy –Weekly T&L and Schedule partitions in same monthly tablespaces Archive Policy –T&L: 108 weeks –Schedules: 56 weeks Schedules populated first, T&L later. –Schedules tend to be nearer start of tablespace –Schedules purged first leaving hole than cannot be trimmed off. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 41

42. Free Space Fragmentation Free Space left in middle of tablespace –Dropping objects –Compressing objects can leave holes Less Severe since Oracle 9i –Uniform extent sizes –Bitmap space map UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 42

43. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 43

44. Oracle RMAN Backup Backs up block with SCN –If an object is dropped blocks still have SCN –Empty blocks can still be backed up. Can only reduce backup size by trimming free space from end of data files. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 44

45. Defragmentation of Free Space PL/SQL utility to rebuild objects nearest end of data file into spaces in the middle of the tables –Needs to be done during an application outage. Frees space at end of data file that can be trimmed off. –Thus reducing size of database and backup. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 45

46. Oracle Flashback Database SQL Rollback –We are used to the idea of rolling back transaction that have not been committed. Flashback database –Generates additional flashback logging –Rollback committed transaction and DDL but not through a drop/resize data file –Whole database option Highly effective in testing archive/purge UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 46

47. Future: Oracle 11g Automatic Partition Creation Range-Range partitioning Advanced Compression –Licenced Option –Supports DML Active Data-Guard –Off-load query activity to standby database UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 47

48. PROGRESS UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 48

49. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 49

50. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 50 0.91Tb/yr 0.48Tb/yr

51. UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 51

52. Achievements Database Size –4.5TB => <2.25TB (with GP purge) Still have to purge old schedule and T&L data Expected Performance Improvements –GP calculation and other processing –T&L processing –Financial Reports UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 52

53. Conclusion It depends… UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 53

54. QUESTIONS? Nullius in verba UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 54

55. You know my methods, apply them. A Study in Scarlet, Arthur Conan-Doyle UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge ©2013 www.go-faster.co.uk 55

56. Partition, Compress, Archive and Purge Keep Your System on the Road David Kurtz Go-Faster Consultancy Ltd. david.kurtz@go-faster.co.uk www.go-faster.co.uk