Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com

Agenda Progress architecture Hardware configuration discussion Database maintenance Performance tuning Database future direction

Database Internals Internal Blocks Physical Layout Data blocks Index blocks Other block types Physical Layout Data storage areas Primary recovery area After image journal Other storage

Database Blocks Master Blocks Data Block (RM Block) Index Block (Ix Block) Index Anchor Block Free Blocks Empty Blocks Notes:

Master Block This stores the “master” information for the database including: Area Status (opened, closed, crashed) Last opened date & time High water mark for the area Last backup date & time This information and more can be retrieved through the virtual system tables (VSTs) Notes:

Data Blocks These are also known as RM (Record Manager) blocks Can contain information from one or more tables They can be “full” (RM Blocks) Partially full (RM Chain blocks) Notes:

Record Storage In most environments, records are mixed from different tables in the same block Progress can store from 1 to 256 records per block per storage area All areas for a database must have the same block size (1 – 8kb) Total records per area is fixed More records per block equals lower total blocks Notes: Prior to version 9 any block 4k or less could only have 32 records and 8k blocks could have 64 records.

Index Blocks Also known as IX blocks Only contain information from one index Always considered partially full, blocks will split to accommodate growth Each block contains the address of itself, the next and previous blocks to support forward and reverse searches Notes:

Index Structure Balanced B-tree Compressed data All data access* through index *Except rowid access Notes:

Progress Index Layout Example 43-54 55-65 32-41 86-98 66-85 32-65 66-98 2-31 22-31 2-11 12-21 Record 65 Record 55 . Record 41 Record 32 Record 54 Record 43 Record 98 Record 86 Record 85 Record 66 Record 31 Record 22 Record 11 Record 2 Record 21 Record 12 Notes:

Free Blocks Contain address information No affiliation with IX or RM until utilized Under high water mark of the database Notes:

Empty Blocks White space No addresses Under total blocks of the database (area) Above high water mark Notes: No individual block allocation Reduced fragmentation of the database Utilities (Example: index rebuild) do not look at these blocks

Storage Areas Data objects Control Area Schema/Default Area Primary recovery area Application Data area(s) Notes:

Storage Areas – Data Objects Index object Table object Schema object Sequences Notes: Index objects contain index information. Each index object contains information about only 1 index. Table objects contain table information. Each index object contains information about only 1 table. There is only 1 schema object per database which contains information about all tables and indexes stored in the database. Sequences are stored in the schema (default) area but are treated differently than schema information. Think of this information as “quasi-schema”.

Control Area Always has a .db extension Describes the “physical schema” or layout of the database Lists storage areas and extents associated with a database Also known as the structure file Notes:

Default Storage Area Also known as the “schema” area This is always area 6 All information that is not assigned a storage area will be stored here All information is stored in this area if the data is converted from version 8 with a conv89 Notes:

Application Data Areas An area can contain 1 or more data objects An area can have 1 or more extents Tables and indexes can share areas These use area numbers 7-1000 Notes: The goal of areas should be to reduce downtime and improve performance Data that is sequentially may perform better if isolated to it’s own area The number of records per block is tunable per area

Primary Recovery Area General information Format Reuse Same as bi file(s) in version 8 and earlier This is always area number 3 Notes:

Primary Recovery Area General Info This process is automatic and can’t be turned off This file is vital to database integrity both physical and logical This file is generally sequential Notes:

Primary Recovery Area Format Each block is a cluster Each cluster contains information regarding transactions to allow transaction undo and redo The transaction information is called notes Each note contains a transaction id Notes:

Primary Recovery Area Reuse Clusters fill sequentially When the last formatted cluster is reached there is a reuse decision point The “oldest” cluster is examined to determine if it can be reused Then, either the oldest cluster is reused or another cluster is added, formatted and used Notes: Cluster reuse decision point. Once there are no more formatted clusters to use the decision point process happens at every checkpoint. A checkpoint is a synchronization point between memory, the database and the before image file that occurs every time a before image cluster is filled. Are all of the transactions in the “oldest” cluster completed? Has the before image truncate interval (-G Default is 60 seconds) expired?

Progress Memory Architecture The database engine can be serverless The database engine can be multi-server Progress applications can be host-based Progress applications can be 2-tier client/server Progress applications can be n-tier client/server Notes:

Shared Memory Host-based Configuration Record locks (-L) Buffers (-B) Index Cursors (-c) After Image Buffers Notes: This is a visual representation of the Progress database broker process. It is a misnomer to call this process a server as it is commonly called as each local user is self service. The local users connect directly to shared memory and update the structures without the help of a server process. The latch control table ensures that two people cannot update the same portion of memory at the same time. Before Image Buffers User Control Table Server Control Table Latch Control Table Other Stuff Hash Table

What are Latches? Concurrency control mechanism Very course in old versions of Progress More granular in current versions of Progress

Shared Memory Client/Server Configuration Record locks (-L) Listen Socket Buffers (-B) Index Cursors (-c) After Image Buffers Notes: The listen socket “listens” for requests for connection to come in over the network and then spawns servers or connects the user to an existing server for the duration of their session. All subsequent requests from that session will be directed to the server. A session will maintain a connection to a single server until it is disconnected. Before Image Buffers User Control Table Server Control Table Latch Control Table Other Stuff Servers Hash Table

Shared Memory Client/Server Configuration Listen Socket Database Broker Memory Notes: The listen socket “listens” for requests for connection to come in over the network and then spawns servers or connects the user to an existing server for the duration of their session. All subsequent requests from that session will be directed to the server. A session will maintain a connection to a single server until it is disconnected. AppServer AppServer Servers

Hardware Configurations Disk Considerations Memory Allocation CPU Considerations

Disk Contention In most environments disks are the largest area for improvement. All of the data flows from the disks to the other resources so this effects both local and networked users Notes: Disk at the root of 90% of performance problems. It may be the application asking for too much data (improper index use), a bad layout or not enough buffers but in any case the disks play a large role.

Balancing Disk I/O Balancing disk I/O is the process of making sure you are using all of the available disk resources (filesystems, disks and controllers) are working equally as hard at load. This is also called eliminating variance. A well tuned system will have less than a 15% variance Notes:

What Causes Disk I/O? Operating system (swapping and paging) Progress Database (DB and BI) Application (code and temp files) Other applications Notes:

What RAID Really Means RAID has many levels. I will only cover a few RAID 0: This level is also called striping. RAID 1: This is referred to as mirroring. RAID 5: Most common RAID level RAID 10: This is mirroring and striping. Also known as RAID 0 + 1 Notes:

Raid 0: Striping Disk 1 Disk 2 Disk 3 Volume Set Disk Array Stripe 1 Notes:

Raid 0: Striping (continued) Good for read and write I/O performance No failover protection Lower data reliability (1 fails they all fail) Notes:

Raid 1: Mirroring Primary Parity Disk 1 Disk 2 Parity 1 Parity 2 Notes:

Raid 1: Mirroring (continued) OK for read and write applications Good failover protection High data reliability Most expensive in terms of hardware Notes:

Raid 5: Poor Man’s Mirroring This is the kiss of death for OLTP performance User information is striped Parity information is striped WITH user information OK for 100% read only applications Poor performance for writes Notes: RAID 5’s problem lies in it’s write overhead. Each write must do the following: Write the primary information Calculate the parity information (compress) Write the calculated parity information. Software RAID compounds the problem as the calculations of parity contend for CPU with the user processes. Manufacturers will contend that they have “solved” the problem with hardware accelerators but RAID 5 will never be able to perform as well as standard mirroring (RAID 1).

Raid 10: Mirroring and Striping Good for read and write applications High level of data reliability though not as high as RAID 1 due to striping Just as expensive as RAID 1 Notes:

Software Methods for I/O Distribution Manual spread of data across non-striped disks Better control as you can see where the I/O is going More attention by system administrator is needed Notes:

Options Progress multi-volume Progress storage areas 8K database block size BI Cluster size Use page writers Move the temp-file I/O with -T Location of application files Use of program libraries to reduce I/O Notes:

Multi-Volume Database Progress-specific way to distribute I/O Only way to eliminate I/O indirection in a Progress environment Only way to pre-allocate database blocks Every database is multi-volume in Progress version 9 Notes: I/O indirection is when a file is so large that all of it’s addresses cannot be stored in a single i-node table (the i-node table stores the logical to physical address map for the file). If the file is very large (over 500MB) there is a probability that the file will require multiple tables to store the data and additional seeks will be required to locate information on the disk. OS manufacturers will content that indirection will not occur until files are greater the 2GB but testing has proven that the value is much lower. 500MB is a safe choice but if you have a very large database you can use 1GB extents and probably will not see indirection.

Storage Areas Benefits Drawbacks Greater control of location of data Minimize downtime for utilities Stripe some, leave some on straight disks Drawbacks More things to break More complex to monitor Notes:

Storage Areas - Control A storage area can hold 1 or more data objects (index, table, schema, …) Separate schema from data if possible Try to keep the number of areas manageable, only add more areas for valid business reasons Notes:

Minimize Downtime Smaller data areas allow utilities, such as off line index rebuild, to run faster as they have less blocks to scan Notes:

Database Administration Tools Backup and restore After image journaling Other Utilities

probkup Pros Cons Progress aware Supports online backup easy Slower than OS methods Does not backup more than the database Notes: Progress aware: knows the structure of the database Know the database status (running, down, crashed, …) Support incremental backups Will not backup empty blocks Compresses free blocks This is the only utility that can backup the database while it is running in multi-user mode. probkup only backs up the database and before image files. It does not get application files, the log file, or after image files. OS utilities must backup the entire database including the empty blocks at the end of each storage area.

prorest Utility to restore a Progress backup Can restore to a different structure provided there are enough storage areas Syntax: prorest dbname device_or_filename [-list | -vp | -vf] Notes: -list Provides a description of all application data storage areas contained within a database backup. The information can be used to create a new structure description file. -vp This is a verification pass on the tape. Probkup reads the backup volumes and computes and compares the backup blocks CRC with those in the block headers. -vf Specifies that the restore utility compares the backup to the database block-for-block. This will not work if the database you are trying to compare with is in use.

After Imaging Pros Cons Allows you to recover to present Recover from media failure Only way to “repair” catastrophic user error Cons Additional point of failure Adds complexity to the system Performance impact Notes:

How After Imaging Works FOR EACH CUSTOMER: UPDATE CUSTOMER. END. Before image note written After image note written Notes:

How to Integrate After Imaging In conjunction with a backup site To update a report server As a means of backup Notes: Every high availability system should have a tested backup strategy that includes after imaging.

AI to Update a Backup Site Poor man’s replication Allows for periodic update of a copy of the database The copy can then be backed up with a conventional backup mechanism Notes: Progress supports software replication for sites that need 100% up-to-date replication. In most cases, a system that includes asynchronous replication using after imaging will provide the proper level of protection from data loss.

AI to Update a Report Server Similar to keeping a backup site Requires two copies of the database in addition to the original (one for update and a second for reporting) The reporting database is a copy of the backup that is done periodically to keep the data synchronized

AI as a Means of Backup Not generally a good idea Increased recovery time Reduced reliability Backup the database each weekend Backup the AI file(s) each weeknight Notes:

Progress® Utilities Index rebuild DB analysis idxbuild idxfix idxcompact DB analysis Notes:

Progress® Utilities (continued) Truncate BI BI Grow Table move Index move Database log truncation Notes:

idxbuild Can only be run on a database that has been shutdown Can be run on 1 or more indexes Syntax: proutil <dbname> -C idxbuild [-TB n] [-TM n] [-T dirname] Notes: You will get significantly better performance if you rebuild indexes with sorting Sorting can take as much space as the database itself, in most cases 50% of the database size will be taken up by sorting. If the sort files will exceed 2GB you must use a dbname.srt file to specify the location and size of the sort files. Generally, it is a good idea to limit the size of each sort file to 500 MB. If you use a dbname.srt file you do not need to specify a –T parameter

idxfix Verifies index to record linkage Verifies index block to index block linkage Works online while in multi-user mode Syntax: proutil <dbname> -C idxfix Notes: Fixes index corruption only You will get a menu like this: Index Fix Utility Select one of the following: 1. Scan records for missing index entries. 2. Scan indexes for invalid index entries. 3. Both 1 and 2 above. 4. Cross-reference check of multiple indexes for a table. 5. Build indexes from existing indexes. 6. Delete one record and its index entries. 7. Quit Option 1 Scans the database records for missing or incorrect index entries. Option 2 Scans the index for corrupt index entries. You can scan 1 or more indexes Option 3 Does option 1 and option 2 Option 4 Prompts you for the table and indexes for which you want to run a cross-reference check. Looks for both invalid index entries and invalid records. Option 5 Prompts you to specify the table and the index that you want to use as the source from which to build another index for the same table. Option 6 Prompts you to specify the recid of the record you want to delete. Deletes one record and all its indexes from the database. Option 7 Ends the PROUTIL Index Fix utility

idxcompact Fast way to compress (reorganize) indexes online Utility will pass through the index several times (number of index levels + 1) Runs online Syntax: proutil <dbname> -C idxcompact [owner- name.]table-name.index-name [n] Notes: The default level of compaction is 80%, The n at the end of the command specifies the level of compaction that you want. If the index has a greater level of compaction than you specify it will retain that level of compaction after the utility is run. The index compacting utility operates in phases: Phase 1: If the index is a unique index, the delete chain is scanned and the index blocks are cleaned up by removing deleted entries. Phase 2: The non-leaf levels of the B-tree are compacted starting at the root working toward the leaf level. Phase 3: The leaf level is compacted. You cannot run any other administrative operation on an index that is being compacted, idxfix can be run serially but not in parallel with index compact

Database Analysis ixanalys – analysis of indexes chanalys – analysis of record chains dbanalys – analysis of records and indexes Syntax: proutil <dbname> -C XXanalys Notes: Sample output dbanalys: SUMMARY FOR AREA "Inventory" : 8 ------------------------------------------------------- Records Indexes Combined NAME Size Tot % Size Tot % Size Tot % PUB.Bin 26.3K 1.1 16.4K 0.7 42.8K 1.8 PUB.InventoryTrans 3661B 0.1 0B 0.0 3661B 0.1 PUB.Item 7662B 0.3 8117B 0.3 15779B 0.6 PUB.POLine 217.5K 9.1 51.9K 2.2 269.4K 11.3 PUB.PurchaseOrder 68.6K 2.9 19.0K 0.8 87.6K 3.7 PUB.Supplier 1177B 0.0 97B 0.0 1274B 0.1 Sample output ixanalys: INDEX BLOCK SUMMARY FOR AREA "Order" : 11 Table Index Fields Levels Blocks Size % Util Factor PUB.BillTo custnumbillto 46 2 1 1 27B 2.7 1.0 PUB.Order CustOrder 48 2 2 44 39.1K 91.4 1.5 OrderDate 49 1 2 33 18.5K 57.7 2.1 OrderNum 47 1 2 38 35.3K 95.5 1.5 PUB.ShipTo custnumshipto 45 2 1 1 36B 3.6 1.0

Truncate BI Reduce for size of the BI file Change the cluster size of the BI file Change the block size of the BI file Syntax: proutil <dbname> -C truncate bi [-bi n] [-biblocksize n] [-G n] Notes: Use only when modifying BI cluster or block size Or After abnormal growth of the BI file (after a large schema change) -bi specifies the BI cluster size in kb. Generally, a BI cluster size between 1024 (1 MB) and 8192 (8 MB) is ideal depending on transaction load -biblocksize specifies the BI block size in kb (16kb max) Either 8kb (v9 default) or 16 kb (Maximum) is ideal

BI Grow After truncation it is best to pre-grow your BI file to it’s anticipated size Keeps BI sequential (good for performance) Database must be shutdown Syntax: proutil <dbname> -C bigrow n Notes: The n at the end of the command specifies the number of BI clusters that you want to add to the BI file

Table Move Allows the movement from one storage area to another Works “online” Uses 4-times the amount of BI space as is taken up by the table Syntax: proutil <dbname> -C tablemove [owner- name.]table-name table-area [index-area] Notes: While the table is being moved a lock is placed on the entire table and any updates to this table will be delayed until the table move is complete. This is why the word online is quoted above. owner-name: Specifies the owner of the table containing the data you want to dump. You must specify an owner name unless the table's name is unique within the database, or the table is owned by "PUB." By default, Progress 4GL tables are owned by PUB. table-name: The name of the table to be moved. table-area: Specifies the target storage area name which the table is to be moved. Optionally, You can specify the name of the target index area. If the target index area is supplied, the indexes will be moved to that area. Otherwise they will be left in their existing location. Note: Area names with spaces in the name must be quoted, for example, "Area Name." index-area

Index Move Allows movement of indexes from one storage area to another Works “online” Uses a significant amount of BI space Syntax: proutil db-name -C indexmove [owner- name.]table-name.index-name area-name Notes: Basically, the index is locked (either shared or exclusive) for the duration of the process. The PROUTIL INDEXMOVE utility operates in two phases: Phase 1: The new index is being constructed in the new area. The old index remains in the old area and all users can continue to use the index for read operations. Phase 2: The old index is being killed and all the blocks of the old index are being removed to the free block chain. For a large index, this phase might take a significant amount of time. During this phase all operations on the index are blocked until the new index is available; users accessing the index might experience a freeze in their applications. owner-name: Specifies the owner of the table containing the data you want to dump. You must specify an owner name unless the table's name is unique within the database, or the table is owned by "PUB." By default, Progress 4GL tables are owned by PUB. table-name: Specifies the source table containing the index to be moved. index-name: Specifies the name of an index to move. area-name: Specifies the target storage area name into which the index is to be moved. Note: Area names that contain spaces must be quoted. For example, "Area Name."

Database Log Truncation Reduces the size of the log file Database must be down for it to work Syntax: prolog <dbname> Notes: There is a truncate_log script in the $SCRIPTS directory that will truncate log files while the database is up and running.

Performance Tuning - Basics Before Image cluster size Database block size Tuning APWs Memory tips Increasing CPU efficecy

Networking Tips Keep things local No temp files on network drives Move the application “close” to the user Use -cache to speed initial connection Use -pls if you are using program libraries over the network Application issues are magnified over a network (field-lists, no-lock, indexes, …) Notes: The best way to improve performance in a network environment is to avoid using the network. This is the reason web applications are more efficient than n-tier and n-tier is more efficient than client server. A well written network application will work well locally but the converse is not true.

Networking Tips (Continued) -Mm 8192 to increase the tcp packet size from 1k to 8k -Ma Increase the number of servers to reduce or eliminate server contention Notes: Progress will fill these packets as much a possible and when smaller amounts of data are sent (lock requests) then smaller packets will be sent. View -Mm as a upper size limit.

Stripe Some, Leave Others Flat Tables that are accessed sequentially may benefit from being isolated to their own table space Randomly accessed tables will generally perform better on striped volumes Disk systems that have read ahead algorithms will help sequential access most when placed on a single disk (or mirror) Notes:

8k Block Size Most systems will benefit from using 8k block size (NT should use 4k) You will retrieve more information per physical I/O especially on index reads I/O is done how the operating likes it to be done Notes: You may need to increase the number of records per block to avoid wasting space in the database.

BI Cluster Size Somewhere between 1MB and 4MB works for most people If you are checkpointing every 2 minutes or more often during peak periods increase the cluster size If you a “workgroup” version of Progress leave your cluster size alone (512kb) Don’t forget to use bigrow to avoid allocating clusters one at a time Notes: There are cases where larger cluster sized are necessary. In load scenarios it is generally a good idea to set the BI cluster size to a large value (32MB). This is especially important when using the binary load facility. BI Grow pre-formats BI clusters to eliminate the eliminate the initial formatting of clusters. It also allows the BI file to be used and reused in a more sequential manner.

Progress® Page Writers Every database that does updates should have a before image writer (BIW) Every database that does updates should have at least 1 asynchronous page writer (APW) Every database that is using after imaging should have a after image writer (AIW) Notes:

Tuning APWs Start with 1 APW Monitor buffers flushed at checkpoint on the activity screen (option 5) in promon If buffers flushed increases during the “important” hours of the day add 1 APW Notes: If you do an online backup or a quiet point you will see additional buffers flushed and these need additional buffers need to be ignored. There are cases where buffers flushed cannot be eliminated (i.e.. Online backup), so if you add APWs and buffers flushed does not decrease go back to the previous number of APWs.

Use -T to Level Disk I/O Local (host based) users and batch jobs should use the -T parameter to place their temporary file (.srt, .pge, .lbi, …) I/O on a drive that is not working as hard as the other drives on the system Note: -T should never point to a network drive Notes: Network users (Client/Server) should always keep the temporary files local to the client process. By default, the temporary files go in the working directory of the client process.

Application Files Keep paths short say run <subdir>/program to eliminate unnecessary searches Put programs into libraries (prolib) to reduce I/O to temp files Libraries use a hashed search mechanism for better performance Notes:

Memory Contention Memory should be used to reduce disk I/O. Broker (server) side parameters should be tuned first and then user parameters can be modified. In a memory lean situation, memory should be taken away from individual users before reducing broker parameters Notes: A memory lean situation can cause a disk contention issue by swapping

Memory Hints Swapping is bad, buy more memory or reduce parameters to avoid it Increase -B in 10% increments until the point of diminishing returns or swapping, whichever comes first Use V9 private buffers (-Bp) for reporting Do not use private buffers (-I) prior to V9 Notes: Use the memory you have but leave a buffer for growth. At small -B settings (Less than 100MB) higher incremental changes may get you to the proper setting faster.

Memory Hints(continued) Use memory for the users closest to the customer first (developers increase last) Use -Bt for large temp tables Set -bibufs between 50 and 120. Look at the activity screen in promon (BI buffer waits) to see if additional tuning is necessary. Start with 50 as this will work for the vast majority of people Notes:

CPU Contention High CPU activity is not bad in and of itself but high system CPU activity is bad and should be corrected Notes: High system CPU activity can be caused by other operations or functions, like NFS or Samba.

Components of CPU Activity USER - This is what you paid for SYSTEM - This is overhead WAIT - This is waste IDLE - This is nothing ;-) Notes:

CPU Activity Goals The goal is to have as much USER time as possible with as little SYSTEM and WAIT A practical split is USER: 70% SYSTEM: 20% WAIT: 0% IDLE: 10% Notes: In many cases you can get 75+% User so try to get the best you can by adjusting -spin, -napmax and workloads. Remember: Look at your baseline timings and make sure they are improving. Good statistics do not equal good performance is all cases.

Eliminating High SYSTEM CPU Activity Always use -spin Use a setting of 1 for single CPU systems Use a higher setting for multiple CPU systems Testing has shown that the optimal setting for -spin is somewhere between 2000 and 10000. First try 2000 -napmax should default to 5000 but in some late 7 and early 8 versions of Progress it is set to 100 which is way too low Notes: Spin can be modified within promon or through VSTs. Promon: R&D, option 4, Option 4, Option 1 To see the value of napmax: promon, R&D, debghb, Option 4, Option 4

Eliminating High WAIT CPU Activity WAIT = Waiting on I/O If you still have IDLE time it generally is not a big problem Look at paging/swapping first Next look at your disk I/O Notes: Generally, it is good to maintain about 5% idle. In some cases, Wait will be logged as idle.

Progress Database Future Directions Increased uptime through online utilities Increased speed of utilities to maintain the database Support for clusters to increase reliability Open standards support

Replication New feature in 9.1D of Progress Fathom High Availability Allows for single or bi-directional replication Target database can be used for update or reporting

Replication (continued) Source database has an agent that forwards changes to the target database(s) Only one agent per database One or more targets per agent Raw record format is used to increase performance and reduce overhead

Questions