1. Progress Database Setup, Maintenance and Tuning
Adam Backman
V.P. of Technology
White Star Software

2. Agenda Progress architecture Hardware configuration discussion
Database maintenance
Performance tuning
Database future direction

3. Database Internals Internal Blocks Physical Layout Data blocks
Index blocks
Other block types
Physical Layout
Data storage areas
Primary recovery area
After image journal
Other storage

4. Database Blocks Master Blocks Data Block (RM Block)
Index Block (Ix Block)
Index Anchor Block
Free Blocks
Empty Blocks
Notes:

5. Master Block
This stores the “master” information for the database including:
Area Status (opened, closed, crashed)
Last opened date & time
High water mark for the area
Last backup date & time
This information and more can be retrieved through the virtual system tables (VSTs)
Notes:

6. Data Blocks These are also known as RM (Record Manager) blocks
Can contain information from one or more tables
They can be “full” (RM Blocks)
Partially full (RM Chain blocks)
Notes:

7. Record Storage
In most environments, records are mixed from different tables in the same block
Progress can store from 1 to 256 records per block per storage area
All areas for a database must have the same block size (1 – 8kb)
Total records per area is fixed
More records per block equals lower total blocks
Notes:
Prior to version 9 any block 4k or less could only have 32 records and 8k blocks could have 64 records.

8. Index Blocks Also known as IX blocks
Only contain information from one index
Always considered partially full, blocks will split to accommodate growth
Each block contains the address of itself, the next and previous blocks to support forward and reverse searches
Notes:

9. Index Structure Balanced B-tree Compressed data
All data access* through index
*Except rowid access
Notes:

10. Progress Index Layout Example
43-54
55-65
32-41
86-98
66-85
32-65
66-98
2-31
22-31
2-11
12-21
Record 65
Record 55 .
Record 41
Record 32
Record 54
Record 43
Record 98
Record 86
Record 85
Record 66
Record 31
Record 22
Record 11
Record 2
Record 21
Record 12
Notes:

11. Free Blocks Contain address information
No affiliation with IX or RM until utilized
Under high water mark of the database
Notes:

12. Empty Blocks White space No addresses
Under total blocks of the database (area)
Above high water mark
Notes:
No individual block allocation
Reduced fragmentation of the database
Utilities (Example: index rebuild) do not look at these blocks

13. Storage Areas Data objects Control Area Schema/Default Area
Primary recovery area
Application Data area(s)
Notes:

14. Storage Areas – Data Objects
Index object
Table object
Schema object
Sequences
Notes:
Index objects contain index information. Each index object contains information about only 1 index.
Table objects contain table information. Each index object contains information about only 1 table.
There is only 1 schema object per database which contains information about all tables and indexes stored in the database.
Sequences are stored in the schema (default) area but are treated differently than schema information. Think of this information as “quasi-schema”.

15. Control Area Always has a .db extension
Describes the “physical schema” or layout of the database
Lists storage areas and extents associated with a database
Also known as the structure file
Notes:

16. Default Storage Area Also known as the “schema” area
This is always area 6
All information that is not assigned a storage area will be stored here
All information is stored in this area if the data is converted from version 8 with a conv89
Notes:

17. Application Data Areas
An area can contain 1 or more data objects
An area can have 1 or more extents
Tables and indexes can share areas
These use area numbers
Notes:
The goal of areas should be to reduce downtime and improve performance
Data that is sequentially may perform better if isolated to it’s own area
The number of records per block is tunable per area

18. Primary Recovery Area General information Format Reuse
Same as bi file(s) in version 8 and earlier
This is always area number 3
Notes:

19. Primary Recovery Area General Info
This process is automatic and can’t be turned off
This file is vital to database integrity both physical and logical
This file is generally sequential
Notes:

20. Primary Recovery Area Format
Each block is a cluster
Each cluster contains information regarding transactions to allow transaction undo and redo
The transaction information is called notes
Each note contains a transaction id
Notes:

21. Primary Recovery Area Reuse
Clusters fill sequentially
When the last formatted cluster is reached there is a reuse decision point
The “oldest” cluster is examined to determine if it can be reused
Then, either the oldest cluster is reused or another cluster is added, formatted and used
Notes:
Cluster reuse decision point.
Once there are no more formatted clusters to use the decision point process happens at every checkpoint.
A checkpoint is a synchronization point between memory, the database and the before image file that occurs every time a before image cluster is filled.
Are all of the transactions in the “oldest” cluster completed?
Has the before image truncate interval (-G Default is 60 seconds) expired?

22. Progress Memory Architecture
The database engine can be serverless
The database engine can be multi-server
Progress applications can be host-based
Progress applications can be 2-tier client/server
Progress applications can be n-tier client/server
Notes:

23. Shared Memory Host-based Configuration
Record locks
(-L)
Buffers (-B)
Index Cursors
(-c)
After Image
Buffers
Notes:
This is a visual representation of the Progress database broker process. It is a misnomer to call this process a server as it is commonly called as each local user is self service. The local users connect directly to shared memory and update the structures without the help of a server process.
The latch control table ensures that two people cannot update the same portion of memory at the same time.
Before Image
Buffers
User
Control
Table
Server
Control
Table
Latch
Control
Table
Other
Stuff
Hash Table

24. What are Latches? Concurrency control mechanism
Very course in old versions of Progress
More granular in current versions of Progress

25. Shared Memory Client/Server Configuration
Record locks
(-L)
Listen
Socket
Buffers (-B)
Index Cursors
(-c)
After Image
Buffers
Notes:
The listen socket “listens” for requests for connection to come in over the network and then spawns servers or connects the user to an existing server for the duration of their session. All subsequent requests from that session will be directed to the server.
A session will maintain a connection to a single server until it is disconnected.
Before Image
Buffers
User
Control
Table
Server
Control
Table
Latch
Control
Table
Other
Stuff
Servers
Hash Table

26. Shared Memory Client/Server Configuration
Listen
Socket
Database
Broker
Memory
Notes:
The listen socket “listens” for requests for connection to come in over the network and then spawns servers or connects the user to an existing server for the duration of their session. All subsequent requests from that session will be directed to the server.
A session will maintain a connection to a single server until it is disconnected.
AppServer
AppServer
Servers

27. Hardware Configurations
Disk Considerations
Memory Allocation
CPU Considerations

28. Disk Contention
In most environments disks are the largest area for improvement. All of the data flows from the disks to the other resources so this effects both local and networked users
Notes:
Disk at the root of 90% of performance problems. It may be the application asking for too much data (improper index use), a bad layout or not enough buffers but in any case the disks play a large role.

29. Balancing Disk I/O
Balancing disk I/O is the process of making sure you are using all of the available disk resources (filesystems, disks and controllers) are working equally as hard at load. This is also called eliminating variance. A well tuned system will have less than a 15% variance
Notes:

30. What Causes Disk I/O? Operating system (swapping and paging) Progress
Database (DB and BI)
Application (code and temp files)
Other applications
Notes:

31. What RAID Really Means RAID has many levels. I will only cover a few
RAID 0: This level is also called striping.
RAID 1: This is referred to as mirroring.
RAID 5: Most common RAID level
RAID 10: This is mirroring and striping. Also known as RAID 0 + 1
Notes:

32. Raid 0: Striping Disk 1 Disk 2 Disk 3 Volume Set Disk Array Stripe 1
Notes:

33. Raid 0: Striping (continued)
Good for read and write I/O performance
No failover protection
Lower data reliability (1 fails they all fail)
Notes:

34. Raid 1: Mirroring Primary Parity Disk 1 Disk 2 Parity 1 Parity 2
Notes:

35. Raid 1: Mirroring (continued)
OK for read and write applications
Good failover protection
High data reliability
Most expensive in terms of hardware
Notes:

36. Raid 5: Poor Man’s Mirroring
This is the kiss of death for OLTP performance
User information is striped
Parity information is striped WITH user information
OK for 100% read only applications
Poor performance for writes
Notes:
RAID 5’s problem lies in it’s write overhead. Each write must do the following:
Write the primary information
Calculate the parity information (compress)
Write the calculated parity information.
Software RAID compounds the problem as the calculations of parity contend for CPU with the user processes.
Manufacturers will contend that they have “solved” the problem with hardware accelerators but RAID 5 will never be able to perform as well as standard mirroring (RAID 1).

37. Raid 10: Mirroring and Striping
Good for read and write applications
High level of data reliability though not as high as RAID 1 due to striping
Just as expensive as RAID 1
Notes:

38. Software Methods for I/O Distribution
Manual spread of data across non-striped disks
Better control as you can see where the I/O is going
More attention by system administrator is needed
Notes:

39. Options Progress multi-volume Progress storage areas
8K database block size
BI Cluster size
Use page writers
Move the temp-file I/O with -T
Location of application files
Use of program libraries to reduce I/O
Notes:

40. Multi-Volume Database
Progress-specific way to distribute I/O
Only way to eliminate I/O indirection in a Progress environment
Only way to pre-allocate database blocks
Every database is multi-volume in Progress version 9
Notes:
I/O indirection is when a file is so large that all of it’s addresses cannot be stored in a single i-node table (the i-node table stores the logical to physical address map for the file). If the file is very large (over 500MB) there is a probability that the file will require multiple tables to store the data and additional seeks will be required to locate information on the disk.
OS manufacturers will content that indirection will not occur until files are greater the 2GB but testing has proven that the value is much lower. 500MB is a safe choice but if you have a very large database you can use 1GB extents and probably will not see indirection.

41. Storage Areas Benefits Drawbacks Greater control of location of data
Minimize downtime for utilities
Stripe some, leave some on straight disks
Drawbacks
More things to break
More complex to monitor
Notes:

42. Storage Areas - Control
A storage area can hold 1 or more data objects (index, table, schema, …)
Separate schema from data if possible
Try to keep the number of areas manageable, only add more areas for valid business reasons
Notes:

43. Minimize Downtime
Smaller data areas allow utilities, such as off line index rebuild, to run faster as they have less blocks to scan
Notes:

44. Database Administration Tools
Backup and restore
After image journaling
Other Utilities

45. probkup Pros Cons Progress aware Supports online backup easy
Slower than OS methods
Does not backup more than the database
Notes:
Progress aware:
knows the structure of the database
Know the database status (running, down, crashed, …)
Support incremental backups
Will not backup empty blocks
Compresses free blocks
This is the only utility that can backup the database while it is running in multi-user mode.
probkup only backs up the database and before image files. It does not get application files, the log file, or after image files.
OS utilities must backup the entire database including the empty blocks at the end of each storage area.

46. prorest Utility to restore a Progress backup
Can restore to a different structure provided there are enough storage areas
Syntax:
prorest dbname device_or_filename [-list | vp | -vf]
Notes:
-list
Provides a description of all application data storage areas contained within a database backup. The information can be used to create a new structure description file.
-vp
This is a verification pass on the tape. Probkup reads the backup volumes and computes and compares the backup blocks CRC with those in the block headers.
-vf
Specifies that the restore utility compares the backup to the database block-for-block. This will not work if the database you are trying to compare with is in use.

47. After Imaging Pros Cons Allows you to recover to present
Recover from media failure
Only way to “repair” catastrophic user error
Cons
Additional point of failure
Adds complexity to the system
Performance impact
Notes:

48. How After Imaging Works
FOR EACH CUSTOMER:
UPDATE CUSTOMER.
END.
Before image note written
After image note written
Notes:

49. How to Integrate After Imaging
In conjunction with a backup site
To update a report server
As a means of backup
Notes:
Every high availability system should have a tested backup strategy that includes after imaging.

50. AI to Update a Backup Site
Poor man’s replication
Allows for periodic update of a copy of the database
The copy can then be backed up with a conventional backup mechanism
Notes:
Progress supports software replication for sites that need 100% up-to-date replication. In most cases, a system that includes asynchronous replication using after imaging will provide the proper level of protection from data loss.

51. AI to Update a Report Server
Similar to keeping a backup site
Requires two copies of the database in addition to the original (one for update and a second for reporting)
The reporting database is a copy of the backup that is done periodically to keep the data synchronized

52. AI as a Means of Backup Not generally a good idea
Increased recovery time
Reduced reliability
Backup the database each weekend
Backup the AI file(s) each weeknight
Notes:

53. Progress® Utilities Index rebuild DB analysis idxbuild idxfix
idxcompact
DB analysis
Notes:

54. Progress® Utilities (continued)
Truncate BI
BI Grow
Table move
Index move
Database log truncation
Notes:

55. idxbuild Can only be run on a database that has been shutdown
Can be run on 1 or more indexes
Syntax:
proutil <dbname> -C idxbuild [-TB n] [-TM n] [-T dirname]
Notes:
You will get significantly better performance if you rebuild indexes with sorting
Sorting can take as much space as the database itself, in most cases 50% of the database size will be taken up by sorting.
If the sort files will exceed 2GB you must use a dbname.srt file to specify the location and size of the sort files. Generally, it is a good idea to limit the size of each sort file to 500 MB.
If you use a dbname.srt file you do not need to specify a –T parameter

56. idxfix Verifies index to record linkage
Verifies index block to index block linkage
Works online while in multi-user mode
Syntax:
proutil <dbname> -C idxfix
Notes: Fixes index corruption only You will get a menu like this:
Index Fix Utility
Select one of the following:
1. Scan records for missing index entries.
2. Scan indexes for invalid index entries.
3. Both 1 and 2 above.
4. Cross-reference check of multiple indexes for a table.
5. Build indexes from existing indexes.
6. Delete one record and its index entries.
7. Quit
Option 1 Scans the database records for missing or incorrect index entries.
Option 2 Scans the index for corrupt index entries. You can scan 1 or more indexes
Option 3 Does option 1 and option 2
Option 4 Prompts you for the table and indexes for which you want to run a cross-reference check. Looks for both invalid index entries and invalid records.
Option 5 Prompts you to specify the table and the index that you want to use as the source from which to build another index for the same table.
Option 6 Prompts you to specify the recid of the record you want to delete. Deletes one record and all its indexes from the database.
Option 7 Ends the PROUTIL Index Fix utility

57. idxcompact Fast way to compress (reorganize) indexes online
Utility will pass through the index several times (number of index levels + 1)
Runs online
Syntax:
proutil <dbname> -C idxcompact [owner- name.]table-name.index-name [n]
Notes:
The default level of compaction is 80%, The n at the end of the command specifies the level of compaction that you want. If the index has a greater level of compaction than you specify it will retain that level of compaction after the utility is run.
The index compacting utility operates in phases:
Phase 1: If the index is a unique index, the delete chain is scanned and the index blocks are cleaned up by removing deleted entries.
Phase 2: The non-leaf levels of the B-tree are compacted starting at the root working toward the leaf level.
Phase 3: The leaf level is compacted.
You cannot run any other administrative operation on an index that is being compacted, idxfix can be run serially but not in parallel with index compact

58. Database Analysis ixanalys – analysis of indexes
chanalys – analysis of record chains
dbanalys – analysis of records and indexes
Syntax:
proutil <dbname> -C XXanalys
Notes: Sample output dbanalys:
SUMMARY FOR AREA "Inventory" : 8
Records Indexes Combined
NAME Size Tot % Size Tot % Size Tot %
PUB.Bin K K K
PUB.InventoryTrans B B B
PUB.Item B B B 0.6
PUB.POLine K K K
PUB.PurchaseOrder K K K
PUB.Supplier B B B
Sample output ixanalys:
INDEX BLOCK SUMMARY FOR AREA "Order" : 11
Table Index Fields Levels Blocks Size % Util Factor
PUB.BillTo
custnumbillto B
PUB.Order
CustOrder K
OrderDate K
OrderNum K
PUB.ShipTo
custnumshipto B

59. Truncate BI Reduce for size of the BI file
Change the cluster size of the BI file
Change the block size of the BI file
Syntax:
proutil <dbname> -C truncate bi [-bi n]
[-biblocksize n] [-G n]
Notes:
Use only when modifying BI cluster or block size Or
After abnormal growth of the BI file (after a large schema change)
-bi specifies the BI cluster size in kb.
Generally, a BI cluster size between 1024 (1 MB) and 8192 (8 MB) is ideal depending on transaction load
-biblocksize specifies the BI block size in kb (16kb max)
Either 8kb (v9 default) or 16 kb (Maximum) is ideal

60. BI Grow
After truncation it is best to pre-grow your BI file to it’s anticipated size
Keeps BI sequential (good for performance)
Database must be shutdown
Syntax:
proutil <dbname> -C bigrow n
Notes:
The n at the end of the command specifies the number of BI clusters that you want to add to the BI file

61. Table Move Allows the movement from one storage area to another
Works “online”
Uses 4-times the amount of BI space as is taken up by the table
Syntax:
proutil <dbname> -C tablemove [owner- name.]table-name table-area [index-area]
Notes:
While the table is being moved a lock is placed on the entire table and any updates to this table will be delayed until the table move is complete. This is why the word online is quoted above.
owner-name: Specifies the owner of the table containing the data you want to dump. You must specify an owner name unless the table's name is unique within the database, or the table is owned by "PUB." By default, Progress 4GL tables are owned by PUB.
table-name: The name of the table to be moved.
table-area: Specifies the target storage area name which the table is to be moved.
Optionally, You can specify the name of the target index area. If the target index area is supplied, the indexes will be moved to that area. Otherwise they will be left in their existing location.
Note: Area names with spaces in the name must be quoted, for example, "Area Name."
index-area

62. Index Move Allows movement of indexes from one storage area to another
Works “online”
Uses a significant amount of BI space
Syntax:
proutil db-name -C indexmove [owner- name.]table-name.index-name area-name
Notes:
Basically, the index is locked (either shared or exclusive) for the duration of the process.
The PROUTIL INDEXMOVE utility operates in two phases:
Phase 1: The new index is being constructed in the new area. The old index remains in the old area and all users can continue to use the index for read operations.
Phase 2: The old index is being killed and all the blocks of the old index are being removed to the free block chain. For a large index, this phase might take a significant amount of time. During this phase all operations on the index are blocked until the new index is available; users accessing the index might experience a freeze in their applications.
owner-name: Specifies the owner of the table containing the data you want to dump. You must specify an owner name unless the table's name is unique within the database, or the table is owned by "PUB." By default, Progress 4GL tables are owned by PUB.
table-name: Specifies the source table containing the index to be moved.
index-name: Specifies the name of an index to move.
area-name: Specifies the target storage area name into which the index is to be moved.
Note: Area names that contain spaces must be quoted. For example, "Area Name."

63. Database Log Truncation
Reduces the size of the log file
Database must be down for it to work
Syntax:
prolog <dbname>
Notes:
There is a truncate_log script in the $SCRIPTS directory that will truncate log files while the database is up and running.

64. Performance Tuning - Basics
Before Image cluster size
Database block size
Tuning APWs
Memory tips
Increasing CPU efficecy

65. Networking Tips Keep things local
No temp files on network drives
Move the application “close” to the user
Use -cache to speed initial connection
Use -pls if you are using program libraries over the network
Application issues are magnified over a network (field-lists, no-lock, indexes, …)
Notes:
The best way to improve performance in a network environment is to avoid using the network. This is the reason web applications are more efficient than n-tier and n-tier is more efficient than client server.
A well written network application will work well locally but the converse is not true.

66. Networking Tips (Continued)
-Mm 8192 to increase the tcp packet size from 1k to 8k
-Ma Increase the number of servers to reduce or eliminate server contention
Notes:
Progress will fill these packets as much a possible and when smaller amounts of data are sent (lock requests) then smaller packets will be sent.
View -Mm as a upper size limit.

67. Stripe Some, Leave Others Flat
Tables that are accessed sequentially may benefit from being isolated to their own table space
Randomly accessed tables will generally perform better on striped volumes
Disk systems that have read ahead algorithms will help sequential access most when placed on a single disk (or mirror)
Notes:

68. 8k Block Size
Most systems will benefit from using 8k block size (NT should use 4k)
You will retrieve more information per physical I/O especially on index reads
I/O is done how the operating likes it to be done
Notes:
You may need to increase the number of records per block to avoid wasting space in the database.

69. BI Cluster Size Somewhere between 1MB and 4MB works for most people
If you are checkpointing every 2 minutes or more often during peak periods increase the cluster size
If you a “workgroup” version of Progress leave your cluster size alone (512kb)
Don’t forget to use bigrow to avoid allocating clusters one at a time
Notes:
There are cases where larger cluster sized are necessary. In load scenarios it is generally a good idea to set the BI cluster size to a large value (32MB). This is especially important when using the binary load facility.
BI Grow pre-formats BI clusters to eliminate the eliminate the initial formatting of clusters. It also allows the BI file to be used and reused in a more sequential manner.

70. Progress® Page Writers
Every database that does updates should have a before image writer (BIW)
Every database that does updates should have at least 1 asynchronous page writer (APW)
Every database that is using after imaging should have a after image writer (AIW)
Notes:

71. Tuning APWs Start with 1 APW
Monitor buffers flushed at checkpoint on the activity screen (option 5) in promon
If buffers flushed increases during the “important” hours of the day add 1 APW
Notes:
If you do an online backup or a quiet point you will see additional buffers flushed and these need additional buffers need to be ignored.
There are cases where buffers flushed cannot be eliminated (i.e.. Online backup), so if you add APWs and buffers flushed does not decrease go back to the previous number of APWs.

72. Use -T to Level Disk I/O
Local (host based) users and batch jobs should use the -T parameter to place their temporary file (.srt, .pge, .lbi, …) I/O on a drive that is not working as hard as the other drives on the system
Note: -T should never point to a network drive
Notes:
Network users (Client/Server) should always keep the temporary files local to the client process.
By default, the temporary files go in the working directory of the client process.

73. Application Files Keep paths short
say run <subdir>/program to eliminate unnecessary searches
Put programs into libraries (prolib) to reduce I/O to temp files
Libraries use a hashed search mechanism for better performance
Notes:

74. Memory Contention
Memory should be used to reduce disk I/O. Broker (server) side parameters should be tuned first and then user parameters can be modified. In a memory lean situation, memory should be taken away from individual users before reducing broker parameters
Notes:
A memory lean situation can cause a disk contention issue by swapping

75. Memory Hints
Swapping is bad, buy more memory or reduce parameters to avoid it
Increase -B in 10% increments until the point of diminishing returns or swapping, whichever comes first
Use V9 private buffers (-Bp) for reporting
Do not use private buffers (-I) prior to V9
Notes:
Use the memory you have but leave a buffer for growth.
At small -B settings (Less than 100MB) higher incremental changes may get you to the proper setting faster.

76. Memory Hints(continued)
Use memory for the users closest to the customer first (developers increase last)
Use -Bt for large temp tables
Set -bibufs between 50 and Look at the activity screen in promon (BI buffer waits) to see if additional tuning is necessary. Start with 50 as this will work for the vast majority of people
Notes:

77. CPU Contention
High CPU activity is not bad in and of itself but high system CPU activity is bad and should be corrected
Notes:
High system CPU activity can be caused by other operations or functions, like NFS or Samba.

78. Components of CPU Activity
USER - This is what you paid for
SYSTEM - This is overhead
WAIT - This is waste
IDLE - This is nothing ;-)
Notes:

79. CPU Activity Goals
The goal is to have as much USER time as possible with as little SYSTEM and WAIT
A practical split is USER: 70%
SYSTEM: 20%
WAIT: 0%
IDLE: 10%
Notes:
In many cases you can get 75+% User so try to get the best you can by adjusting -spin, napmax and workloads.
Remember: Look at your baseline timings and make sure they are improving.
Good statistics do not equal good performance is all cases.

80. Eliminating High SYSTEM CPU Activity
Always use -spin
Use a setting of 1 for single CPU systems
Use a higher setting for multiple CPU systems
Testing has shown that the optimal setting for -spin is somewhere between 2000 and First try 2000
-napmax should default to 5000 but in some late 7 and early 8 versions of Progress it is set to 100 which is way too low
Notes:
Spin can be modified within promon or through VSTs.
Promon:
R&D, option 4, Option 4, Option 1
To see the value of napmax:
promon, R&D, debghb, Option 4, Option 4

81. Eliminating High WAIT CPU Activity
WAIT = Waiting on I/O
If you still have IDLE time it generally is not a big problem
Look at paging/swapping first
Next look at your disk I/O
Notes:
Generally, it is good to maintain about 5% idle.
In some cases, Wait will be logged as idle.

82. Progress Database Future Directions
Increased uptime through online utilities
Increased speed of utilities to maintain the database
Support for clusters to increase reliability
Open standards support

83. Replication New feature in 9.1D of Progress Fathom High Availability
Allows for single or bi-directional replication
Target database can be used for update or reporting

84. Replication (continued)
Source database has an agent that forwards changes to the target database(s)
Only one agent per database
One or more targets per agent
Raw record format is used to increase performance and reduce overhead

85. Questions