1. OPS-10: Moving V8/V9 RDBMS to OpenEdge® 10
Rob Marshall
Principal Solutions Consultant

2. What’s in OpenEdge RDBMS?
Performance
Visibility
Type II Storage Areas
Fast Drop & Temp tables
Increased shmem –B 1 billion
Internal algorithmic enhancements
Buffers, Locks, Indexing
Improved APW scanning
Auto Record Defrag
Enhanced Txn Backout
New Defaults
Log File
New format
Significant events
Improved management
Db I/O by User by Object
Database Describe
Large Database Support
64 bit Rowids
64 bit Sequences
64 bit Integer Datatype
Large Index Key Entries (1970)
32,000 areas
8 TB Shmem
Datatype Support
OpenEdge is a re-branding of the Progress products that you already know.
There is an abundant list of new features in OpenEdge
BLOB, CLOB
Datetime, Datetime-TZ
INT64 (no conversion)
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

3. What’s in OpenEdge RDBMS?
High Availability
Maintenance
Online Schema adds
Sequences
Tables
Fields
Indexes w/fast fill
Online space management
Enabled/Disable AI online
Enable AI Mgmt online
HA Clusters Bundled
Index Rebuild
By area, table, active status
.st file syntax checker
AI Management
Multi threaded utilities
idxbuild, binary dump
Binary Dump without index
Binary Load Performance
Index Fix with NO-LOCK
OpenEdge is a re-branding of the Progress products that you already know.
There is an abundant list of new features in OpenEdge
Security
SSL
Auditing
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

4. Agenda General Migration Strategy The Fast and Easy Upgrade
Physical Upgrade
Tuning Opportunity
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

5. First detail plan, review plan, test plan THEN execute…
Basic Steps
First detail plan, review plan, test plan THEN execute…
Preparation
Truncate BI, Disable AI, 2PC, and Replication (V9)
Backup (V8/9)
Install
Install OpenEdge
Don’t need to uninstall V8/9
*Don’t overwrite your current Progress Directory
Upgrade
Upgrade DB to OpenEdge 10
Do your backups !!!!
Recompile/Re-deploy your ABL code if client is OpenEdge
Run the Application and Test, Test and….. Test
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

6. The Application Safe Fast and Loose The Rollout
Install OpenEdge on your existing test machine
proutil <test db> -C conv910*
Recompile application* and test
Fast and Loose
Upgrade a remote or local client site
The Rollout
Upgrade remote systems with OpenEdge
Remote client server, Remote Application servers
“re-deploy” newly built application (PROPATH) and test
* ABL code needs to be recompiled/re-deployed only when upgrading the client to R10. In 3-tier configurations (with AppServer™) the client could still be V9. Not possible via SQL or V8.
* You will have to convert a V8 db to V9 before converting to OpenEdge
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

7. Agenda General Migration Strategy The Fast and Easy Upgrade
Physical Upgrade
Tuning Opportunity
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

8. Deployment Preparation Install OpenEdge 10 on server machine
Truncate BI
Disable AI, 2PC, Replication (V9)
Backup database (V8/9)
Validate backup
Install OpenEdge 10 on server machine
And everywhere else! (Clients must be upgraded before the server can be)
Recompile and re-deploy application (if need be)
Client/Server V9 to OpenEdge 10 disallowed
V9 Clients to R10 AppServer to R10 Database is allowed, No SQL V9 to OpenEdge permitted
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

9. Connectivity: Progress® V9 and OpenEdge 10
ABL mixed configurations: Progress V9 and OpenEdge 10 are supported
One version back (clients to servers)
One version forward (clients to Application Server)
SQL must match
Client
Application Server
Database 10 10 10 10 10 9 10 9 9
NEW in
OpenEdge 10 9 10 10 9 10 9
Refer to product by product information for further details

10. Database Conversion Run conversion utility
_proutil <db> -C conv910 –B 512
Conversion runs in 5-minutes or less
Basically just a schema upgrade
No changes to user records or indexes
No changes to physical structures
Backup database
Re-start database and application
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

11. You are now “Good To Go” with OpenEdge
No physical changes to existing user data
Data remains in Type I storage areas
Data fragmentation exists as before
Optimize physical layout when time permits...
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

12. Agenda General Migration Strategy The Fast and Easy Upgrade
Physical Upgrade
Tuning Opportunity
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

13. Storage Areas? Performance, Scalability & Maintenance
Take advantage of new features
No adverse application effects
Physical reorg does NOT change the application
Object location is abstracted from the language by an internal mapping layer
Different physical deployments can run with the same compiled r-code
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

14. How to Get There Preparation (same as before) Before Physical Reorg
Truncate BI, disable AI, backup, validate, install…
Before Physical Reorg
Upgrade database to OpenEdge 10
conversion utility
prostrct create (a must if changing block size)
Physical Updates (no r-code changes required)
Separate schema from user data
Create new storage areas
Specify records per block
Specify Type II cluster sizes
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

15. How to Get There Physical Reorg Online options vs offline options
Spread data out amongst new areas
Move indexes
Online options vs offline options
Database block size changes are offline
After Reorg
Reclaim Unused Space
Truncate old data area
Delete old data area
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

16. How to Get There In Place (same database) New database
Transformation done all at once or over time
Add new storage areas to existing database
Migrate data from old areas to new
Reclaim space from old areas
New database
Transformation done in one maintenance window
Dump old data
Create new database
Load into new database
Prodel old database
Mix of Option #1 and Option #2 (custom)
Move data from old database to new database
Reclaim space by deleting old database
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

17. Separate user data from schema
Getting started
Separate user data from schema
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

18. Moving schema tables Separate Schema from user data (in place):
proutil <db> -C mvsch (offline operation)
Schema Area
Renames existing
schema area
Old Default Area
For best performance and Ease of maintenance…
Schema move DOES not increase the size of the database if the Old Default Area space is reclaimed.
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

19. Moving schema tables Separate Schema from user data:
proutil <db> -C mvsch (offline operation)
Schema Area
Renames existing
schema area
Creates new
schema area
Old Default Area
Schema
Area
Moves schema
Tables & Indexes
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

20. Moving schema tables Separate Schema from user data:
proutil <db> -C mvsch (offline operation)
Schema Area
Renames existing
schema area
Creates new
schema area
Old Default Area
Schema
Area
Moves schema
Tables & Indexes
You move data
To new areas
User
Area
User
Area
User
Area
User
Area
You truncate
Old Default Area
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

21. Physical Changes: Location, Location, Location
Create .st file with new layout
Set records per block
Use Type II Storage Areas
Tables 64 or 512 block clusters
Indexes 8 or 64 block clusters
d “Cust/Bill Indexes“:7,1;8 /d_array2/myDB_7.d1 f
d “Cust/Bill Indexes“:7,1;8 /d_array2/myDB_7.d2 #
d “Customer Data“:8,16;64 /d_array2/myDB_8.d1 f
d “Customer Data“:8,16;64 /d_array2/myDB_8.d2
d “Billing Data“:9,32; /d_array2/myDB_9.d1 f
d “Billing Data“:9,32; /d_array2/myDB_9.d2
If in place, watchout for collisions.
Notice area number specification purposely missing.
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

22. The Structure file format is valid. (12619)
Physical Changes
Validate first
prostrct add <db> new.st -validate
Then update
prostrct add <db> new.st
OR:
prostrct addonline <db> new.st
The Structure file format is valid. (12619)
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

23. Moving Tables and Indexes
3 Options for Data Movement
Table move and Index move
Online (by primary index)
Dump and Load (D&L)
With or without index rebuild
Application must be offline
Suggestion: Mix of option #1 and #2
1st purge/archive unneeded data
Table move small tables (number of blocks)
D&L everything else
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

24. Option #1: Table/Index Move
Pros and Cons
Advantages:
Online (with no-lock access)
Dump & load in one step
Schema is automatically updated
No object # changes to deal with
Parallelism
Disadvantages:
Only No-lock accessible during move
Moves but doesn’t “rebuild” indexes
Too slow for large tables
Changes are logged!
BI growth may be of concern
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

25. Table Move
proutil <db> -C tablemove [ owner-name . ] table-name table-area [ index-area ]
Move/reorg a table by its primary index
Move a table AND its indexes
Preferred performance
Fast for small tables
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

26. Index Move
proutil <db> -C idxmove [ owner-name . ] table-name . index-name
area-name
Move an index from one area to another
Does NOT alter/optimize index block layout
Fast but does not “rebuild” indexes
Online but changes to owning table blocked
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

27. Option #2: Dump and Load 3 Dump and Load Flavors Textual Data Binary
ASCII dump
ASCII load
Bulk load followed by index rebuild
Binary
Binary dump
Binary load
With index rebuild
Followed by index rebuild
Custom (Textual or raw)
D&L with triggers
Buffer-Copy / Raw-data-transfer / Export/Import
Can be tricky, you may want help
Beware of multi loaders as the order of the loaded records is typically random
State that customer dumps you must do your self.
Can dump multiple subsets of the data
Reference Tom’s talk “Highly Parallel Dump and Load”
Not going to state anything else on this.
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

28. Dump and Load General Strategy
Create new database structure
Add to existing DB
New database
Run tabanalys
Dump table data sequence values, _User table
Data definitions
Dump definitions
Modify storage area locations
Load definitions
Load table data
Build indexes (if needed) [10.1C can specify pack]
Backup
If you have the disk space, creating a new db saves time
Running tabanalysis to validate number of records
To load the new definitions to an existing database you must delete the old table
* If you have the disk space, creating a new db saves time
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

29. Dumping the data
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

30. Dictionary Data Dump Database Admin tool
OR: run prodict/dump_d.p(<table>, <dir>,<codepage>).
Advantages:
Fast and Easy
Parallel
No endian issues
Disadvantages:
2 GB File size limit** Pre 10.1C
Can’t choose dump order
Have to dump entire table
Must ensure no one changes table between D&L
Parallel refers to the ability to dompu more than one table at a time.
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

31. Using Binary Dump Advantages: Disadvantages: Fastest and Easy
No 2 GB file size limit
No endian issues
Can choose dump order (by index)
Can dump table data in portions
Multi threaded (10.1B)
Can dump multiple tables concurrently (parallel)
Disadvantages:
Must ensure no table changes between D&L (unless using triggers as well)
Since not character based, can’t change char set on load
Can’t ETL to another vendor
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

32. Binary Dump Specified
proutil <db> -C dumpspecified <table.field> <operator> ‘field-value1 AND operator value2’ <dir> -preferidx <idx-name>
10.1B03 allows multiple values
Switches
table.field MUST be lead participant in index
Valid operators: LT, GE, LE, GT, EQ
-preferidx determines specific index to use
-index, -thread are ignored
Performance
Threaded is preferred
Can run in parallel with many unique values
Cautions
Avoid using descending indexes
There is a risk of missing a range
To get all data, specify LT for one process, GE for the other process.
Need Median Value
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

33. Finding the median value
Binary Dump Specified
Finding the median value
define variable i as integer no-undo.
define variable max-recs as integer initial 0 no-undo.
define variable median as integer no-undo.
for each mytable NO-LOCK use-index <preferred-idx>:
max-recs = max-recs + 1.
end.
median = max-recs / 2.
do i = 1 to median:
find next mytable NO-LOCK
use-index <preferred-idx>.
display i substr(field1, 1, 192).
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

34. Binary Dump Threaded
proutil <db> -C dump <table> <dir> -index <index #>
-thread 1 -threadnum <n>
-dumpfile <filelist> -Bp 64
-index <n>
Choose index based on read order
-index 0
Faster dump, slower read
Assumes coming from Type II
-thread indicates threaded dump
# threads automatic (# CPUs)
–threadnum max of # CPUs * 2
Threads only available in multi user mode
Workgroup only supports 1 thread
-dumpfile used as input for load
First introduced in 10.1A
-Bp used to avoid contention in the bufferpool LRU for multiple dumps
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

35. Don’t forget SQL SQL Tables and Views
These need to be dumped and loaded as well
You can use different commands to move the information:
sqlschema
sqldump
sqlload
First introduced in 10.1A
-Bp used to avoid contention in the bufferpool LRU for multiple dumps
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

36. Reorganize the Area/Object Configuration
Data Dump Completed.
Reorganize the Area/Object Configuration
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

37. Dump & Modify data definitions
Use Data administration tool
OR:
run prodict/dump_df.p(‘ALL’, ‘<mydb>.df’, ‘ ’).
If using bulk load:
run prodict/dump_fd.p(‘ALL’, ‘<mydb>.fd’).
Data dump used. Can be done on a table or ALL basis
Explain good practices of naming areas and separating data and indexes.
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

38. Dump & Modify data definitions
Update .df files
Optionally delete old table
Change table’s area information
Delete/Drop tables
Load data definitions
Data administration tool
OR: run prodict/load_df.p(“<mytable>.df").
ADD TABLE "mytable2"
AREA “Old Default Area"
DROP Table “mytable2”
AREA “New Data Area"
Data dump used. Can be done on a table or ALL basis
Explain good practices of naming areas and separating data and indexes.
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

39. Alternative Data Definition Modification
If all data in area dumped…
Truncate objects for fast move/delete
proutil <db> -C truncate area “Old Default Area”
Warns then deletes data (but NOT schema)
Rebuild/activate empty indexes (if moving)
proutil <db> -C idxbuild inactiveindexes
Can be done ONLINE, not just the build but the activate
Move “empty” tables/indexes to new area
proutil <db> -C tablemove <table> <area> [ index-area ]
Table/index move preserves the table/index numbers
Neither method has any effect on CRCs
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

40. Load the data back in (finally). Remember to load all data files.
Be sure to validate data loaded.
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

41. Loading Things to consider... Enable large file support
In the Operating System (ulimit)
In the Filesystem / volume groups
In the Database
Must rebuild indexes
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

42. Bulkload
proutil <db> -C bulkload <fd-file> -B 1000 –i –Mf 10
Data input from dictionary or custom data dump
Mentioned here for completeness only
Drawbacks:
2 GB file limit (pre 10.1C)
Loads one table at a time (single user)
Does not insert index entries
Requires index rebuild as separate step
No advantage over other loads
Slower than all other loads
Must rebuild indexes
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

43. Dictionary Load Data Administration Tool OR:
run prodict/load_d.p(‘table1’, ‘table1.d’).
Data input from dictionary or custom data dump
2 GB file limit per load (pre 10.1C)
Load data in parallel (to separate tables)
Inserts index entries
Index tree not perfect
Performance close to binary load + index rebuild
(when loading multiple tables)
Must rebuild indexes
The index trees grow and are split as they grow so index layout not as perfect as idxbuild
Single table, binary load 116% faster
Multi table, binary load only 15% faster!
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

44. Binary Load Load to new or truncated area
proutil <db> -C load <table>.bd [build]
Load to new or truncated area
Truncated rather than “emptied”
Parallel load to different tables
Same or different areas without scatter! When using Type II Areas
Optionally load with build indexes
Somewhat better performance
Faster dump times I have heard are 0 to 10X difference
Multi user dump -Bp
Truncate rather than emptied because of rm free chain layout
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

45. From a threaded dump or dumpspecified…
Binary Load
From a threaded dump or dumpspecified…
proutil <db> -C load <table>.bd
-dumplist <filename>
Dump List File:
/usr1/db/mytable.bd
/usr1/db/mytable2.bd
/usr1/db/mytable3.bd
Must load ALL dumps (.db, db2, .db3, …)
Faster dump times I have heard are 0 to 10X difference
Multi user dump -Bp
Truncate rather than emptied because of rm free chain layout
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

46. Tuning the Process Tune for high (non-recoverable) activity Dump with
–RO, high –B and/or -Bp
Dump on index with fewest # blocks (if possible)
Load with
High –B, –r** or –i**
BIW, 1.5 APW’s per CPU,
Very Large BI clusters with 16K BI blocks
No AI/2PC
Spread data, BI and temp files across disks / controllers
Building indexes where appropriate adds parallelism to the index rebuild and avoids additional scanning of the data!
** only use -r & -i when complete data recovery possible
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

47. Build Indexes (where applicable)
After the Load
Build Indexes (where applicable)
proutil <db> -C idxbuild [ all |table <table> | area <area> schema <owner> | activeindexes | inactiveindexes] [-thread n] [-threadnum n] [-T <dir>] [-TM n] [–TB <blocksize>] [-B n] [-SG n] [-SS <file>] [-pfactor n]
Many new idxbuild choices
Helpful parameters
-SG 64 (sort groups)
-SS filename (file containing sort file list)
-TM 32 (merge buffers)
-TB 31 (temp block size)
-B 1000
Run tabanalys
validate # records
Backup your database
Index build choices are active, inactive, by table, by area etc.
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

48. For areas that were emptied…
Reclaim space
For areas that were emptied…
proutil <db> -C truncate area <area-name>
Warns then deletes data
proutil <db> -C truncate area
Only truncates “empty” areas (but all of them)
Area logically truncated (option #2)
Extents can be deleted
prostrct remove <db> d <old-area-name>
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

49. Don’t Forget.... After the Load Think your done…
Run UPDATE STATISTICS for SQL/ODBC
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

50. Agenda General Migration Strategy The Fast and Easy Upgrade
Physical Upgrade
Tuning Opportunity
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

51. Tuning Opportunity
-Bt (temp tables are Type II storage areas) – client parameter
10.1B+ changes default Temp table block size
From 1K to 4K
–tmpbsize 1 restores old behavior – client parameter
Monitor BI Cluster Size
BI notes are bigger in OpenEdge 10
BI grow
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

52. In Summary Conversion is quick Rollout can be simple
Physical Upgrade at your leisure
Lots of physical re-org options
Rollout can be simple
10,000+ customers on OpenEdge
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

53. Relevant Exchange Sessions
OPS-1: DBA How Healthy is Your Database Today?
OPS-8: Alerts, Alarms, Pages and Harbingers of Trouble…
OPS-14: Effective OpenEdge Database Configuration
OPS-18: Data Management and Platforms Roadmap
OPS-19: What is IPv6 and Why Should I Care
OPS-20: Data Management and Platforms Info Exchange
OPS-23: OpenEdge Performance Basics
OPS-28: A New Spin on Some Old Latches
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

54. ?
Questions
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

55. Thank You
OPS-10: Moving V8/V9 RDBMS to OpenEdge 10

56. OPS-10: Moving V8/V9 RDBMS to OpenEdge 10