1. DB-20: Highly Parallel Dump & Load
Tom Bascom
VP Technology
White Star Software

2. Agenda… Why Dump and Load? What Methods are Available?
Using a Highly Parallel Approach
Demo!

3. Why Dump and Load? Because it is X months since the last time.
Because the vendor said so.
Because we must in order to upgrade Progress.
Because it will improve performance.
Bad reason – if it isn’t broke, don’t fix it!
Bad reason – why does the vendor say this? What else could you do?
Bad reason – you do NOT need to d&l in order to upgrade. (provide example of CCLP benefits when moving from v8 to v9)
Bad reason – if true then there is something else going on that needs to be fixed so that you can have your weekends back.
These aren’t really reasons – they’re excuses

4. X Why Dump and Load? Because it is X months since the last time.
Because the vendor said so.
Because we must in order to upgrade Progress.
Because it will improve performance.
Bad reason – if it isn’t broke, don’t fix it!
Bad reason – why does the vendor say this? What else could you do?
Bad reason – you do NOT need to d&l in order to upgrade. (provide example of CCLP benefits when moving from v8 to v9)
Bad reason – if true then there is something else going on that needs to be fixed so that you can have your weekends back.
These aren’t really reasons – they’re excuses

5. Good Reasons to D&L Improve sequential data access (reports).
Consolidate fragmented records.
To recover corrupted data.
In order to move to a new platform.
In order to reconfigure a storage management decision.
In order to take advantage of hot new db features!

6. Hot Features That Need D&L
Variable Block Sizes
Storage Areas
Variable Rows Per Block
Data Clusters (Type II Areas)

7. Some Dump & Load Choices
“Classic” Dictionary Dump & Load
Bulk Loader
Binary Dump & Load
Vendor Supplied Utilities
Parallelization
Many Benefits
Very difficult to effectively balance manually.
Automating the process
Describe these in more detail – perhaps have some comparative numbers available. Pros & cons for each method. Talk about mixing methods (taking advantage of d&l to filter junk data for instance…) No single method is “always best”
Replication based d&l is too exotic for many people…
Present hard data (perhaps add a slide or two?) to debunk:
-RO is the only way to go (it is often marginal…)
Single threaded Index Rebuild is always faster than inline
Dictionary load is slow (post v9 it’s nearly as fast as binary…)

8. Classic Dictionary D&L
1st the dump…
Cust
Order-line
order
hist
This is like choosing the right algorithm – 1st choose the algorithm; then optimize the details (-RO, dictionary/binary, etc…)
Single thread of execution, poor utilization of machine resources, very, very slow…
Gets comparatively worse over time as machines become more powerful – more disks, more memory and SMP becomes commonplace.
… and then the load.
cust
Order-line
order

9. Classic Dictionary D&L
This can take DAYS (or even WEEKS!)
This is like choosing the right algorithm – 1st choose the algorithm; then optimize the details (-RO, dictionary/binary, etc…)
Single thread of execution, poor utilization of machine resources, very, very slow…
Gets comparatively worse over time as machines become more powerful – more disks, more memory and SMP becomes commonplace.

10. Multi-threaded D&L 1st the dump… … and then the load. cust order hist
Order-line
Doing multiple simultaneous dumps was the first step – in older releases there was little if any benefit to multiple simultaneous loads. In fact it might be slower to do so. Many large dump & loads used binary load which required a single threaded index rebuild prior to v9. But post v9 multiple simultaneous index rebuilds become feasible…
cust
Order-line
order

11. Multi-threaded D&L Much more reasonable than “classic.”
Generally can be done in a weekend.
But still too long for many situations:
Threads aren’t easily balanced.
Longest thread determines total time.
Timing varies – it is hard to predict.
This is like choosing the right algorithm – 1st choose the algorithm; then optimize the details (-RO, dictionary/binary, etc…)
Single thread of execution, poor utilization of machine resources, very, very slow…
Gets comparatively worse over time as machines become more powerful – more disks, more memory and SMP becomes commonplace.

12. Overlapping, Parallel, Balanced D&L
Both the dump…
cust
cust
order
hist
Order-line
Order-line
order
… and the load – in parallel, multi-threaded and load-balanced.
REVISE TO REFLECT THE POTEMKIN METHOD
Numbers of dumpers & loaders don’t need to match.
Load balancing is difficult.
Best route through may not be obvious – the shortest path is the longest dump plus load time for a single table. Different tables may hold individual dump or load longest times. The table with the longest path may not be the longest of either…
And it requires multiple dry runs. The configuration, timing and ordering all interact – what was fast on one run may not be fast on the next run.

13. Parallel Balanced D&L Usually a big improvement.
Often can be done in hours.
But difficult to configure and manage:
You need an intelligent and dynamic control program.
This is like choosing the right algorithm – 1st choose the algorithm; then optimize the details (-RO, dictionary/binary, etc…)
Single thread of execution, poor utilization of machine resources, very, very slow…
Gets comparatively worse over time as machines become more powerful – more disks, more memory and SMP becomes commonplace.

14. Highly Parallel D&L
Explain mapping to machine resources. Show how multiple machines could participate. Emphasize importance of I/O performance...

15. Demo !!!
Kick off the dump process…

16. Resources SUSE Linux 9 Server 2 x 2ghz (Xeon w/HT) 512 MB RAM
3 internal disks
Various External Disks
5GB Demo Database
85 Tables
250 Indexes
Various table and record sizes
Get a photo of the highly advanced server being used ;-)
Flesh out the details. Particularly regarding the configuration of the disks.
Note that if the load fails the old db is still ready to go – it hasn’t been destroyed.

17. The Storage Area Design
Separate Tables from Indexes.
Use a single Type 1 Area for tiny “miscellaneous” tables that have low activity.
Isolate Small, but very active, tables in Type 2 Areas (or standalone areas if v9).
Organize Large Tables by RPB in Type 2 Areas
Any table over 1 million rows
Any table over 1GB
Isolate very large indexes
Discuss alternative SA designs that were considered -- S, M, L record sizes, “fitting” RPB to record size, balancing SA size
Talk about small but very active tables and their impact.

18. The Storage Area Design
b exch06.b1 f #
d "Schema Area":6,64 .
d "Misc":10,256 exch06_10.d1 f 768
d "SmallTables":12,128;8 exch06_12.d1 f
d "RPB256":14,256;512 exch06_14.d1 f
d "RPB128":16,128;512 exch06_16.d1 f
d "RPB64":18,64;512 exch06_18.d1 f
d "RPB64":18,64;512 exch06_18.d2 f
d "RPB32":20,32;512 exch06_20.d1 f
d "RPB32":20,32;512 exch06_20.d2 f
d "RPB16":22,16;512 exch06_22.d1 f
d "Indexes":99,8 exch06_99.d1 f
Discuss alternative SA designs that were considered -- S, M, L record sizes, “fitting” RPB to record size, balancing SA size
Talk about small but active tables and their impact.
Cluster size for index area – didn’t make any impact but there might be advantages in the future.

19. Highly Parallel Dump & Load
90 threads, 16 concurrent.
Dump threads are the same as load threads.
Easy monitoring and control.
No latency – the load finishes when the dump finishes.
No intermediate disk IO:
More efficient usage of disk resources.
Higher overall throughput.
No index rebuild.
No need for table analysis.
Fewer disk “hot spots” – we don’t have (as many) periods where some disks are idle while others are swamped.

20. The 80/20 Rule 98% of the tables are done very quickly
2% of the tables take most of the time…
… and nearly all of the space
These are the tables that we target with multiple threads!
80/20 is “loosely speaking”… The point is that just a few tables are usually responsible for most of the timeline.

21. Multiple Threads Table t Criteria
──────── ─ ──────────────────────────────────────────────────────────────────
document 1 where src.document.app# <
document 2 where src.document.app# >= and src.document.app# <
document 3 where src.document.app# >= and src.document.app# <
document 4 where src.document.app# >= and src.document.app# <
document 5 where src.document.app# >= and src.document.app# <
document 6 where src.document.app# >= and src.document.app# <
document 7 where src.document.app# >=
LOGS where type = "P" and ( logdate <= 12/31/2003 )
LOGS where type = "P" and ( logdate >= 01/01/2004 and logdate <= 06/30/2004 )
LOGS where type = "P" and ( logdate >= 07/01/2004 and logdate <= 12/31/2004 )
LOGS where type = "P" and ( logdate >= 01/01/2005 and logdate <= 06/30/2005 )
LOGS where type = "P" and ( logdate >= 07/01/2005 and logdate <= 12/31/2005 )
LOGS where type = "P" and ( logdate >= 01/01/2006 )
LOGS where type = "L" and ( logdate <= 08/01/2002 )
LOGS where type = "L" and ( logdate > 08/01/2002 )
LOGS where type = "M"
LOGS where type = "U"
Don’t obsess about getting all of the threads perfectly balanaced – they won’t be. But try not to have a single outlier that takes 12 hours longer than all of the rest…
Stress importance of equality match on leading component in LOGS criteria – and the risk of missing keys 
LOGS example needs “src.LOGS.” to actually work but it doesn’t fit

22. Dump Factors Major factors that contribute to dump time:
Data distribution (scatter factor)
Index Selection
-B, -Bp, -RO
Storage Area Configuration (dedicated areas and type 2 areas can be very, very fast)
Disk Configuration and Throughput
proutil dbname –C tabanalys (when doing a binary d&l)
The table analysis is often the constraint on timelines! But you may need it in order to validate success – especially of a bin dump.

23. Concurrency -- Dumping
This is on my 2 (4) CPU Linux box dumping from a (cough) RAID5 (cough) with 4 disks…
Results from an IBM P570 and an EMC disk array are rather different…

24. Take a Load Off Factors that contribute to load time:
Progress version
Build indexes
-i, -bibufs, -bigrow, -B, -TB, -TM, -SG
Storage Area Configuration
Disk Configuration and Throughput
-T files
-noautoreslist (aka “forward-only”)
The importance of testing
Impact of –i:
With –i 37:39, without 57:17… % longer or 1/3rd faster…

25. Concurrency -- Loading
This is on my 2 (4) CPU Linux box dumping from a (cough) RAID5 (cough) with 4 disks…

26. Buffer-Copy & Raw-Transfer
Very Fast
Eliminates The Middle Man (temp file IO operations)
Provides fine-grained, ABL level of control
Allows on the fly data manipulation
Useful when merging databases
Can use with remote connections to bridge version numbers.
In OE10.1 performance is essentially equal
Cannot use RAW-TRANSFER with -RO
I usually use proutil dbname –C convXY rather than bridging version numbers…
“Failed to get transaction info for RAW-TRANSFER statement. (11424)”
Soooo… how does this compare to binary?
Binary dump & load : 42 minutes plus index rebuild : 26 minutes.
Buffer copy: 37:39.
Twice as fast!

27. The Code start.p dlctl.p dlclear.p dlwrap.p dlx.p
Discuss the scripts and configuration files that make this all possible…
Bounce in and out of the in-process d&l to illustrate points.

28. start.p /* start.p * */ run ./dotp/dlclear.p. pause 1 no-message.
run ./dotp/dlctl.p.
run ./dotp/dlmon.p.
return.

29. dlclear.p /* dlclear.p * */ define new global shared variable
dlstart as character no-undo format "x(20)".
dlstart = "".
for each dlctl exclusive-lock:
assign
dlctl.dumped = 0
dlctl.loaded = 0
dlctl.err_count = 0
dlctl.note = ""
dlctl.xstatus = "" .
end.
return.

30. dlctl.p
/* dlctl.p * */
for each dlctl no-lock where dlctl.active = yes
by dlctl.expected descending:
os-command silent
value(
"mbpro -pf dl.pf -p ./dotp/dlwrap.p -param " + '"' +
string( dlctl.thread ) + "|" +
dlctl.tblname + "|" +
dlctl.criteria +"|" +
dlctl.pname + '"' +
" >> /tmp/dl/error.log 2>&1 ~&" ).
end.
return.
“by descending” starts larger tables first!

31. dlwrap.p /* dlwrap.p * */ define variable thr as character no-undo.
define variable tbl as character no-undo.
define variable cri as character no-undo.
define variable pnm as character no-undo.
thr = entry( 1, session:parameter, "|" ).
tbl = entry( 2, session:parameter, "|" ).
cri = entry( 3, session:parameter, "|" ).
pnm = entry( 4, session:parameter, "|" ).
if pnm = "" then pnm = "dlx".
pnm = "./dotp/" + pnm + ".p".
run value( pnm ) value( thr ) value( tbl ) value( cri ).
return.
We’re passing compile on the fly arguments to dlx.p…

32. dlx.p - part 1 /* {1} = thread number * {2} = table name
* {3} = WHERE clause */
define variable flgname as char no-undo initial /tmp/dl/{2}.{1}.flg".
define variable logname as char no-undo initial "/tmp/dl/{2}.{1}.log".
define stream unloaded.
output stream unloaded to value( "/tmp/dl/{2}.{1}.d" ).
output to value( logname ) unbuffered.
put today " " string( time, "hh:mm:ss" ) " {2} {1}" skip.
disable triggers for dump of src.{2}.
disable triggers for load of dst.{2}.
define query q for src.{2}.
open query q for each src.{2} no-lock {3}.

33. dlx.p - part 2
outer_loop: do for dlctl, dst.{2} while true transaction:
inner_loop: do while true:
get next q no-lock.
if not available src.{2} then leave outer_loop. /* end of table */
create dst.{2}.
buffer-copy src.{2} to dst.{2} no-error.
d = d + 1.
if error-status:num-messages = 0 then
l = l + 1.
else
do:
delete dst.{2}.
e = e + 1.
export stream unloaded src.{2}.
find dl.dlctl exclusive-lock
where dlctl.tblname = "{2}" and dlctl.thread = {1}.
dlctl.err_count = e.
next inner_loop.
end.
Outer loop & inner loop…
GET NEXT could easily be replaced with FIND NEXT if version < 9, RAW-TRANSFER is not so easy 
If there is an error then
EXPORT the error for recovery purposes
Update control info with the error count
Increment the dumped & loaded record counts.

34. dlx.p - part 3 if d modulo 100 = 0 then do:
find dl.dlctl exclusive-lock
where dlctl.tblname = "{2}" and dlctl.thread = {1}.
assign
dlctl.dumped = d
dlctl.loaded = l .
file-info:file-name = flgname.
if file-info:full-pathname = ? then
dlctl.note = "stopped".
leave inner_loop.
end.
end. /* inner_loop */
end. /* outer_loop */
Group creates into transactions of 100
Update the control data
Check for a graceful shutdown request (flag file)

35. dlx.p - part 4 do for dl.dlctl transaction:
find dl.dlctl exclusive-lock
where dlctl.tblname = "{2}" and dlctl.thread = {1}.
assign
dlctl.dumped = d
dlctl.loaded = l
dlctl.err_count = e
dlctl.xstatus = "done"
dlctl.note =
( if dlctl.note <> "stopped" then "complete" else "stopped" ) .
if dlctl.note <> "stopped" then
do:
if d < dlctl.expected then dlctl.note = "short".
if d > dlctl.expected then dlctl.note = "long".
end.
if e > 0 then dlctl.note = dlctl.note + "+errors".
Group creates into transactions of 100
Update the control data
Check for a graceful shutdown request (flag file)

36. Validation Belt & Braces!!! Compare Record Counts
dlcompare.p (for binary d&l)
Check Logs for Known & Unknown Errors
grep –i –f error.list
fail, error, (1124)
Check for Success
grep “ 0 errors” /tmp/dl/*.log | wc –l
grep “ dumped” /tmp/dl/*.log
grep “ loaded” /tmp/dl/*.log
ls –l /tmp/dl/*.d
Handling problems – bin load and other intermediate file techniques provide simple recovery for many failure modes.
Forgetting AI at this point is all too easy…

37. Post Load Monitoring Watch for very rapid growth:
May be a rows-per-block assignment error (i.e. data areas with RPB = 1).
May be a cluster-size error (i.e. index areas with 512 blocks per cluster).
May be an unannounced application change.
It is not unusual for there to be somewhat elevated IO for a short period of time (perhaps a week or two).
Handling problems – bin load and other intermediate file techniques provide simple recovery for many failure modes.
Forgetting AI at this point is all too easy…

38. Tip Sheet!
Don’t rely on announcements to keep users out of your system!
Shut down unnecessary services.
Don’t forget cron jobs!
Disable remote access.
Rename the database, the directory, the server…
Take the server off the network.
Always perform NON-DESTRUCTIVE Dump & Load!
Get a photo of the highly advanced server being used ;-)
Flesh out the details. Particularly regarding the configuration of the disks.
Note that if the load fails the old db is still ready to go – it hasn’t been destroyed.

39. Recap Excellent Reasons to Dump & Load: How To Get There Safely…
Move to a new platform!
Reconfigure a storage configuration!!
Take advantage of a hot new db features!!!
Especially Type 2 Areas!!!!
How To Get There Safely…
… and Quickly!

40. ?
Questions