1. Informix Enterprise Replication in 5 easy Steps WAIUG Forum December 8-9, 2006
Introduction
We are going to cover how to set-up basic ER. Your mileage may vary.
Nancy Balsbaugh

2. What are we going to cover?
Overview
Types of Enterprise Replication
Requirements – Linux, UC5
How to set it up
How to verify/check replication
Troubleshooting
Pitfalls and Gotchas
Magic Cookies
Introduction
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

3. Types of Enterprise Replication
Primary – Target (tree)
Simple
No Conflict
Works or Not
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

4. Types of Enterprise Replication cont’d
Update Anywhere
Complex
Conflict resolution
May Partially Work
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

5. Requirements Connectivity Sqlhosts .rhosts /etc/services /etc/hosts
g_hdr group i=1
g_server group i=2
server1 onsoctcp hostname server1 g=g_server1
g_server group i=3
server2 onsoctcp hostname server2 g=g_server2
.rhosts
/etc/services
/etc/hosts
Template File <filename>
Get your database servers working first! Then start replication set-up.
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

6. Requirements Cont’d Onconfig Dbserveralias – used in the sqlhosts file
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

7. How To Set It Up Step 1 - Define the Servers
Step 2 - Define the replicates
Step 3 - Change the state of the replicate (start replicating)
Step 4 - Synchronize the Data (copy pre-replication data)
Step 5 - Verify
5 steps
Actually replicates after 3 steps
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

8. Step 1 - Define the servers
Participants, master, conflict resolution rules and scope, frequency, error logging, full rows or changed columns, triggers, etc.
Primary
cdr define server --connect=server1 --idle=0 --ats=/errorlog/ats --ris=/errorlog/ris --init g_server1
Secondary
cdr define server -c server1 --idle=0 --init --leaf --sync=g_server1 server2
ATS – aborted transaction spooling, RIS – row information spooling, Init - option initializes the server, Idle – idle timeout, Leaf means tree hierarchy – leave that out if you want update anywhere
Define is very important…if not done, it won’t work!
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

9. After the first server is defined cdr list server
SERVER ID STATE STATUS QUEUE CONNECTION CHANGED
g_server Active Local
5 steps
Actually replicates after 3 steps
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

10. After the 2nd server is defined cdr list server
SERVER ID STATE STATUS QUEUE CONNECTION CHANGED
g_server Active Local
g_server Active Connected Nov 14 13:47:09
Active – Connected is how you should see everything.
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

11. Cdr list of tree hierarchy
Listing g_server1
SERVER ID STATE STATUS QUEUE CONNECTION CHANGED
g_server Active Connected Nov 14 13:46:16
g_server Active Connected Nov 14 13:47:29
g_server Active Local
**********
Listing g_server2
g_server Active Local
g_server Active Connected Nov 14 13:46:17
Listing g_server3
g_server Active Local
g_server Active Connected Nov 14 13:47:28
Active – Connected is how you should see everything.
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

12. Update Anywhere cdr list
Listing g_server1
SERVER ID STATE STATUS QUEUE CONNECTION CHANGED
g_server Active Connected Nov 14 13:46:16
g_server Active Connected Nov 14 13:47:29
g_server Active Local
**********
Listing g_server2
g_server Active Local
g_server Active Connected Nov 14 13:46:17
Listing g_server3
g_server Active Local
g_server Active Connected Nov 14 13:47:28
Active – Connected is how you should see everything.
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

13. Step 2 – Option 1 - Define the replicates
cdr define replicate --conflict=timestamp --scope=row --ats --ris --master=g_server1 <replicate_name>
Active – Connected is how you should see everything.
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

14. Step 2 – Option 1 - Define the replicates
cdr list repl
CURRENTLY DEFINED REPLICATES
REPLICATE: r_test1
STATE: Active ON:g_a1
CONFLICT: Always Apply
FREQUENCY: immediate
QUEUE SIZE:
PARTICIPANT: testdb:informix.test1
OPTIONS: transaction,ris,ats,fullrow
REPLID: / 0x10001
REPLMODE: PRIMARY ON:g_a1
APPLY-AS: INFORMIX ON:g_a1
REPLTYPE: Master
REPLICATE: r_test2
PARTICIPANT: testdb:informix.test2
REPLID: / 0x10002
Active – Connected is how you should see everything.
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

15. Step 2 - Option 2 – Define a Template
cdr define template <templatename>
--conflict=timestamp
--scope=row --ats --ris --firetrigger
--master=g_server1
--database=dbname --file=<filename>
The file contains a list of tables to be replicated – easiest way.
Active – Connected is how you should see everything.
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

16. Step 2 - Option 2 – Define a Template
cdr list template
TEMPLATE DATABASE TABLES
==============================================
Template_name dbname informix.table1
dbname informix.table2
dbname informix.table3
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

17. Step 3 - Option 1 – Change the replicate
cdr start replicate <replicate_name> g_server1 g_server2
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

18. Step 3 - Option 2 - Realize the template
cdr realize template <templatename> g_server1
cdr realize template <templatename> g_server2 g_server3
This copies the replicate to the other servers and changes it’s state – “starts it”
Do the primary first!
Then broadcast the rest
Copies the replicates and starts replication.
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

19. Synchronize the data
cdr sync replset -m g_server1 -s <templatename>
--firetrigger g_server2 g_server3
Active – Connected is how you should see everything.
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

20. How to Verify/Check Replication
cdr check replicateset -c server1 -m g_server1 -s <templatename> -v -a
This will check data on all the nodes (-a) against the master server1
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

21. Troubleshooting To Repair Replication
cdr check replicateset -c server1 -m g_server1 -s <templatename> -v –a –R
Cdr list of all the servers (with the connect option)
Cdr err of all the servers (with the connect option)
Check all the servers message logs
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

22. cdr err –a Without Errors
SERVER:SEQNO REVIEW TIME ERROR
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

23. cdr err –a With Errors SERVER:SEQNO REVIEW TIME ERROR
g_server2:164           N   :39:01     31   GC operation participate replicate 'sync_17797_131103_ ' failed: undefined replicate
g_server3:165           N   :39:01     31   GC operation participate replicate 'sync_17797_131103_ ' failed: undefined replicate
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

24. Online Log when replication is being set-up
12:10:39 Building 'syscdr' database ...
12:10:40 'syscdr' database built successfully.
12:10:42 CDR queuer initialization complete
12:10:42 CDR NIF listening on asf://g_server1
12:13:10 Checkpoint Completed: duration was 1 seconds.
12:13:10 Checkpoint loguniq 298, logpos 0x132a7018, timestamp: 0xf5f9165 .
12:20:22 ER checkpoint started
12:20:25 DDR Log Snooping - Snooping started in log 299
12:25:41 Checkpoint Completed: duration was 0 seconds.
12:25:41 Checkpoint loguniq 299, logpos 0x3bb9018, timestamp: 0xf63be8a
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

25. Pitfalls and Gotchas
cdr err will mark errors and you’ll need to make sure you use cdr err –a to see all errors that haven’t been zapped.
If Update-Anywhere, then use the –c option to connect to the other servers and look from their perspective.
Use the –c (connect) option to make control codes come from one central server.
Define can only be done serially – same with all the commands. Do not run them in parallel. To speed things up – Use broadcast mode.
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

26. Pitfalls and Gotchas cont’d
You must list all servers for cdr list otherwise 2 servers can have problems between them that doesn’t affect the primary. If you are looking only at the primary, you won’t see it.
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

27. Magic Cookies Check the return code Check the output of cdr err –a
Use cdr err --zap to clear out errors
Always check the list of all the servers and the errors on all the servers.
Use cdr finderr to lookup errors
Scripts Included! Very Basic! Need improvement. Mostly so you can see the cdr commands. Feel free to improve upon them and send them back to me.
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

28. cdr finderr 12:11:10 CDR GC synchronization error 35 finderr 35
Operation would block.
An operating-system error code with the meaning shown was unexpectedly returned to the database server. If the error recurs, note all circumstances and contact Technical Support at
WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

29. cdr finderr cont’d cdr finderr 35 35 server not defined in sqlhosts
Sqlhosts on leaf
g_hdr group i=1
g_server1 group i=2
server onsoctcp SERV1 server g=g_server1
g_server2 group i=3
server onsoctcp SERV2 server g=g_server2

30. Q & A Any Questions? Source: Placeholder for Notes is 14 points
Introduction
We are going to
Source: Placeholder for Notes is 14 points

31. Thank You
Nancy Balsbaugh

32. To list all the replication servers (cdr list)
#!/bin/ksh
echo "Building custom scripts for this Cluster"
echo "#!/bin/ksh" > cdrlist.new
chmod 700 cdrlist.new
echo "echo \"***********************\"" >> cdrlist.new
echo "date" >> cdrlist.new
echo "echo \"Starting Listing Servers\"" >> cdrlist.new
for i in `cat pub` do
echo "Adding in the publisher $i"
echo "echo \"Listing g_$i\"" >> cdrlist.new
echo "cdr list server -c $i" >> cdrlist.new
echo "echo \"Return value:\$?\"" >> cdrlist.new
echo "echo \"\"" >> cdrlist.new
echo "echo \"**********\"" >> cdrlist.new
done
export pub=$i
echo "pub = $pub"
for i in `cat servers`
echo "Adding in the subscriber $i"

33. To see errors (cdr err) echo "#!/bin/ksh" > cdrerr.new
chmod 700 cdrerr.new
echo "echo \"***********************\"" >> cdrerr.new
echo "date" >> cdrerr.new
echo "echo \"Starting Listing Servers\"" >> cdrerr.new
for i in `cat pub` do
echo "Adding in the publisher $i"
echo "echo \"Showing Errors from g_$i\"" >> cdrerr.new
echo "cdr err -a -c $i" >> cdrerr.new
echo "echo \"Return value:\$?\"" >> cdrerr.new
echo "echo \"\"" >> cdrerr.new
echo "echo \"**********\"" >> cdrerr.new
done
export pub=$i
echo "pub = $pub"
for i in `cat servers`
echo "Adding in the subscriber $i"

34. To clear errors in cdr err
echo "#!/bin/ksh" > cdrzap.new
chmod 700 cdrzap.new
echo "echo \"***********************\"" >> cdrzap.new
echo "date" >> cdrzap.new
echo "echo \"Starting Listing Servers\"" >> cdrzap.new
for i in `cat pub` do
echo "Adding in the publisher $i"
echo "echo \"Showing Errors from g_$i\"" >> cdrzap.new
echo "cdr err -z -c $i" >> cdrzap.new
echo "echo \"Return value:\$?\"" >> cdrzap.new
echo "echo \"\"" >> cdrzap.new
echo "echo \"**********\"" >> cdrzap.new
done
export pub=$i
echo "pub = $pub"
for i in `cat servers`
echo "Adding in the subscriber $i"

35. To Run Cdr Check echo "#!/bin/ksh" > cdrcheck.new
chmod 700 cdrcheck.new
echo "date" >> cdrcheck.new
echo "echo \"Running cdr check for status of Cluster\"" >> cdrcheck.new
echo "time cdr check replicateset -c $pub -m g_$pub -s <templatename> -v -a >> /home/informix/cdr_check_output_all_\`date +%Y%m%d%H%M\` 2>&1" >> cdrcheck.new
echo "echo \"\"" >> cdrcheck.new
echo "echo \"Showing Errors from g_$i\"" >> cdrcheck.new
for i in `cat servers` do
echo "Adding in the subscriber $i"
echo "$i" >> cdrcheck.new
done
echo "-v -a >> /home/informix/cdr_check_output_all_\`date +%Y%m%d%H%M\` 2>&1" >> cdrcheck.new

36. To Delete a server echo "#!/bin/ksh" > deleteserv.new
chmod 700 deleteserv.new
echo "date" >> deleteserv.new
for i in `cat servers` do
echo "Adding in $i"
echo "echo \"Deleting g_$i\"" >> deleteserv.new
echo "cdr delete server g_$i --connect=$i" >> deleteserv.new
echo "echo \"Return value:\$?\"" >> deleteserv.new
done
for i in `cat pub`
echo "Adding in the publisher $i"
echo "cdr delete server g_$i" >> deleteserv.new

37. To Define a server echo "#!/bin/ksh" > defserv.new
chmod 700 defserv.new
echo "date" >> defserv.new
echo "echo \"Starting Define Servers\"" >> defserv.new
for i in `cat pub` do
echo "Adding in the publisher $i"
echo "echo \"Defining g_$i\"" >> defserv.new
echo "cdr define server --connect=$i --idle=0 --init g_$i" >> defserv.new
echo "echo \"Return value:\$?\"" >> defserv.new
done
export pub=$i
echo "pub = $pub"
for i in `cat servers`
echo "Adding in the subscriber $i"
echo "cdr define server -c $i --idle=0 --init --leaf --sync=g_$pub g_$i" >> defserv.new

38. To Define a template echo "#!/bin/ksh" > deftemp.new
chmod 700 deftemp.new
echo "date" >> deftemp.new
echo "echo \"Starting Define Template\"" >> deftemp.new
export pub=`cat pub`
echo "pub = $pub"
export database=`cat database`
echo "database = $database"
echo "cdr define template <templatename> --conflict=always --scope=row --master=g_$pub --database=$database --file=<filename>" >> deftemp.new
echo "echo \"Return value:\$?\"" >> deftemp.new

39. To Realize the template (copy the template to the other servers and turn on replication)
echo "#!/bin/ksh" > realizetemp.new
chmod 700 realizetemp.new
echo "date" >> realizetemp.new
echo "echo \"Starting Realize Template\"" >> realizetemp.new
for i in `cat pub` do
echo "Adding in the publisher $i"
echo "echo \"Realizing template on g_$i\"" >> realizetemp.new
echo "cdr realize template <templatename> g_$i" >> realizetemp.new
echo "echo \"Return value:\$?\"" >> realizetemp.new
done
export pub=$i
echo "pub is $pub"
echo "Adding in the subscribers"
echo "echo \"Realizing template on subscribers\"" >> realizetemp.new
echo "cdr realize template <templatename>" >> realizetemp.new
for i in `cat servers`
echo " g_$i" >> realizetemp.new
echo "YOU NEED TO MODIFY REALIZETEMP"

40. To Synchronize the servers
echo "#!/bin/ksh" > syncsubs.new
chmod 700 syncsubs.new
echo "date" >> syncsubs.new
echo "echo \"Starting to Sync\"" >> syncsubs.new
for i in `cat pub` do
echo "cdr sync replset -m g_$i -s <templatename> --firetrigger" >> syncsubs.new
for i in `cat servers`
echo g_$i >> syncsubs.new
done
echo "echo \"Return value:\$?\"" >> syncsubs.new
Done
echo "YOU NEED TO MODIFY SYNCSUBS"