1. DBA’s Guide to Physical Dataguard Part II

2. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 2 / 43 Overview

3. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 3 / 43 DB PRIMARY DB Standby Database DB STANDBY DB apply Archive Logs a1a1 a1a1 a2a2 a2a2 TRANSPORT scp, ftp, nfs Archive Logs arc0 a1a1 a1a1 a2a2 a2a2 a3a3 a3a3 Redo logs r1r1 r1r1 lgwr r4r4 r4r4 r3r3 r3r3 r2r2 r2r2

4. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 4 / 43 Dataguard::Overview

5. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 5 / 43 VMware::Overview

6. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 6 / 43 VMware::Network Setup # Windows Host Entries 127.0.0.1 localhost ::1 localhost # Linux VM Host (Guest) Entries / Listener fails to start without this entry 127.0.0.1 localhost.localdomain localhost # Dataguard VMware Configuration on all hosts 192.168.196.1 meme.ahgvm.me 192.168.196.2 tintintintin.ahgvm.me 192.168.196.3 haddockhaddock.ahgvm.me 10.0.0.1 me-prime-pri.ahgvm.me 10.0.0.2 tintin-pritintin-pri.ahgvm.mesnowy-a snowy-a.ahgvm snowy-a.ahgvm.me 10.0.0.3 haddock-pri haddock-pri.ahgvm.mesnowy-b snowy-b.ahgvm snowy-b.ahgvm.me

7. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 7 / 43 Dataguard::Transports MAX PERFMAX AVAILMAX PROT REDO ARCH LGWR SYNC ASYNC SYNC WRITE AFFIRM NOAFFIRM AFFIRM STBY REDO NOYES

8. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 8 / 43 Dataguard::Transport Parameters

9. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 9 / 43 DB STANDBY DB Dataguard::LGWR Transport DB PRIMARY DB Archive Logs ARCn a1a1 a1a1 a2a2 a2a2 a3a3 a3a3 Redo logs r1r1 r1r1 LGWR r3r3 r3r3 r2r2 r2r2 Archive Logs ARCn a1a1 a1a1 a2a2 a2a2 a3a3 a3a3 Standby Redo logs r1r1 r1r1 RFS r3r3 r3r3 r2r2 r2r2 LNSnMRP

10. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 10 / 43 FSFO Architecture DB PRIMARY Redo logs LGWR LNSn DB STANDBY MRP Real-Time Apply Standby Redo logs RFS OBSERVER DMON Broker Cfg. Broker Cfg. spfile

11. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 11 / 43 FSFO Requirements

12. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 12 / 43 DGMGRL listener service

13. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 13 / 43 DGMGRL listener service lsnr_snowy_dg_001 = (description = (address_list= (address=(protocol=tcp) (host=snowy-a.ahgvm.me) (port=5701)) ) sid_list_lsnr_snowy_dg_001 = (sid_list= (sid_desc=(sdu=32767) (sid_name = snowy) (oracle_home = /orah/oracle/10204) ) sid_list_lsnr_snowy_dg_001 = (sid_list= (sid_desc=(sdu=32767) (sid_name = snowy) (oracle_home = /orah/oracle/10204) ) (sid_desc=(sdu=32767) (sid_name = snowy) (oracle_home = /orah/oracle/10204) (global_dbname = snowy_a_DGMGRL.ahgvm) ) lsnr_snowy_dg_001 = (description = (address_list= (address=(protocol=tcp) (host=snowy-b.ahgvm.me) (port=5703)) ) sid_list_lsnr_snowy_dg_001 = (sid_list= (sid_desc=(sdu=32767) (sid_name = snowy) (oracle_home = /orah/oracle/10204) ) sid_list_lsnr_snowy_dg_001 = (sid_list= (sid_desc=(sdu=32767) (sid_name = snowy) (oracle_home = /orah/oracle/10204) ) (sid_desc=(sdu=32767) (sid_name = snowy) (oracle_home = /orah/oracle/10204) (global_dbname = snowy_b_DGMGRL.ahgvm) )

14. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 14 / 43 DGMGRL listener registration db_unique_name=snowy_a local_listener='(address_list= (address= (protocol=tcp) (host=snowy-a.ahgvm.me) (port=5701) ) (address= (protocol=tcp) (host=snowy-a.ahgvm.me) (port=5702) ) )' db_unique_name=snowy_b local_listener='(address_list= (address= (protocol=tcp) (host=snowy-b.ahgvm.me) (port=5703) ) (address= (protocol=tcp) (host=snowy-b.ahgvm.me) (port=5704) ) )‘

15. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 15 / 43 Application listener FQDN lsnr_snowy_general = (address= (protocol=tcp) (host=snowy-cname.ahgvm.me) (port=1526) ) lsnr_snowy_general = (address= (protocol=tcp) (host=snowy-a.ahgvm.me) (port=1526) ) lsnr_snowy_general = (address= (protocol=tcp) (host=snowy-cname.ahgvm.me) (port=1526) ) lsnr_snowy_general = (address= (protocol=tcp) (host=snowy-b.ahgvm.me) (port=1526) )

16. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 16 / 43 DGMGRL TNS uses service snowy_a.ahgvm = (DESCRIPTION = (SDU = 32767) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP) (HOST = snowy-a.ahgvm.me) (PORT = 5701) ) (ADDRESS = (PROTOCOL = TCP) (HOST = snowy-a.ahgvm.me) (PORT = 5702) ) (CONNECT_DATA = (SERVICE_NAME = snowy_a_DGMGRL.ahgvm) (SERVER = DEDICATED) ) snowy_b.ahgvm = (DESCRIPTION = (SDU = 32767) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP) (HOST = snowy-b.ahgvm.me) (PORT = 5703) ) (ADDRESS = (PROTOCOL = TCP) (HOST = snowy-b.ahgvm.me) (PORT = 5704) ) (CONNECT_DATA = (SERVICE_NAME = snowy_b_DGMGRL.ahgvm) (SERVER = DEDICATED) )

17. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 17 / 43 Application TNS snowy.ahgvm = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP) (HOST = snowy-cname.ahgvm.me) (PORT = 1526) ) (CONNECT_DATA = (SERVICE_NAME = snowy) (SERVER = DEDICATED) ) snowy.ahgvm = (DESCRIPTION = (FAILOVER = ON)(LOAD_BALANCE=OFF) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP) (HOST = snowy-a.ahgvm.me) (PORT = 1526) ) (ADDRESS = (PROTOCOL = TCP) (HOST = snowy-b.ahgvm.me) (PORT = 1526) ) (CONNECT_DATA = (SERVICE_NAME = snowy) (SERVER = DEDICATED) ) UNTESTED

18. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 18 / 43 Standby Redo Logs

19. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 19 / 43 Standby Redo Logs SQL> select type as "Types", count(distinct group#) as "Groups" from v$logfile group by type; Types Groups ------- ---------- ONLINE 3 SQL> select a.group#, a.type, b.status, a.member, b.bytes/power(2,20) from v$logfile a, v$log b where a.group# = b.group# ; Group Type Status Log File Size/MB ----- ---------- ---------- ------------------------------ ------- 1 ONLINE CURRENT /dbh/snowy/redo1/redo1a.log 50 1 ONLINE CURRENT /dbh/snowy/redo2/redo1b.log 50 2 ONLINE INACTIVE /dbh/snowy/redo2/redo2a.log 50 2 ONLINE INACTIVE /dbh/snowy/redo3/redo2b.log 50 3 ONLINE INACTIVE /dbh/snowy/redo3/redo3a.log 50 3 ONLINE INACTIVE /dbh/snowy/redo1/redo3b.log 50

20. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 20 / 43 Standby Redo Logs alter database add standby logfile group 4 '/dbh/snowy/redo1/red04a.log' size 50M; alter database add standby logfile group 5 '/dbh/snowy/redo1/red05a.log' size 50M; alter database add standby logfile group 6 '/dbh/snowy/redo1/red06a.log' size 50M; alter database add standby logfile group 7 '/dbh/snowy/redo1/red07a.log' size 50M; alter database add standby logfile group 8 '/dbh/snowy/redo1/red08a.log' size 50M; alter database add standby logfile group 9 '/dbh/snowy/redo1/red09a.log' size 50M;

21. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 21 / 43 Flashback Database

22. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 22 / 43 Prot. Level: MAX AVAILABILITY db_unique_name=snowy_a log_archive_dest_2 = 'db_unique_name=snowy_b SERVICE=snowy_b valid_for=(online_logfile, primary_role) REOPEN=60 OPTIONAL LGWR SYNC AFFIRM' db_unique_name=snowy_b log_archive_dest_2 = 'db_unique_name=snowy_a SERVICE=snowy_a valid_for=(online_logfile, primary_role) REOPEN=60 OPTIONAL LGWR SYNC AFFIRM'

23. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 23 / 43 Prot. Level: MAX AVAILABILITY

24. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 24 / 43 Real Time Apply

25. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 25 / 43 Broker Configuration

26. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 26 / 43 Conversion to spfile SQL> create spfile=‘/orah/oracle/admin/snowy/pfile/spfilesnowy.ora’ from pfile; SQL> SHUTDOWN IMMEDIATE; $ cd $ORACLE_HOME/dbs $ mv initsnowy.ora initsnowy.pfile $ echo “spfile=/orah/oracle/admin/snowy/pfile/spfilesnowy.ora” > initsnowy.ora SQL> /* STARTUP OPEN on PRIMARY, STARTUP MOUNT on STANDBY */ SQL> alter system set dg_broker_start= true scope=both;

27. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 27 / 43 Create DGMGRL Configuration $ORACLE_HOME/bin/dgmgrl / DGMGRL for Linux: Version 10.2.0.4.0 – Production Copyright (c) 2000, 2005, Oracle. All rights reserved. Welcome to DGMGRL, type "help" for information. Connected. DGMGRL> create configuration 'FSFOsnowy' as > primary database is 'snowy_a' > connect identifier is snowy_a.ahgvm; Configuration "FSFOsnowy" created with primary database "snowy_a" DGMGRL> add database 'snowy_b' as > connect identifier is snowy_b.ahgvm > maintained as physical; Database "snowy_b" added

28. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 28 / 43 Show DGMGRL Configuration DGMGRL> show configuration Configuration Name: FSFOsnowy Enabled: NO Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: snowy_a - Primary database snowy_b - Physical standby database Current status for "FSFOsnowy": DISABLED

29. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 29 / 43 Show snowy_a Configuration I DGMGRL> show database verbose snowy_a Database Name: snowy_a Role: PRIMARY Enabled: NO Intended State: OFFLINE Instance(s): snowy Properties: InitialConnectIdentifier = 'snowy_a.ahgvm' ObserverConnectIdentifier = '' LogXptMode = 'ASYNC' Dependency = '' DelayMins = '0' Binding = 'OPTIONAL' MaxFailure = '0' MaxConnections = '1' ReopenSecs = '300' NetTimeout = '180' LogShipping = 'ON' PreferredApplyInstance = '' ApplyInstanceTimeout = '0' ApplyParallel = 'AUTO' StandbyFileManagement = 'auto' ArchiveLagTarget = '0’ Continues on Next Slide

30. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 30 / 43 Show snowy_a Configuration II ArchiveLagTarget = '0' LogArchiveMaxProcesses = '2' LogArchiveMinSucceedDest = '1' DbFileNameConvert = '' LogFileNameConvert = ', ' FastStartFailoverTarget = '' StatusReport = '(monitor)' InconsistentProperties = '(monitor)' InconsistentLogXptProps = '(monitor)' SendQEntries = '(monitor)' LogXptStatus = '(monitor)' RecvQEntries = '(monitor)' HostName = 'tintin.ahgvm.me' SidName = 'snowy' LocalListenerAddress = '(address=(protocol=tcp)(host=snowy-a.ahgvm.me)(port=5701))' StandbyArchiveLocation = 'dgsby_snowy_a' AlternateLocation = '' LogArchiveTrace = '0' LogArchiveFormat = 'snowy-%t_%s_%r.arc' LatestLog = '(monitor)' TopWaitEvents = '(monitor)' Current status for "snowy_a": DISABLED Continued from Last Slide

31. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 31 / 43 Edit snowy_a Properties DGMGRL> edit database snowy_a set property 'LogXptMode'='SYNC'; Property "LogXptMode" updated DGMGRL> edit database snowy_a set property 'NetTimeout'='10'; Property "NetTimeout" updated DGMGRL> edit database snowy_a set property 'StandbyArchiveLocation'='/dbh/snowy/arch'; Property "StandbyArchiveLocation" updated DGMGRL> show database verbose snowy_a Database Name: snowy_a Role: PRIMARY Enabled: NO Intended State: OFFLINE Instance(s): snowy Properties:.... LogXptMode = 'SYNC‘.... NetTimeout = '10‘.... StandbyArchiveLocation = '/dbh/snowy/arch‘.... Current status for "snowy_a": DISABLED

32. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 32 / 43 Edit snowy_b Properties DGMGRL> edit database snowy_b set property 'LogXptMode'='SYNC'; Property "LogXptMode" updated DGMGRL> edit database snowy_b set property 'NetTimeout'='10'; Property "NetTimeout" updated DGMGRL> show database verbose snowy_b Database Name: snowy_b Role: PHYSICAL STANDBY Enabled: NO Intended State: OFFLINE Instance(s): snowy Properties:.... LogXptMode = 'SYNC'.... NetTimeout = '10‘.... StandbyArchiveLocation = '/dbh/snowy/arch/'.... Current status for "snowy_b": DISABLED

33. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 33 / 43 Enable Configuration DGMGRL> enable configuration Enabled. DGMGRL> show configuration Configuration Name: FSFOsnowy Enabled: YES Protection Mode: MaxAvailability Fast-Start Failover: DISABLED Databases: snowy_a - Primary database snowy_b - Physical standby database Current status for "FSFOsnowy": SUCCESS Current status for "FSFOsnowy": Warning: ORA-16610: command 'Broker automatic health check' in progress Current status for "FSFOsnowy": Warning: ORA-16610: command 'ENABLE DATABASE snowy_b' in progress

34. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 34 / 43 Enable Fast Start Failover DGMGRL> enable fast_start failover; Enabled. DGMGRL> show configuration Configuration Name: FSFOsnowy Enabled: YES Protection Mode: MaxAvailability Fast-Start Failover: ENABLED Databases: snowy_a - Primary database snowy_b - Physical standby database - Fast-Start Failover target Current status for "FSFOsnowy": SUCCESS

35. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 35 / 43 Check Fast Start Failover Fast Start Failover Status: ============================================================ DB_UNIQUE_NAME : snowy_a SWITCHOVER_STATUS : SESSIONS ACTIVE FORCE_LOGGING : YES PROTECTION_MODE : MAXIMUM AVAILABILITY PROTECTION_LEVEL : MAXIMUM AVAILABILITY FLASHBACK_ON : YES FS_FAILOVER_STATUS : SYNCHRONIZED FS_FAILOVER_OBSERVER_HOST : hulk.ilmtech.com FS_FAILOVER_CURRENT_TARGET : snowy_b FS_FAILOVER_THRESHOLD : 30 FS_FAILOVER_OBSERVER_PRESENT : NO Fast Start Failover Status: ============================================================ DB_UNIQUE_NAME : snowy_b SWITCHOVER_STATUS : SESSIONS ACTIVE FORCE_LOGGING : YES PROTECTION_MODE : MAXIMUM AVAILABILITY FLASHBACK_ON : YES FS_FAILOVER_STATUS : SYNCHRONIZED FS_FAILOVER_OBSERVER_HOST : hulk.ilmtech.com FS_FAILOVER_CURRENT_TARGET : snowy_b FS_FAILOVER_THRESHOLD : 30 FS_FAILOVER_OBSERVER_PRESENT : NO

36. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 36 / 43 Observer dgmgrl sys/oracle@snowy DGMGRL for Linux: Version 10.2.0.4.0 - Production Copyright (c) 2000, 2005, Oracle. All rights reserved. Welcome to DGMGRL, type "help" for information. Connected. DGMGRL> start observer Observer started

37. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 37 / 43 Failover 14:44:29.28 Tuesday, May 19, 2009 Initiating fast-start failover to database “snowy_b"... Performing failover NOW, please wait... Failover succeeded, new primary is “snowy_b" 14:45:04.03 Tuesday, May 19, 2009 14:46:40.77 Tuesday, May 19, 2009 Initiating reinstatement for database “snowy_a"... Reinstating database “snowy_a", please wait... Operation requires shutdown of instance “snowy" on database “snowy_a" Shutting down instance “snowy"... ORA-01109: database not open Database dismounted. ORACLE instance shut down. Operation requires startup of instance “snowy" on database “snowy_a" Starting instance “snowy"... ORACLE instance started. Database mounted. Continuing to reinstate database “snowy_a"... Reinstatement of database “snowy_a" succeeded 14:47:51.45 Tuesday, May 19, 2009

38. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 38 / 43 Switchover DGMGRL> switchover to snowy_a Performing switchover NOW, please wait... Operation requires shutdown of instance "snowy" on database "snowy_b" Shutting down instance "snowy"... ORA-01109: database not open Database dismounted. ORACLE instance shut down. Operation requires shutdown of instance "snowy" on database "snowy_a" Shutting down instance "snowy"... ORA-01109: database not open Database dismounted. ORACLE instance shut down. Operation requires startup of instance "snowy" on database "snowy_b" Starting instance "snowy"... ORACLE instance started. Database mounted. Operation requires startup of instance "snowy" on database "snowy_a" Starting instance "snowy"... ORACLE instance started. Database mounted. Switchover succeeded, new primary is "snowy_a"

39. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 39 / 43 Role Trigger : DB_ROLE_CHANGE

40. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 40 / 43 Thank You Ahbaid Gaffoor ahbaid@{amazon.com|a9.com|att.net}

41. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 41 / 43 References

42. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 42 / 43 declare cursor cdb is select FORCE_LOGGING, PROTECTION_MODE, PROTECTION_LEVEL, FLASHBACK_ON, DB_UNIQUE_NAME, SWITCHOVER_STATUS, FS_FAILOVER_STATUS, FS_FAILOVER_CURRENT_TARGET, FS_FAILOVER_THRESHOLD, FS_FAILOVER_OBSERVER_PRESENT, FS_FAILOVER_OBSERVER_HOST from v$database; procedure print(s varchar2) is begin dbms_output.put_line(s); end print; procedure cr is begin print(chr(10)); end cr; procedure line(l number := 60, c varchar2 := '=') is begin print(rpad(c,l,c)); end line; begin for db in cdb loop cr; print('Fast Start Failover Status:'); line; print(rpad('DB_UNIQUE_NAME',31)||': '||db.DB_UNIQUE_NAME); print(rpad('SWITCHOVER_STATUS',31)||': '||db.SWITCHOVER_STATUS); print(rpad('FORCE_LOGGING',31)||': '||db.FORCE_LOGGING); print(rpad('PROTECTION_MODE',31)||': '||db.PROTECTION_MODE); print(rpad('PROTECTION_LEVEL',31)||': '||db.PROTECTION_LEVEL); print(rpad('FLASHBACK_ON',31)||': '||db.FLASHBACK_ON); print(rpad('FS_FAILOVER_STATUS',31)||': '||db.FS_FAILOVER_STATUS); print(rpad('FS_FAILOVER_OBSERVER_HOST',31)||': '||db.FS_FAILOVER_OBSERVER_HOST); print(rpad('FS_FAILOVER_CURRENT_TARGET',31)||': '||db.FS_FAILOVER_CURRENT_TARGET); print(rpad('FS_FAILOVER_THRESHOLD',31)||': '||db.FS_FAILOVER_THRESHOLD); print(rpad('FS_FAILOVER_OBSERVER_PRESENT',31)||': '||db.FS_FAILOVER_OBSERVER_PRESENT); cr; end loop; end; / Appendix: fsfo_check.sql

43. DBA’s Guide to Physical Dataguard Part II, NoCOUG May 21 st 2009 43 / 43 create or replace trigger T_FLIP_CNAME after db_role_change on database declare FLIP_DBCNAME_SCRIPT constant varchar2(100) := '/utilities/flip_dbcname'; DBCNAME_JOB constant varchar2(10) := 'DB_CNAME_FLIP'; dbrole VARCHAR2(30); dbname VARCHAR2(30); e_flip_dbcname_script EXCEPTION; PRAGMA EXCEPTION_INIT (e_flip_dbcname_script, -27369); begin select database_role into dbrole from v$database; if (dbrole = 'PRIMARY') then begin dbms_scheduler.drop_job(ROLE_JOB); exception when others then null; end; dbms_scheduler.create_job( job_name => DBCNAME_JOB, job_type => 'EXECUTABLE', number_of_arguments => 1, job_action => FLIP_DBCNAME_SCRIPT, enabled => false, auto_drop => true ); select instance_name into dbname from v$instance; dbms_scheduler.set_job_argument_value( job_name => DBCNAME_JOB, argument_position => 1, argument_value => dbname); begin dbms_scheduler.run_job(DBCNAME_JOB); exception when e_flip_dbcname_script then raise_application_error(-20001, 'flip failed in ' ||FLIP_DBCNAME_SCRIPT); end; end if; exception when others then raise_application_error(-20002, 'DB_ROLE_CHANGE trigger failed '||SQLERRM); end; Appendix: role_change_trigger.sql