1. Jim Czuprynski OnX Enterprise Solutions March 7, 2016
Regression Analyzed, Tout de Suite: Leveraging Oracle 12c Database Real Application Testing (RAT) Suite
Jim Czuprynski
OnX Enterprise Solutions
March 7, 2016

2. My Credentials 35+ years of database-centric IT experience
Oracle ACE Director
Oracle DBA since 2001
Oracle 9i, 10g, 11g OCP
100+ articles on databasejournal.com and ioug.org
Regular speaker at Oracle OpenWorld, IOUG COLLABORATE, and OTN ACE Tours
Oracle-centric blog (Generally, It Depends)
Taught core Oracle DBA courses (G/I+RAC, Exadata, Performance Tuning, Data Guard) for Oracle University

3. Who We Are Solution Provider focused on Technology Solutions
Comprehensive Service Offerings
Extensive reach — U.S., Canada, and the U.K.
$700M annual revenue with over $100M in Services
Industry Certifications across a broad selection of best-in-class IT manufacturers and technologies
Significant investment in technical staff and facilities to support our clients’ projects
Solution Provider focused on
Data Center Solutions
Comprehensive Service Offerings
Extensive reach — USA, Canada and Europe
$700M annual revenue with over $100M in Services
Industry Certifications across a broad selection of best-in-class IT manufacturers and technologies
Significant investment in technical staff and facilities to support our clients’ projects

4. Our Agenda Real Application Testing (RAT): Concepts and Terminology
SQL Performance Analyzer (SPA)
Database Capture (DBC) and Database Replay (DBR)
Oracle 12cR1 Enhancements
Workload Intelligence
Workload Scale-Up
Consolidated Database Replay
Leveraging Complementary Features and Use Cases
Practical Performance Tuning Scenarios
Database Migration Scenarios
Q+A

5. Real Application Testing: Concepts and Architecture

6. Regression Happens … Often at the Worst Time
How can I determine if even making a minor change to my database environment will impact application performance?
Different settings for initialization parameters
Reversing a non-default setting can often cause performance deflection
Different database versions or releases
Remember, the Oracle query optimizer is “upgraded” as part of just about every new release
New versions of hardware, storage, or operating system
ASM vs. NFS
Enterprise Systems vs. “home-grown” environments
Linux vs. Solaris vs. HPUX vs. AIX

7. Regression Happens … Often at the Worst Time
How can I stress-test my current or a future database environment beyond the point of no return?
This is also known as testing to destruction
Usually involves generating an overwhelming number of users / sessions / requests for similar data simultaneously
May also require combining unanticipated simultaneous workloads that have yet to occur at the same time
Identifying regression vectors and impacts may be extremely difficult
Complex systems mean there’s often multiple interactions between components that are triggering regression
Often degrades to finger-pointing between IT groups

8. RAT Suite: Poor Acronym. Great Toolset!
SQL Performance Analyzer
Analyzes SQL statements that comprise application workloads or subsets thereof
Detects changes in statement execution plans because of:
Different initialization parameters settings
Different database or optimizer versions
Different hardware, storage, or operating system
Regressing SQL statements can be analyzed for better performance via SQL Tuning Advisor
Aimed primarily at improving query performance, not DML

9. RAT Suite: Database Workload Capture
Captures an entire application workload, including DML
Workloads are captured in the precise order in which they occur so that they can be played back in that same order … or not
Workloads can be filtered based on execution attributes
Database Service, Module, Action, Program, User, Instance
AWR data can be captured for later comparison with replays

10. Database Workload Capture
These shadow workload capture processes record each user session’s activity against the production database with minimal overhead (usually < 2% extra CPU)
Shadow Workload
Capture Processes
SCC
WorkloadCapture Directory
SCP
Application Workloads
DW + DSS
OLTP
BATCH + IDL
PRODUCTION ( )
+DATA
DBMS_WORKLOAD_CAPTURE procedures will start, filter, and stop the capture of actual application workloads into shadow workload files containing binary representation of DML and queries

11. RAT Suite: Database Workload Replay
Replays all or part of a captured application workload
Statements are replayed against test system, including DML
By default, workload is replayed in precisely the same order as it was recorded
Workloads can be selectively ramped up:
Overwhelming number of users / sessions / requests for same data
Unexpected combination of simultaneous yet normal workloads
This enables testing to destruction, i.e. beyond a system’s current capacity to execute the workload

12. Database Workload Replay
TEST / QA ( ) (rewound to before production capture point)
+DATA
WorkloadCapture Directory
WRC
Actually, there are two sets of captured workload files: the workloads themselves as well as the order in which transactions and queries originally occurred
Workload Replay Clients (WRCs) read the captured workload files and play them back in original order (or not, if desired!)
DBMS_WORKLOAD_REPLAY procedures will start, filter, and stop the replay of shadow workload files containing binary versions representing actual application workloads generated

13. Real Application Testing: New Features in 12cR1

14. Consolidated Replay
Permits the combination of different workloads captured from different databases at different times into one consolidated workload. Use cases include:
Same workload, but generated in different 10gR2, 11gR1, and 11gR2 environments
Same workload generated on same database, but during completely different timeframes
Different workloads generated on same database captured during wildly diverse conditions

15. Workload Scale-Up
Enables the combination of workloads for simultaneous execution for potential “test to destruction”
Schema Remapping allows replaying a single workload into multiple schemas simultaneously
Time Shifting permits lining up different workloads so that their peak activity periods occur simultaneously
Workload Folding separates a single workload captured over a longer duration into different sub-workloads, then playing those workloads back simultaneously

16. Workload Scale-Up: Examples
Time Shifting: Combine 07:00 – 14:00 and 17:00 – 23:00 workloads into a single workload during replay
Workload Folding: Cut 07:00 – 14:00 workload into three separate sub-workloads and then replay them simultaneously
“Time-Shifted” Workload
17:00 – 23:00
07:00 – 14:00
“Folded” Workload

17. Workload Intelligence
Workload Intelligence analyzes a captured workload for execution patterns that aren’t obvious to human brains
Java-based program
Requires creation of a separate schema for capturing results of analysis
Returns one or more sets of SQL statements that comprise a workload pattern

18. Real Application Testing: Real-World Scenarios

19. Scenario #1: Upgrade from 11gR2 to 12cR1
Current Database Environment:
Oracle 11gR2 ( ) database on Microsoft Windows
Supports primarily OLTP workload, some DSS, limited contention
New Database Environment:
Oracle 12cR1 ( ) database on Oracle Linux
A large non-partitioned table will now be partitioned
Does the new environment handle current workloads?
Queries against newly-partitioned table must perform 2X – 3X faster
Do any existing queries demonstrate regression?
Will placing tables in IMCS offer any benefits for analytic queries?

20. Scenario #2: Transitioning to Hybrid Cloud
Current Database Environment:
Oracle 11gR2 ( ) database
Supports purely data warehouse / analytic workloads
Database server is “bare metal” Exadata X2-2
New Database Environment:
Oracle 12cR1 ( ) database, same application workloads
Transitioning to Oracle Cloud DBaaS environment (Exadata X5-2)
Does the new environment handle current workloads?
Do any existing queries demonstrate regression?
Is an Oracle Cloud Database Extreme Performance environment required to handle the application workload?

21. Scenario #3: Test to Destruction
Current Database Environment:
Oracle 11gR2 ( ) database
Oracle Enterprise Linux 5.6 virtualized via VirtualBox
Supports both OLTP and DSS workloads with limited contention:
Peak workload for OLTP between 10: :00, 60 user sessions
Peak workload for DSS between 19:00 – 21:00, 80 user sessions
New Database Environment:
Oracle 12cRx (12.2.x.x) database
Oracle Enterprise Linux 7.2 virtualized via VMWare ESX
Must support peak OLTP and DSS workloads simultaneously
50% more memory available
Oracle In-Memory Option licensing warily considered
CIO and CFO must know: Is this new environment sufficient?

22. Capturing a Workload 1 Create appropriate directory objects: 3
DROP DIRECTORY DBRCONTROL;
CREATE DIRECTORY DBRCONTROL AS '/home/oracle/DBRControl';
GRANT READ, WRITE ON DIRECTORY DBRCONTROL TO PUBLIC;
… allow the capture to continue until all desired workloads have been recorded … 3
BEGIN
DBMS_WORKLOAD_CAPTURE.ADD_FILTER(
fname => 'SVC_ADHOC'
,fattribute => 'SERVICE'
,fvalue => 'ADHOC');
. . .
fname => 'SVC_TPCH'
,fvalue => 'TPCH');
DBMS_WORKLOAD_CAPTURE.START_CAPTURE(
name => 'DBR_CAPTURE_100'
,dir => 'DBRCONTROL'
,duration => NULL
,default_action => 'EXCLUDE'
,auto_unrestrict => TRUE
,capture_sts => TRUE
,sts_cap_interval => 5);
END; /
Apply filters to exclude any undesired user sessions, then start the capture process … 2
… and then end the capture process once all workloads are complete: 4
BEGIN
DBMS_WORKLOAD_CAPTURE.FINISH_CAPTURE(
timeout => 90
,reason => 'Termination (after 90 seconds)');
END; /

23. Consolidated Replay: Setup
Create appropriate directory objects and new user account for replay clients: 1
Prepare the Replay Schedule from previously-captured workloads: 2
DROP DIRECTORY DBRCONS;
CREATE DIRECTORY DBRCONS AS '/home/oracle/DBRControl';
GRANT READ, WRITE ON DIRECTORY DBRCONS TO PUBLIC;
DROP DIRECTORY DBCOLTP;
CREATE DIRECTORY DBCOLTP AS '/home/oracle/DBRControl/OLTP';
GRANT READ, WRITE ON DIRECTORY DBCOLTP TO PUBLIC;
DROP DIRECTORY DBCDSS;
CREATE DIRECTORY DBCDSS AS '/home/oracle/DBRControl/DSS';
GRANT READ, WRITE ON DIRECTORY DBCDSS TO PUBLIC;
DROP USER wrc CASCADE;
CREATE USER wrc
IDENTIFIED BY wrc
DEFAULT TABLESPACE users
TEMPORARY TABLESPACE temp
PROFILE DEFAULT
QUOTA UNLIMITED ON users;
GRANT CONNECT, DBA TO wrc;
. . .
cpt_id :=
DBMS_WORKLOAD_REPLAY.ADD_CAPTURE(
capture_dir_name => 'DBCOLTP'
,start_delay_seconds => 0
,stop_replay => FALSE
,take_begin_snapshot => TRUE
,take_end_snapshot => TRUE
,query_only => FALSE );
capture_dir_name => 'DBCDSS'
DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE;
END; /
DECLARE
cpt_id NUMBER;
BEGIN
DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE(
capture_dir => 'DBCOLTP');
capture_dir => 'DBCDSS');
DBMS_WORKLOAD_REPLAY.SET_REPLAY_DIRECTORY(
replay_dir => 'DBRCONS');
DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE (
replay_dir_obj => 'DBRCONS'
,schedule_name => 'TEST2DESTRUCT');
. . .

24. Consolidated Replay: Preparation and Startup
Initialize the consolidated replay and remap connections as necessary: 3
BEGIN
DBMS_WORKLOAD_REPLAY.INITIALIZE_CONSOLIDATED_REPLAY(
replay_name => 'DBR_CONS_100'
,schedule_name => 'TEST2DESTRUCT‘);
DBMS_WORKLOAD_REPLAY.REMAP_CONNECTION(
connection_id => 1
,replay_connection =>
'(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ora12201)(PORT=1521))
(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=DSS)))');
. . .
END; /
BEGIN
DBMS_WORKLOAD_REPLAY.PREPARE_CONSOLIDATED_REPLAY(
synchronization => TRUE
,connect_time_scale => 0
,think_time_scale => 0
,think_time_auto_correct => FALSE
,capture_sts => TRUE
,sts_cap_interval => 5 );
END; /
Prepare the Consolidated Replay: 4
Kick off the minimum necessary Workload Replay Clients: 5
$> wrc wrc/wrc mode=replay replaydir=/home/oracle/DBRControl
$> wrc wrc/wrc mode=replay replaydir=/home/oracle/DBRControl
BEGIN
DBMS_WORKLOAD_REPLAY.SET_REPLAY_TIMEOUT(FALSE);
DBMS_WORKLOAD_REPLAY.START_CONSOLIDATED_REPLAY;
END; /
Start the Consolidated Replay task: 6

25. Consolidated Replay: Monitoring via SQL*Plus
SELECT id
,name
,schedule_name
,capture_id
,user_calls
,status
,divergence_load_status DLS
,TO_CHAR(start_time,'yyyy-mm-dd hh24:mi:ss') start_dtm
,TO_CHAR(end_time,'yyyy-mm-dd hh24:mi:ss') end_dtm
,duration_secs
,num_clients
,num_clients_done
,(dbtime/ ) dbtime_ss
,(think_time/ ) think_time_ss
,(network_time/ ) network_time_ss
,pause_time
FROM dba_workload_replays;
SELECT id
,name
,dbname
,dbversion
,directory
,capture_id
,user_calls
,status
,TO_CHAR(prepare_time,'yyyy-mm-dd hh24:mi:ss') prepare_dtm
,synchronization
,connect_time_scale
,think_time_scale
,think_time_auto_correct TTAC
,scale_up_multiplier SCM
,default_action
FROM dba_workload_replays;
Think
# of Replay Think Time Scale
Rply Replay User Prepare Synch Connect Time Auto Up Default
ID# Operation Calls Status Time ronized? TimeScale Scale Corre Multipler Action
167 DBR_CONS_ COMPLETED SCN FALSE INCLUDE
18:22:04
157 DBR_CONS_ COMPLETED SCN FALSE INCLUDE
16:17:57
113 DBR_CONS_ COMPLETED SCN FALSE INCLUDE
17:36:42
102 DBR_CONS_ COMPLETED SCN FALSE INCLUDE
17:00:50
# of Replay Replay Replay Total Clnt Think Netwk Pause
Rply User Start End Time Clnt Sess DB Time Time Time Time
ID# Calls Status Time Time (s) Sess Done (s) (s) (s) (s)
COMPLETED
18:23: :33:33
COMPLETED
16:18: :33:05
COMPLETED
17:37: :46:35
COMPLETED
17:01: :10:19

26. Consolidated Replay: Monitoring via OEM 13c CC

27. Consolidated Replay: Results
VARIABLE replay_report CLOB;
VARIABLE capture_id NUMBER;
VARIABLE replay_id NUMBER;
BEGIN
:replay_id := 167;
:replay_report :=
DBMS_WORKLOAD_REPLAY.REPORT(
replay_id => :replay_id
,format => 'TEXT');
END; /
SET LONGCHUNKSIZE 1000
SET LONG
SET LINESIZE 2000
SET PAGESIZE 0
PRINT :replay_report
. . .
Replay Schedule Information Schedule TEST2DESTRUCT
contains 2 captures:
| Information | Capture |
| Name | DBR_CAPTURE_ |
| Status | COMPLETED |
| Database Name | ORCL |
| Database Version | |
| Start Time | :27: |
| End Time | :32: |
| Duration | 5 minutes 35 seconds |
| DB Time | seconds |
| User Calls | |
| Average Active Sessions | |
| Directory Object | DBCOLTP |
| DB Name | DB Id | Release | RAC | Replay Name | Replay Status |
| NCDB122 | | | NO | DBR_CONS_100 | COMPLETED |
Replay Information
| Information | Replay |
| Name | DBR_CONS_ |
| Status | COMPLETED |
. . .
| AWR DB Id | |
| AWR Begin Snap Id | |
| AWR End Snap Id | |
| PL/SQL Capture Mode | TOP_LEVEL |
| Replay Schedule Name | TEST2DESTRUCT |

28. Consolidated Replay: AWR Delta Comparisons

29. Thank You

30. Thank You For Your Kind Attention!
If you have any questions or comments, feel free to:
me at
Follow my blog (Generally, It Depends):
Follow me on Twitter
Connect with me on LinkedIn (Jim Czuprynski)