1. Deadly Sins against Database Performance
Vladimir Andreev
Semantec GmbH

2. About the Author 7 years of experience with Oracle
Specialization in Performance Tuning
Performance Assessments of many production databases
Different industries (automotive, pharmaceutical, chip production, public sector, banking, machine building, etc.)
Different workloads (OLTP, DW, Web)
Different sizes (100MB – 500GB)
Different architectures (2-tier, 3-tier)
Different application technologies (PL/SQL, C++, Java, Forms, ABAP, Teamcenter, Vignette, …)
I have reviewed the performance of many production databases in the past years. During these reviews, I often come across design and development patterns that not only hurt, but in some cases literally kill the performance of a database.
Since many of these bad practices -- un-shareable SQL and single-row processing, for example -- have been widely discussed in the specialised literature and forums, I was surprised to find "live specimens" out in the field. This presentation shows these mistakes, how to detect them in a production system, how to fix them, and - most importantly - how to avoid making them in the first place.

3. Top Problems
Architecture: database “independence” through isolation layers
Sub-optimal performance in any database
Higher development and maintenance costs
Design: row-by-row processing
Hard limits on performance and scalability
Coding: Un-shareable SQL
Disaster waiting to happen

4. Middle Layers Application Servers SAP, BAAN Custom Design
Database Isolation Layers
DAO/OLEDB, ODBC, JDBC, etc.
J2EE Persistency Layers (e.g. TopLink)
Custom Design Persistency layers

5. Database Isolation Layers
Are cast in stone
Export an API with limited or no support for:
Non-standard SQL (with, merge, connect by, analytics, hints…)
stored procedure calls
“ref cursor”, object types, etc.
bind variables
Insist on maintaining the data model, incl. indexes
Generate sub-optimal SQL

6. Live Specimens “We have no control over the SQL”
“The Application Server is not supported with CBO”
“We need to trick the middle layer”
Problem: A customer customises extensively a packaged application. The application exports a limited data access API for customisations. The API does not support stored procedure calls, hierarchical queries, analytics, aggregations, joins, or other ‘advanced’ Oracle features. The customer needs to select a branch of a hierarchy stored in a table.
Solution 1: The customer creates a view that uses CONNECT BY to traverse the hierarchy. Problem: How to pass the starting node and “connect by” parameters to the view?
Solution 1a: Insert the starting node’s ID into another table (don’t commit), run the view that uses a function in its “start with” and “connect by” clauses to retrieve the “parameter”, rollback.
Results: ‘select parameter from parameter_table’ executed up to 6000 times per second; each of these taking up to 30 logical reads to scan the “empty” parameter_table (consistent gets…); in peak times, these calls are responsible for over 90% of the instance’s logical reads and CPU usage; several times the CPU utilisation hits 100% and the response times become chaotic (from sub-second to several minutes).

7. Row-by-Row Processing
Seems “easy to code and understand”, but
Has many negative consequences:
Round-trip overhead
Context switching
Increased server load
Increased network load
That’s why it is sometimes called “Slow-by-Slow processing”

8. Slow-by-Slow Processing
Over the network
Increased traffic
Multiplies the network overhead
On the same machine (via IPC)
Voluntary context switches
In the same database
“Context switches” between the PL/SQL and SQL machine
Single-row processing causes extremely high amounts of voluntary context switches with all the associated OS overhead. Under heavy user load (many concurrent users doing the same kind of single-row processing), the overhead can become so significant that it can consume the majority of the available CPU power.
In the more common configuration, where the client and the server are located on different machines and communicate not via IPC (Inter-Process Communication), but via a network, the overhead caused by context switches is added to the network overhead, degrading the overall performance to unacceptable levels even faster, i.e., limiting the system scalability even further. There is no cure or “workaround” for this design flaw other than changing the design itself. Adding more CPU power might provide temporary and marginal improvements, but this system will not scale until this design flaw is in place.
Counter-driven data access loops should be replaced with cursor-driven loops, preferably using array binds. When possible, joins, aggregates, filters, sorts, etc. should be performed in the database, and not in the middle layer or the client.

9. Live Specimen Create or replace view VM01_WORKFLOW_STEPS_ACT as
SELECT T.ID, T.USER_ID, T.WORKFLOW_ID, T.CONTENT_ID
FROM TM01_WORKFLOW_STEPS T
WHERE NUMMER = FM01_WORKFLOW_STEPS_ACT(T.CONTENT_ID);
Create or replace
FUNCTION "FM01_WORKFLOW_STEPS_ACT" (vCONTENT_ID IN NUMBER)RETURN NUMBER IS
vNr NUMBER;
BEGIN
SELECT MAX(NUMMER) INTO vNr FROM TM01_WORKFLOW_STEPS WHERE CONTENT_ID=vCONTENT_ID;
RETURN vNr;
EXCEPTION
WHEN OTHERS
THEN RETURN NULL;
END FM01_WORKFLOW_STEPS_ACT;

10. Better
Create or replace view VM01_WORKFLOW_STEPS_ACT as
SELECT T.ID, T.USER_ID, T.WORKFLOW_ID, T.CONTENT_ID
FROM TM01_WORKFLOW_STEPS T
WHERE NUMMER = (select max(nummer) from TM01_WORKFLOW_STEPS where CONTENT_ID=T.CONTENT_ID);
Or, even better:
select * from (
SELECT T.ID, T.USER_ID, T.WORKFLOW_ID, T.CONTENT_ID,
max(nummer) over (partition by CONTENT_ID) mNum
FROM TM01_WORKFLOW_STEPS T )
where nummer=mNum;

11. Effects on a 6-CPU System
CPU count: 6
Architecture: Application server processes running on the same machine (IPC)
A batch job was started around 14:40 on to produce a typical load for the benefit of analysis. This job created two database sessions, the more active of which was monitored for 31 minutes (15:45-16:16) and analysed in detail.
This session had consumed 85 seconds of CPU time and 69 seconds of non-idle wait time, most of which (68”) – for reading single blocks into memory (indexed accesses). The remaining 1706” include 1361” (22’41”) idle wait time – this is time spent waiting for the application server process to send the next request. The average idle wait (the time between two consecutive requests) was 6 milliseconds.
In these 31 minutes this session made more than 353 000 calls to the database resulting in 212 400 communication round-trips between the Oracle server process and the application server process. Thus, this session alone has caused 111 voluntary context switches per second in the operating system.

12. Un-Shareable SQL Worst kind: Literals instead of binds
Bad kind: Dynamically built in-lists or or-lists
“Harmless” kinds: Capitalisation, white space, comments, variable names

13. Literals and Bind Variables – PL/SQL
Bad:
execute immediate 'delete from T where id='||id_to_delete;
Better:
execute immediate 'delete from T where id=:delID’ using id_to_delete;
Best:
delete from T where id=id_to_delete;

14. Literals and Bind Variables - Java
Bad
Statement stmt = conn.createStatement();
stmt.executeUpdate("delete from T where id=" +Integer.toString(id_to_telete));
Better
PreparedStatement pstmt = conn.prepareStatement("delete from T where id=?");
pstmt.setInt(1, id_to_delete);
pstmt.execute();

15. Live Specimens UPDATE ABRUFE SET ENDE_ABRUF=
TO_DATE(' :50:00','DD.MM.YYYY HH24:MI:SS')
WHERE ABRUF_ID = ;
begin PM01_EXPORT_PROG_BEITRAG(:result, ' '); end;
statement :=
'SELECT t1.faktor
FROM loader_faktoren_pr_segment t1, dim_person t2 WHERE t2.kundennummer = '||val_p_kunden.kundennummer_echt||'
AND t1.art = ''Ausbildung'‘
AND t2.ausbildung_s = t1.schluessel';
DBMS_SQL.PARSE(source_cursor, statement, dbms_sql.NATIVE);
SELECT * FROM apttab
WHERE apttab.finr = 1 AND apttab.artnr = ' '
AND apttab.varnr = 0 AND apttab.appos = 30
Select * from ODIN_STACK where ODS_INDEX ='PODSB_ ';

16. Thank you for your attention!
Visit us here at the conference: Booth C10