SQL Server & Siebel In About One Day

SQL Server & Siebel In About One Day
By Frank McBath

All Alliance Materials
Posted at: All materials referenced in this presentation are posted on the website, too.

Frank McBath Americas Technical Specialist
Siebel Microsoft Global Alliance 18 Years in IT With MS since 1999 Author of SQL Server Backup & Recovery Co-Author of SQL Server High Availability

McBath’s IT Life Roles & Technologies: Programmer Networking Database
4 years Networking 3 years Database 10+ years Enterprise Applications 7 years Consulting 8+ years Production Support 2 years as mgr of MS’s 1.5TB SAP database - Basis Types of Companies: Small, Medium, Huge Non-profit (American Cancer Society) Education (Michigan State University) High Tech (SAP, Microsoft) Outsourcing (Perot) Fortune 100 (Exxon, United Technologies, Cooper Industries)

What I do now… Make you successful using Microsoft technologies and Siebel. Think of me as a broker…

Format of this Day… Up to you… I have much more material than I can cover in a day… Heavy Q&A Go over Tech Notes in detail White board… Demo’s Hands On Use PowerPoint for some structure

Topics for the Day Marketecture Slides Joint Support Queue 64 bit
Performance & Monitoring Query Repro DB Best: Lab Tools Enterprise Manager SQL Agent Query Analyzer Profiler Perfmon Memory: SQL & Siebel /3GB AWE /PAE 64 Bit Disk Architecting a scalable solution EIM Everything you wanted to know Cursors What they are How they work Implicit Cursor Conversion

Marketecture

Siebel 7.7 Benchmark Results
10/04/ ,000 Concurrent User Siebel 7.7 Performance and Scalability Benchmark on HP-UX servers 10/01/2004 8,000 Concurrent User Siebel 7.7 Performance and Scalability Benchmark on HP BladeSystem infrastructure and HP Integrity servers running Microsoft SQL Server 2000 (64-bit) 10/01/2004 3,000 Concurrent User Siebel 7.7 Performance and Scalability Benchmark on a single HP BladeSystem Infrastructure solution running Microsoft SQL Server 2000 08/27/ ,500 Concurrent User Siebel 7.7 Industry Applications Performance and Scalability Benchmark on IBM eServer p690s / IBM DB2 UDB on eServer p5 570 02/17/2004 2,500 Concurrent User Siebel 7 Performance and Scalability Benchmark on IBM eServer pSeries® 650 and IBM DB2 UDB Source: Siebel’s Website

World Class Performance
30,000 concurrent Users

Siebel 7.7 8,000 User PSPP User Benchmark on HP/Windows/SQL64

Latest 7.7 8,000 User Benchmark

Oracle 10K vs. SQL Server 12K Oracle supported 10K users on rp8400 with 16x CPU 875Mhz with Oracle 9.x/Hp-ux posting 35% CPU and 18.2GB memory. MSSQL supported 12K users on rx5670 with 4x CPU 1.5Ghz with sql2K/windows2003 posting 47% CPU and 13.3GB memory. Result: 17.95% less CPU** 26% less memory 60% less cost 20% more users (12K vs 10K) SQL Server bit did more with less. *HP rx5670 is around $50K on HP web if you pay the full price. *HP rp8400 base price $124K ** rp CPU SpecInt 98.2, rx CPU SpecInt 60 ** (cont) (98.2 * 35%) = , (60 * 47%) = 28.2 ** (cont) 1 – (28.2/34.37) = 17.95% ** ** Scott Hall Slide

Memory, CPU & Clustering Limits Windows Server 2003 and SQL Server 2000
Max Resource Supported Windows Server 2003, Standard Edition Enterprise Edition Datacenter Edition 32-bit 64-bit SQL Server 2000, RAM (GB) 2 CPU 4 Cluster Nodes 3 321 641 8 32 64-bit Edition 64 512 Notes, as of March 03: There are no 64-CPU SMP or NUMA machines based on 32-bit chips currently on the market 512 GB of memory is the maximum amount of memory supported today in any SMP or NUMA system hardware available on the market 1 With Address Windowing Extensions (AWE) in SQL Server 2000

SQL Server 64 Bit (I) Scalability Manageability Cost Savings
Optimized for Windows Server 2003 and Itanium Great performance Large memory addressability (up to 32 TB) Nearly unlimited virtual memory (up to 8 TB) I/O savings due to larger memory buffer pools T-SQL code-compatibility with SQL Server 2000 8 node clustering support Same on-disk format as 32-bit for easy migration One setup for database & OLAP based on Windows Installer technology Compelling alternative to expensive Unix solutions Scalability Manageability Cost Savings The highly scalable database platform for memory intensive, performance-critical business applications

SQL Server 64 Bit (II) Memory problems are gone…
No more headaches with 3GB for proc cache But… Have to use client side tools

Joint Support Queue

Joint Support Queue What is it? How can it help you?
How do you engage it? Smitty’s presentation on our website:

PSS Service Offering Health Check Sample of the report
sql_health.doc Sample of the report C:\tmp\workshop\companyx_report.doc Great idea for Pre-Go Live Check

Performance, Monitoring & Life

The Whole Picture Everyone wants 5 9’s…
Architecting a scalable solution… Any of the below are just a part… You need a combination of all of them for a truly highly available system. Disk design Clustering Log Shipping Backups DBCCs Restores Rotations & Offsites Ex. Are you on the call list? Monitoring Processes Sytematized How are things escalated? Documented? Phone Numbers?

Disk Architecture & Siebel
Tech Note 9 Let’s walk through it…

Throughput (I) SQLIOStress.exe Simulates SQL Server Activity.
This article outlines the SQLIOStress utility. You can use this utility to perform stress tests on disk subsystems to simulate Microsoft SQL Server 2000 and Microsoft SQL Server 7.0 read, write, checkpoint, backup, sort, and read ahead activities. The utility was formerly named SQL70IOStress, but it has been upgraded to handle both SQL Server 7.0 and SQL Server 2000 I/O patterns. It has also been renamed SQLIOStress.

Throughput Numbers (II)
backup database siebeldb to disk='NUL:' go Processed pages for database 'siebeldb', file 'sieb752_Data' on file 1. Processed 1 pages for database 'siebeldb', file 'sieb752_Log' on file 1. BACKUP DATABASE successfully processed pages in seconds ( MB/sec).

Throughput (II) Cont. Note the issue with a differential backup and backing up to a NUL: device. Measure throughput with: SAN vendor’s tools Perfmon

Tool Box Windows System Monitor – PERFMON.EXE
System Monitor in the Performance console, started from Administrative Tools Query Analyzer – ISQLW.EXE Graphical showplan Statistics profile Statistics IO Profiler- PROFILER.EXE Spot problematic queries Use the Tuning or Duration templates Monitor the overhead carefully on your system Index Tuning Wizard & Siebel: OLTP & EIM usage Throughput Analysis Backup to NUL: SQLIOStress.exe

Perfmon Make sure you set DISKPERF –Y on command line to get counters dumped Performance Object: Physical Disk Counters: %Disk Time Number for entire logical drive should be less than 100% Avg Disk Queue Length (system requests on avg. waiting for disk access) Want 0 If it is above 2 (def. above 3), look into it See if it’s sustained queueing or temporary Avg. Disk Read/Write /sec (diff counters, and remember Logical vs. Physical) Nice to have: 5 – 7 ms (might be optimistic) Realistic (today’s technology): 20 – 25 ms on a moderately loaded system Log device service write times should be below 20 ms Technology dependent … See BOL (index “Bottlenecks” then “Monitoring Disk Activity”) for some more tips

Know Your Server’s Limits
Avoid Thrashing: CPU & IO. You typically don’t want CPU more than 80% Monitor the disk sub system Watch for disk queueing, but not useful if you don’t know the physical layout of the SAN LUNs. The following Perfmon counters work no matter… Logical Disk -> Avg Disk/Sec [Read | Write | Transfer] Counters are in milliseconds. .010 = 10 milliseconds (ms) What the numbers mean: 10ms Good 10ms to 20ms Reasonable 20ms to 50ms Busy more than 50ms Bad

File System Hot Spots Shows stall time per I/O request on a per file basis in milliseconds. Run and show the output. Cumulative… not for “spot” inspection. select DbId, FileId, IoStallMS/(NumberReads+NumberWrites) ‘ms/io’ from ::fn_virtualfilestats(-1,-1)

System Monitor Useful Counters Processor - % Processor Time
Physical Disk - %Disk Time, Avg. Disk Queue Length Memory – Available MBytes System – Context Switches / sec SQL Server Locks – Lock Waits/sec, Number of Deadlocks/sec SQLServer: Access Methods Full Scans/sec, Page Splits/sec, Table Lock Escalation/sec SQLServer: Buffer Manager Buffer Cache Hit Ratio, Lazy Writes/sec, Page Reads/sec, Page Writes/sec, ReadAhead Pages/sec SQLServer: Databases - Transactions/sec SQLServer: General Statistics - User Connections Q – How to Create a Performance Monitor Log

Troubleshooting SQL Server
Tech Note 5: Materials for Troubleshooting SQL Server Performance Supplemental Reading: 283696INF: Job to Monitor SQL Server 2000 Performance and Activity INF: How to View SQL Server 2000 Activity Data INF: How to View SQL Server 2000 Blocking Data HOW TO: View SQL Server 2000 Performance Data INF: How to Monitor SQL Server 2000 Traces 324692Support WebCast: How to Collect and Analyze Performance Data in Microsoft SQL Server 271509INF: How to Monitor SQL Server 2000 Blocking 224587HOW TO: Troubleshoot Application Performance with SQL Server 224453INF: Understanding and Resolving SQL Server 7.0 or 2000 Blocking Problems 243589HOW TO: Troubleshoot Slow-Running Queries on SQL Server 7.0 or Later 243588HOW TO: Troubleshoot the Performance of Ad-Hoc Queries 251004INF: How to Monitor SQL Server 7.0 Blocking 244455INF: Definition of Sysprocesses Waittype and Lastwaittype Fields for SQL Server 7.0

Poor Index Proc Tech Note 27: Poor Indexes
This stored procedure analyzes tables for “poor” index. Looks for cardinality of “1” which means 1 unique value for all existing rows. (Same slide as a bit later!) Bring up on QA and show

Look at SYSINDEXES Query sysindexes and look for very uniform values.
What are the chances that all indexes will be the same size… 100% NULL or 100% ‘Y’, etc… select name, dpages, reserved, reserved * * 8. 'bytes used' from sysindexes (nolock) where id = object_id('S_OPTY') sp_helpindex S_OPTY S_OPTY_F10 CAMP_CON_ID select count(distinct(CAMP_CON_ID)) from S_OPTY (nolock) 1 Distinct values for over 4 million rows (1 row(s) affected) 33 Indexes (at least) that are poor and take up 3.7G of disk.

name dpage reserved bytes used S_OPTY_P1 446628 521967 4,275,953,664 S_OPTY_U1 26615 53675 439,705,600 S_OPTY_II 18272 36766 301,187,072 S_OPTY_U2 26612 26816 219,676,672 S_OPTY_M56 21670 21784 178,454,528 S_OPTY_M57 S_OPTY_M52 20849 20952 171,638,784 S_OPTY_M53 19554 19648 160,956,416 S_OPTY_M55 19135 19224 157,483,008 S_OPTY_M50 18923 19016 155,779,072 S_OPTY_M51 S_OPTY_M54 S_OPTY_F5 18491 18576 152,174,592 S_OPTY_F4 17955 18040 147,783,680 S_OPTY_M1 17825 17904 146,669,568 S_OPTY_M8 15918 15984 130,940,928 S_OPTY_V2 19020 14128 115,736,576 S_OPTY_F1 14039 14096 115,474,432 S_OPTY_F10 S_OPTY_F3 S_OPTY_F50 S_OPTY_F51 S_OPTY_F52 S_OPTY_F53 14039 14096 115,474,432 S_OPTY_F54 S_OPTY_F57 S_OPTY_F58 S_OPTY_F59 S_OPTY_F6 S_OPTY_F60 S_OPTY_F7 S_OPTY_F8 S_OPTY_F9 S_OPTY_M2 S_OPTY_M3 S_OPTY_M4 S_OPTY_M9 21369 S_OPTY_V1 S_OPTY_V3 S_OPTY_V4 S_OPTY_V5 S_OPTY_V51 S_OPTY_V52 18980 S_OPTY_V53 S_OPTY_M61_X S_OPTY_F61_X S_OPTY_M6 13377 13424 109,969,408

SQL Server Profiler

Profiler Terminology Template Defines criteria for what to monitor
Saved in .tdf file Trace Captures data based upon selected events, data columns, and filters Filter Limits the results (Equal, Not like, etc…) Event Category Defines the way events are grouped Event Action generated within SQL engine SQL Profiler Terminology To use SQL Profiler, you need to understand the terminology that describes the way the tool functions. For example, you create a template that defines the data you want to collect. You collect this data by running a trace on the events defined in the template. While the trace is running, the event classes and data columns that describe the event data are displayed in SQL Profiler. Template A template defines the criteria for each event you want to monitor with SQL Profiler. For example, you can create a template, specifying which events, data columns, and filters to use. Then you can save the template and launch a trace with the current template settings. The trace data captured is based upon the options specified in the template. A template is not executed, and must be saved to a file with the .tdf extension. Trace A trace captures data based upon the selected events, data columns, and filters. For example, you can create a template to monitor exception errors. To do this, you would select to trace the Exception event class, and the Error, State, and Severity data columns, which need to be collected for the trace results to provide meaningful data. After you save the template, you can then run it as a trace, and collect data on any Exception events that occur in the server. This trace data can be saved and then replayed at a later date, or used immediately for analysis. Filter When you create a trace or template, you can define criteria to filter the data collected by the event. If traces are becoming too large, you can filter them based on the information you want, so that only a subset of the event data is collected. If a filter is not set, all events of the selected event classes are returned in the trace output. For example, you can limit the Microsoft® Windows® 2000 user names in the trace to specific users, reducing the output data to only those users in which you are interested. Event Category An event category defines the way events are grouped. For example, all lock events classes are grouped within the Locks event category. However, event categories only exist within SQL Profiler. This term does not reflect the way engine events are grouped. Event An event is an action generated within the Microsoft SQL Server™ engine. For example: The login connections, failures, and disconnections. The Transact-SQL SELECT, INSERT, UPDATE, and DELETE statements. The remote procedure call (RPC) batch status. The start or end of a stored procedure. The start or end of statements within stored procedures. The start or end of an SQL batch. An error written to the SQL Server error log. A lock acquired or released on a database object. An opened cursor. Security permissions checks. All of the data that is generated as a result of an event is displayed in the trace in a single row. This row contains columns of data called event classes that describe the event in detail. Event Class An event class is the column that describes the event that was produced by the server. The event class determines the type of data collected, and not all data columns are applicable to all event classes. Examples of event classes include: SQL:BatchCompleted, which indicates the completion of an SQL batch. The name of the computer on which the client is running. The ID of the object affected by the event, such as a table name. The SQL Server name of the user issuing the statement. The text of the Transact-SQL statement or stored procedure being executed. The time the event started and ended. Data Column The data columns describe the data collected for each of the event classes captured in the trace. Because the event class determines the type of data collected, not all data columns are applicable to all event classes. For example, the Binary Data data column, when captured for the Lock:Acquired event class, contains the value of the locked page ID or row but has no value for the Integer Data event class. Default data columns are populated automatically for all event classes.

Profiler Use Built-in templates Find the worst-performing queries
Filter by duration Identify the cause of a deadlock Monitor stored procedure performance Audit activity – C2 audits Reorder your output columns by Duration, CPU, read, writes, textdata, etc. Select what you need most TSQL StmtStarted, StmtCompleted SP StmtStarted, StmtCompleted, BatchStarted, BatchCompleted, Recompiles Performance – Show all stats (Only when you need detail) Reorder your output columns by Duration, CPU, read, writes, textdata, etc. Filter by Duration to find worst performing queries

Profiler in Production
Can be very CPU intensive My experience in production: 8x at 100% Filter! Filter! Filter! Let the computer tell you what’s going foul To proactively find out what is going wrong Filter on Duration > 30,000ms Run for 24 hours This will show you all the poor running queries on your system.

McBath’s Oil Argument Do you notice your car running better when you change your oil? No. But your car sure does. Make your database server run as efficient as possible. Makes it more scalable. More with less. Can’t make it run faster? Look at IO’s in loops or running all the time. Repack to higher fill factor. 5 to 4 IO’s is a 20% increase.

How to Use Profiler Which events to Watch Filter:
Real Time Sorting out Problems What happens if you don’t… Why am I capturing so much garbage in Profiler?

Capturing Performance Data
TechNote 5: Materials for Troubleshooting Microsoft SQL Server Performance

More Uses for Profiler Capture and filter for 24 hours
Profiler as stress testing tool Capture and play back Will only hammer the database, not the network or the Siebel application

Lab: Profiler EIM Analytics Siebel App
Find the statements that are running more than, for example, more than Reads C:\temp\eim.trc Analytics Find the Perf Problems: C:\temp\analytics000.trc C:\temp\analytics001.trc Siebel App C:\temp\app.trc Tech Note 25: Query Perf Impact

Managing Memory Tech Note 21: Memory boot.ini 3GB AWE PAE
SQL Server BOL:

SQL Parallelism & Siebel
set statistics io off go sp_configure 'max degree of parallelism', 1 reconfigure With Parallelism: Table 'EIM_CON_DTL'. Scan count 1, logical reads 10146, physical reads 0, read-ahead reads 0. Without Parallelism: Table 'EIM_CON_DTL'. Scan count 1, logical reads 552, physical reads 0, read-ahead reads 0.

Indexes: Siebel & SQL Server

Siebel Database Over 2,300 Tables Many with over 120 columns per table
2,200 Clustered Indexes (ROW_ID) 10,500 Non-Clustered Indexes Over Indexed, many NULL indexes 10,000+ Default Constraints 25,000+ Check Constraints Very few Stored Procedures Some Triggers used by workflow / assignment manager

Siebel Logical Schema vs. Physical Schema
Siebel database schema is designed as a cross platform logical schema, which is managed by Siebel Tools The logical schema is translated to a physical schema by using Siebel database utilities, such as DDLIMP The logical schema may be altered and mapped to a compatible physical data type depending on the DB platform and code page

Don’t Be Afraid to Add/Change Indexes…
You almost always have to. Work closely with Siebel Expert Services Make sure your Siebel meta data is in sync with SQL Server meta data… or bad things can happen next time you DDLSYNC… See examples in the next slides

Reality (I) Out of Box… sp_helpindex EIM_CONTACT3 EIM_CONTACT3_T01
EIM_CONTACT3_U1 A Large Customer… sp_helpindex EIM_CONTACT3 EIM_CONTACT3_T01 EIM_CONTACT3_U1

Reality (II) S_CONTACT_EI S_CONTACT_F6_X S_CONTACT_II S_CONTACT_M1
S_CONTACT_P1 S_CONTACT_U1 S_CONTACT_U2 S_CONTACT_V1 S_CONTACT_V2 S_CONTACT_V3 S_CONTACT_V5 S_CONTACT_EI S_CONTACT_F6_X S_CONTACT_II S_CONTACT_M1 S_CONTACT_M50 S_CONTACT_M8 S_CONTACT_ML1_X S_CONTACT_ML2_X S_CONTACT_ML3_X S_CONTACT_ML4_X S_CONTACT_ML5_X S_CONTACT_ML6_X S_CONTACT_P1 S_CONTACT_PREM01_X S_CONTACT_PREM02_X S_CONTACT_U1 S_CONTACT_U2 S_CONTACT_V3 Indexes in RED were custom.

Poor Indexes Get rid of 100% NULL indexes
Cost a lot on INSERTS/UPDATES/DELETES, ex. EIM Cost disk space Cost tape space when you back them up Only time one might be used: on an aggregate. It’s cheaper than a full scan. Rare, though. Stored Procedure below will examine the indexes on the top 100 tables for indexes that probably will not be used.

Query Repro & Cursors

Query Repro Covers about 90% of cases Query Repro: 1 hour
How to capture the commands RPC:Starting v. RPC:Completed Tieing beginning to end via process, spid, ... How to read the query plan Statistics IO, Profile

Siebel OM Query Execution and Fetching Mechanism
20 tables in a join… Siebel OM uses Server side API cursors For List applet functionality i.e. to maintain user state and support pending result sets To support multiple active statements per connections Fast Forward cursor with auto-fetch or Dynamic cursor when accessing text columns Tech Note 25 Sometimes there is an implicit conversions to Keyset (order by not covered by index ) Average fetch size is 3 or 4 rows – this is computed by dividing the ODBC buffer size by row size Siebel uses ODBC Prepare/Execute Example: Select * from table where x = ? What it looks like…

SQL Fetch Mechanism c1 c2 c3 c4 c5 a b c d e f g h i j Siebel 3 rows
sp_cursorprepex (….., 3) c1 c2 c3 c4 c5 a b c d e f g h i j Siebel OM 3 rows sp_cursorfetch (…,3) 3 rows

Build plan and return 40 rows ASAP
Typical Siebel OM Query int int int  Fast Forward, Parameterized, Auto Fetch, Auto Close (undocumented and subject to change) int int exec output, N'',N' SELECT T1.LAST_UPD_BY, T1.ROW_ID, T18.PRTNR_TYPE, T13.CREATED_BY, T2.ASGN_USR_EXCLD_FLG, . T2.PAR_OU_ID FROM dbo.S_PARTY T1 INNER JOIN dbo.S_ORG_EXT T2 ON T1.ROW_ID = T2.PAR_ROW_ID INNER JOIN dbo.S_ACCNT_POSTN T3 ON T2.PR_POSTN_ID = T3.POSITION_ID AND T2.ROW_ID = T3.OU_EXT_ID INNER JOIN dbo.S_PARTY T4 ON T3.POSITION_ID = T4.ROW_ID LEFT OUTER JOIN dbo.S_ORG_EXT T5 ON T2.PAR_OU_ID = T5.PAR_ROW_ID LEFT OUTER JOIN dbo.S_PRI_LST T6 ON T2.CURR_PRI_LST_ID = T6.ROW_ID LEFT OUTER JOIN dbo.S_POSTN T7 ON T2.PR_MGR_POSTN_ID = T7.ROW_ID LEFT OUTER JOIN dbo.S_USER T8 ON T7.PR_EMP_ID = T8.ROW_ID LEFT OUTER JOIN dbo.S_ORG_EXT T9 ON T2.PAR_BU_ID = T9.PAR_ROW_ID LEFT OUTER JOIN dbo.S_ORG_EXT T10 ON T1.PAR_PARTY_ID = T10.PAR_ROW_ID LEFT OUTER JOIN dbo.S_ORG_PRTNR T11 ON T1.ROW_ID = T11.PAR_ROW_ID LEFT OUTER JOIN dbo.S_ORG_EXT_SS T12 ON T1.ROW_ID = T12.PAR_ROW_ID LEFT OUTER JOIN dbo.S_BU T13 ON T1.ROW_ID = T13.PAR_ROW_ID LEFT OUTER JOIN dbo.S_OU_PRTNR_TIER T14 ON T2.PR_PRTNR_TIER_ID = T14.ROW_ID LEFT OUTER JOIN dbo.S_ASGN_GRP T15 ON T2.PR_TERR_ID = T15.ROW_ID LEFT OUTER JOIN dbo.S_INDUST T16 ON T2.PR_INDUST_ID = T16.ROW_ID LEFT OUTER JOIN dbo.S_ADDR_ORG T17 ON T2.PR_ADDR_ID = T17.ROW_ID LEFT OUTER JOIN dbo.S_OU_PRTNR_TYPE T18 ON T2.PR_PRTNR_TYPE_ID = T18.ROW_ID LEFT OUTER JOIN dbo.S_POSTN T19 ON T3.POSITION_ID = T19.PAR_ROW_ID LEFT OUTER JOIN dbo.S_USER T20 ON T19.PR_EMP_ID = T20.PAR_ROW_ID LEFT OUTER JOIN dbo.S_ORG_SYN T21 ON T2.PR_SYN_ID = T21.ROW_ID LEFT OUTER JOIN dbo.S_ORG_BU T22 ON T2.BU_ID = T22.BU_ID AND T2.ROW_ID = T22.ORG_ID LEFT OUTER JOIN dbo.S_PARTY T23 ON T22.BU_ID = T23.ROW_ID LEFT OUTER JOIN dbo.S_ORG_EXT T24 ON T22.BU_ID = T24.PAR_ROW_ID WHERE ((T2.PRTNR_FLG != ''N'') AND ((T13.BU_FLG = ''N'' OR T13.BU_FLG IS NULL) AND T2.PRTNR_FLG = ''Y'')) OPTION(FAST output Build plan and return 40 rows ASAP PAYOFF SLIDE – This is why the optimizer discussion was important – because we make it easy! 28688 – Fast Forword, Parameterized, Auto Fetch, Auto Close 8193 – Read Only, Allow Direct (get default result if no cursor) Fast 40 – build plan that returns first 40 rows as quickly as possible

sp_cursor* calls What is all this stuff? Undocumented calls
To repro query capture RPC:Starting in profiler which will have the correct parameters for the sp_cursor calls Do not compare performance of queries run directly with those run by the app through the cursor model. C:\temp\cursor.xls sp_cursorprepexec procedure Prepares a SQL statement for cursorization and returns an opened cursor as well as a prepared handle that can be subsequently passed to sp_cursorexecute. Syntax: sp_cursorprepexec ( <prepared handle> , <cursor> , <formal parameter dfn> ,<statement> [ ,<scrollopt> [,<ccopt> [ ,<rowcount> [ ,<bound param>] [,...n]]]] ) Arguments: <prepared handle> Required In/out integer Server generated prepared handle identifier. <cursor> Out Integer Server generated cursor identifier. <formal parameter defn> In String Null if the stmt is not a parameterized sql stmt else defn of variables substituted for parameter markers in the stmt. <stmt> This defines the result set of the cursor. <scrollopt> Optional in/out Scroll option: 1: KEYSET 2: DYNAMIC 4: FORWARD_ONLY 8: INSENSITIVE 16: FAST_FORWARD 4096: PARAMETRIZED_STMT 8192: AUTO_FETCH 16384: AUTO_CLOSE 32768: CHECK_ACCEPTED_TYPES This parameter is both input and output because it is possible that the requested option is not appropriate for the cursor defined by <statement>. <ccopt> Concurrency control option: 1: READ_ONLY 2: SCROLL_LOCKS (previously known as LOCKCC) 4: OPTIMISTIC (previously known as OPTCC) 8: OPTIMISTIC (previously known as OPTCCVAL) 8192: ALLOW_DIRECT 16384: UPDT_IN_PLACE 32768: CHECK_ACCEPTED_OPTS <rowcount> out Result set rows <bound_param> in any type

sp_cursor* Dox sp_cursorprepexec procedure
Prepares a SQL statement for cursorization and returns an opened cursor as well as a prepared handle that can be subsequently passed to sp_cursorexecute. Syntax: sp_cursorprepexec ( <prepared handle> , <cursor> , <formal parameter dfn> ,<statement> [ ,<scrollopt> [,<ccopt> [ ,<rowcount> [ ,<bound param>] [,...n]]]] ) Arguments: <prepared handle> Required In/out integer Server generated prepared handle identifier. <cursor> Out Integer Server generated cursor identifier. <formal parameter defn> In String Null if the stmt is not a parameterized sql stmt else defn of variables substituted for parameter markers in the stmt. <stmt> This defines the result set of the cursor. <scrollopt> Optional in/out Integer Scroll option: 1: KEYSET 2: DYNAMIC 4: FORWARD_ONLY 8: INSENSITIVE 16: FAST_FORWARD 4096: PARAMETRIZED_STMT 8192: AUTO_FETCH 16384: AUTO_CLOSE 32768: CHECK_ACCEPTED_TYPES This parameter is both input and output because it is possible that the requested option is not appropriate for the cursor defined by <statement>. <ccopt> Concurrency control option: 1: READ_ONLY 2: SCROLL_LOCKS (previously known as LOCKCC) 4: OPTIMISTIC (previously known as OPTCC) 8: OPTIMISTIC (previously known as OPTCCVAL) 8192: ALLOW_DIRECT 16384: UPDT_IN_PLACE 32768: CHECK_ACCEPTED_OPTS <rowcount> out Result set rows <bound_param> in any type

Why do I get a different query plan in the Query Analyzer ?
Bind value (Prepare Execute model) Hard Coding Values instead of binding at Run Time Cursor (SQL Server API cursor) Not putting the “ODBC Wrapper” SQL hint (Fast 40) Not including compiler options Text column Implicit Cursor Conversion Table Spools in one plan, but not the other Capture on Implicit Cursor Event in Profiler Also capture on “integer data” column

(N)TEXT Columns (N)TEXT column may cause performance problems
In the Siebel database schema A logical TEXT data type is always translated to a physical (N)TEXT column A VARCHAR data type can be translated to either a (N)VARCHAR column or a (N)TEXT column VARCHAR(2000+) is translated to a (N)TEXT column

One size fits all: Implicit Cursor Conversions
One type of cursor is requested, but it cannot be fulfilled in it’s native call. Rather than fail, SQL Server converts internally. Performance problems. For example, Siebel uses an “option fast 40” Fast forward, read only requested with an ORDER BY, yet no index on the WHERE clause that is ordered. SQL Server converts to a KEYSET cursor which spools off to TEMPDB for the sort. Fix: make an index that matches the ORDER BY. KEYSET conversion goes away. SQL Profiler: Event -> Cursors -> CursorsImplicitConversion

Profiler Properties

ODBC implicit cursor conversions: BOL

Query Repro: Quick and Dirty
Capture on RPC: Starting Event on Profiler Cut and paste it into Query Analyzer 99% of the time it will give you the same plan that is coming out of Siebel. DON’T: Spool Siebel out at the client, hard code the values and put into Query Analyzer. Probably won’t work For example, it won’t have the OPTION FAST 40 Don’t hard code variables

How to make the Query in QA
print 'declaring variables' int int int -- -- SCROLLOPT = (AutoClose) (AutoFetch) (Parameterized) + 4 (Forward Only) print 'running sp_cursorprepare' exec output , varchar(30)' -- , N'SELECT * FROM authors WHERE au_lname OPTION (FAST 1)' , N'SELECT * FROM authors WHERE au_lname , 1 output output print 'declaring more variables' int int int = 1 -- SCROLLOPT = (AutoClose) (AutoFetch) + 4 (Forward Only) print 'executing cursor' output, 'R%' print 'select the results from the cusor execute' @P6

Which SPID Eating up CPU

Locating the Query Using the Most CPU
Perfmon Counters: Threads: % Processor Time ID Process (NT PID) ID Thread (KPID) select spid from master..sysprocesses where kpid = <ID Thread> Select all SQLSERVR/0 to SQLSERVR/99 DBCC INPUTBUFFER(spid)

Quick Estimate for Long Running Queries
select datediff(mi, last_batch,getdate())'minutes',spid, waittype, cpu, physical_io, convert(char(15),hostname), convert(char(15),program_name), convert(char(20),getdate()),spid, last_batch, cmd from master..sysprocesses where spid > 50 and cmd not like '%WAIT%' and datediff(mi, last_batch,getdate()) > 1 order by last_batch

Reading Query Plans, IO, & Statistics

Set Statistics Profile/IO
What are they? How to run: one at a time Believe this over everything else. How to Read them 1 read = 8K page Logical v. Physical Read aheads Kevin’s Customer is going crazy. Somewhere in Siebel the SQL they spit out included an UPPER on a ROWID which caused a scan in an inner loop. 560GB read for the query, 140 seconds.

Estimated vs. Actual Query Analyzer gives you and *ESTIMATE*
Statistics profile is the *ACTUAL* plan Can be different based upon several factors, ex. memory pressure and cached information

Statistics IO use siebeldb go set statistics io on
select count(*) from S_CONTACT set statistics io off 6 (1 row(s) affected) Table 'S_CONTACT'. Scan count 1, logical reads 1, physical reads 0, read-ahead reads 0.

Quick Demo Set statistics IO Set statistics profile

Siebel and Update Statistics
Problem: Siebel queries join a lot of tables A lot of the tables joined might be smaller than the threshold to execute an automatic update statistics Result: Bad plans for the join Stale statistics and EIM A Plan that tips over. 98 jobs run in 5 minutes. 2 run in 1 hour or… a lot longer. EIM running faster than Auto Update Stats can kick off. Solution run a manual update statistics on those tables Check rowmodctr in sysindexes for indid=1 See Appendix for code on auto update stats

Example of Stale Statistics
set statistics io on Table ‘S_CONTACT'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0. Table ‘S_OPPTY'. Scan count 4382, logical reads 21439, physical reads 650, read-ahead reads 877. Table ‘S_PARTY'. Scan count 2, logical reads 17573, physical reads 199, read-ahead reads 0. After UPDATE STATISTICS with sample of 10%: Table ‘S_OPPTY'. Scan count 192, logical reads 1440, physical reads 0, read-ahead reads 11. Table ‘S_PARTY'. Scan count 4, logical reads 1507, physical reads 3, read-ahead reads 59.

Waittypes: What they are, why they are important.

Why Wait Types? Wait types will help define where your bottleneck is.
They are seen in the master..sysprocesses table in a column called “waittype”. select waittype, * from master..sysprocesses There are all kinds of waittypes. For example, blocking due to database locks, network io, disk queueing, etc… The key to solving throughput is understanding what’s damming up the river.

Wait Types If a thread goes into a sleep status, a wait state is set.
The wait state is contained in master..sysprocesses in the columns waittype, and lastwaittype. Lastwaittype is a character description of the last wait state for this thread. It is not reset until another wait state occurs. Waittype is a varbinary wait state that is the current wait state. A wait time of 0 means the thread is currently running. See SQL_PERF.DOC for detailed information:

Track_waitstats stored procedure
Track_waitstats is a stored procedure that will capture waitstats from DBCC SQLPERF, and provide a ranking of descending order based on percentage. This is useful in identifying the greatest opportunites for performance improvements. See Appendix for Stored Procedure on Perf See the sample output…

Sample Output

Commentary The above sample shows the majority share of wait time, 48%, being due to network IO waits. Improving network IO is the single largest opportunity for improving application performance. Other lesser opportunities in the above example include LCK_M_X (exlusive locks) and WRITELOG (transaction log). Exclusive lock waits account for almost 13% of total wait time. An examination of transaction management may offer clues as to whether improvements can be made here. WRITELOG means threads are waiting for physical writes to complete to the transaction log. Given the 11% writelog waits, a further analysis of PERFMON disk queues for the transaction log will confirm whether the IO capacity of the transaction log drives have trouble keeping up with write requests as shown by steady and high disk queues.

EIM: Siebel & SQL Server

The Methodology Attached is a PDF of the whole tuning methodology. (33” x 44” Poster). Includes SQL Server & Siebel steps. Also on Alliance website. Siebel Note 409: EIM

Why This Presentation? Everyone does EIM.
All databases have problems doing it. Goal: Show you how to optimize SQL Server for EIM.

Symptoms of EIM? The first several thousand will always go fast.
Performance deteriorates over time in a logarithmic pattern. Why? Because the b-trees grow and more levels to traverse. The curve grows logarithmically, *NOT* arithmetically. Ex. The first 5K rows go in at a rate of 1 minute. After 2 weeks of loading that same 5K rows takes 1 hour.

Phases in Data Loading Get flat file Scrub the data
Load the EIM_* tables Run EIM Check results Archive and restart What everyone wants to do… load 100M rows and leave…

Best First Strategy Drop all but P (cluster) and U (unique) indexes
Remove Hints (IFB File) sp_indexoption Smaller EIM table 20K batch Try and run 2 or more jobs in parallel until disk or CPU the bottleneck. Remember to put indexes back on after load.

Case Study Activities: EIM_ACTIVITY Average Row Size: 2,703 bytes
Batch Size: 20K, Siebel Out of the Box Baseline: 1 min 36 seconds. 12,500 rows/min, 750K rows/hr = 2G/hr Tuning Done: Clustered and Unique index only on both Base & EIM tables sp_indexoption for row locks on EIM_ACTIVITY table. After Tuning: Start: 9:05PM, End: 11:23PM, 138 minutes. 8 EIM threads in parallel against EIM_ACTIVITY Batches: 726 (179 to 905 = 14,520,000 records) 14,520,000 rows / 138 min = 105,217 rows/min, 6.3M rows/hr = 17G /hr HDS / Hitachi 9500 Series SAN with 2G Cache 8 CPU x 8G RAM 50% to 60% CPU No degradation over load run time. Linear.

Efficiently Loading Data (I)
Load into pre-staging tables Scrub in tempdb Minimal logging Recovery model for EIM during the load for the siebeldb. ALTER DATABASE siebeldb SET RECOVERY SIMPLE Implications. Be very aware of them. Read BOL. Scrub in SQL Server: Efficiencies of SQL Server caching, memory management, cpu usage, and a big database backend server. Use set wise processing, not cursors If have to use cursors, use Fast Forward/Read Only PL/SQL Consider use of “NOLOCK” for better performance. Dirty Reads. Works well if you are the only one on the server, etc… See Appendix for performance samples on cursors

Efficiently Loading Data (II)
BULK INSERT vs BCP BULK INSERT is “in memory” BCP is more configurable Both are single threaded Only run on one CPU Run multiple BULK INSERTS at once across multiple CPUs. Use Bulk Insert When Possible Perf is Linear: 5M/sec/cpu on 500Mhz

Bulk Insert vs. BCP Better Design Data Read Into BCP Memory…
Data Read Directly To SQL Server Memory Then Copied To SQL Server Memory.

More On Bulk Loading Gert Draper’s Presentation:
39 Slides of the best information on High Performance Data Loading

Efficiently Loading Data (III)
Rules of thumb for bulk operations: One bulk op per CPU. Limit the batch Commit Size (batch size) to about 2,000 rows per batch. Adjust up or down based on your testing. Remember, if loading in clustered index in sequence, only use one thread. Else, 2 bulk operations would step on each other and block. Bulk operations are very high performance. Conditions in BOL (ex. TABLOCK) Try and load the data pre sorted

Efficiently Loading Data (IV)
Run a Siebel server on the database server so all EIM related traffic is contained on one physical machine. No network if possible. Big Company: Running EIM over network. Put all on database server, cut runtime 30%. Broke up after the load. Presize DB. No autogrow!

Examples of DTS/ETL Designs
Poor Design Too Much Network Traffic Good Design Separate Siebel When Loading Done

Efficiently Loading Data (V)
Disable any triggers on the databases (such as workflow triggers) and then re-apply/enable them after the process is done. If this is not for an initial data load, this means that Workflow Manager or Assignment Manager will not function during the load for the new or updated data. Disable docking replication. Will see in SQL Profiler (ex. INSERT into S_DOCK_TXN_LOG)

Running in Parallel Blocking & Lock escalation sp_indexoption
1 row trick

sp_indexoption Use sp_indexoption to control locking behavior on a more granular level. Key ideas: Enable row level locking Disable page level locking On clustered index of EIM table Base table, less so because clustered index is on ROW_ID which is more random. EIM table clustered index is on BATCH NUMBER What about using more resources? Chances are, single user.

Table Locks and EIM Three types of locks: row, page, table.
SQL Server does dynamic lock escalation. (See with SP_LOCK) If enabled row, disabled page… you still have table locks. This can cause severe concurrency issues. The fix…

The “1 Row Trick” Insert a dummy row into the EIM table.
Begin a transaction, update the dummy row, don’t commit it. This will hold 1 lock on the table and prevent lock escalation to a table level lock. Now only row level locking!

Batches in Parallel sp_indexoption Usage:
EXEC sp_indexoption 'EIM_CONTACT.EIM_CONTACT_U1', 'disallowpagelocks', TRUE go 'allowrowlocks', One Row Trick: begin tran update EIM_CONTACT set ROW_ID = '' where IF_ROW_BATCH_NUM = ‘99’ Note: NO COMMIT!

1211 If still having problems…
Trace Flag 1211 turns off lock escalation at the database server level. Row locks ONLY! Brute force. Have to turn it on via command line “-T1211” sqlservr –d"C:\Program Files\Microsoft SQL Server\MSSQL\Data\master.mdf" –T122 May have to use the “-G” switch. See next slide. Can cause significant memory pressure in an OLTP environment due to the 3GB space. Last resort Don’t forget to turn it off!

-G Specifies the amount of virtual address space (in megabytes) SQL Server will leave available for memory allocations within the SQL Server process, but outside the SQL Server memory pool. This is the area used by SQL Server for loading items such as extended procedure .dll files, the OLE DB providers referenced by distributed queries, and automation objects referenced in Transact-SQL statements. The default is 128 megabytes (MB). Use of this option may help tune memory allocation, but only when physical memory exceeds 2 gigabytes (GB) for the SQL Server 2000 Personal Edition or SQL Server 2000 Standard Edition, or 3 GB for SQL Server 2000 Enterprise Edition. Configurations with less physical memory will not benefit from using this option. Use of this option may be appropriate in large memory configurations in which the memory usage requirements of SQL Server are atypical and the virtual address space of the SQL Server process is totally in use. Incorrect use of this option can lead to conditions under which an instance of SQL Server may not start or may encounter run-time errors. Use the default for the -g parameter unless you see the following warning in the SQL Server error log:WARNING: Clearing procedure cache to free contiguous memoryThis message may indicate that SQL Server is trying to free parts of the SQL Server memory pool in order to find space for items such as extended stored procedure .dll files or automation objects. In this case, consider increasing the amount of memory reserved by the -g switch. Using a value lower than the default will increase the amount of memory available to the buffer pool and thread stacks; this may, in turn, provide some performance benefit to memory-intensive workloads in systems that do not use many extended stored procedures, distributed queries, or automation objects.

SQL Parallelism & EIM (I)
set statistics io off go sp_configure 'max degree of parallelism', 1 reconfigure With Parallelism: Table 'EIM_CON_DTL'. Scan count 1, logical reads 10146, physical reads 0, read-ahead reads 0. Without Parallelism: Table 'EIM_CON_DTL'. Scan count 1, logical reads 552, physical reads 0, read-ahead reads 0.

SQL Parallelism & EIM (II)
May want to turn back on for parallel index creation when putting indexes back on. Then turn it off when Siebel running.

One Huge EIM Table… Same query, more logical/physical reads, also hurts cache… Why? B-Tree is deeper. UPDATE EIM_CON_DTL SET CXP_X_START_DT = DATEADD(hh, - DATEPART(hh, CXP_X_START_DT), DATEADD(mi, - DATEPART(mi, CXP_X_START_DT), DATEADD(ss, -DATEPART(ss, CXP_X_START_DT), DATEADD(ms, - DATEPART(ms, CXP_X_START_DT), CXP_X_START_DT)))) WHERE (IF_ROW_BATCH_NUM = 1 AND CXP_X_START_DT IS NOT NULL) select count(*) from EIM_CON_DTL -- 25,226,736 Table 'EIM_CON_DTL'. Scan count 8, logical reads 13492, physical reads 106, read-ahead reads select * into EIM_CON_DTL from EIM_CON_DTL_full where IF_ROW_BATCH_NUM between 1 and 6 -- 240,765 Table 'EIM_CON_DTL'. Scan count 1, logical reads 10032, physical reads 13, read-ahead reads 96.

Depth of B-Tree & EIM Table
Better Design Pro’s: Less Maint Con’s: More IO Pro’s: More Maint Con’s: Less IO IF_BATCH_NUM =1 IF_BATCH_NUM =1 Less data in EIM table means less hops and IO. More data in EIM table means more hops and IO.

Covered Index Example Covering indexes are not always the answer… IO was significantly less but the run time was the same due to superior hardware. More problems caused with extra indexes than solve. create index EIM_COVER_IDX on EIM_CON_DTL(IF_ROW_BATCH_NUM, T_PARTY__STA, IF_ROW_STAT_NUM, PARTY_UID, PARTY_TYPE_CD, CXP_NAME, CXP_TYPE, T_CX_CONPHONE__STA,T_PARTY__RID, T_CX_CON_NAME__STA) Before: Table 'EIM_CON_DTL'. Scan count 1, logical reads 10032, physical reads 0, read-ahead reads 0. Table 'S_LST_OF_VAL'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0. Table 'CX_CON_NAME'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0. (0 row(s) affected) After: Table 'EIM_CON_DTL'. Scan count 1, logical reads 489, physical reads 0, read-ahead reads 0.

Profiler & EIM Traces Siebel EIM Level Trace Level Maximum Verbose
Capture on events: RPC: COMPLETED SQL: BATCHCOMPLETED Save profiler traces and filter on Reads > These tend to be your problems. Correlate these to the Siebel EIM Trace files.

Hints & EIM (I) WITH HINTS:
/* UPDATE BT SET BT.END_DT = IT.AS_END_DT, BT.NAME = IT.AS_NAME, BT.START_DT = IT.AS_START_DT, BT.X_CHANGE_CODE = IT.X_CHANGE_CODE, BT.X_CHANGE_DATE = IT.X_CHANGE_DATE, BT.X_CHANGE_TYPE = IT.X_CHANGE_TYPE, BT.X_POLICY_TYPE = IT.X_POLICY_TYPE, BT.X_PREMIUM = IT.X_PREMIUM, BT.X_PRINTED_FLG = IT.X_PRINTED_FLG, BT.X_PRODUCT_DESC = IT.X_PRODUCT_DESC, BT.X_PRODUCT_TYPE = IT.X_PRODUCT_TYPE, BT.X_RATE_PLAN_CD = IT.X_RATE_PLAN_CD, BT.X_SOURCE_SYSTEM = IT.X_SOURCE_SYSTEM, BT.LAST_UPD BT.LAST_UPD_BY BT.MODIFICATION_NUM = BT.MODIFICATION_NUM + 1 FROM dbo.S_ASSET BT (INDEX = S_ASSET_P1), dbo.EIM_ASSET IT (INDEX = EIM_ASSET_M1) WHERE (BT.ROW_ID = IT.T_ASSET__RID AND IT.IF_ROW_BATCH_NUM = AND IT.IF_ROW_STAT_NUM = 0 AND IT.T_ASSET__EXS = 'Y' AND IT.T_ASSET__UNQ = 'Y' AND IT.T_ASSET__DUP = 'N' AND IT.T_ASSET__STA = 0) */ WITH HINTS: Table 'S_ASSET'. Scan count 1273, logical reads 4038, physical reads 0, read-ahead reads 0. Table ‘EIM_ASSET'. Scan count 1, logical reads 5875, physical reads 0, read-ahead reads 0. WITHOUT HINTS: Table ‘EIM_ASSET'. Scan count 1, logical reads 1774, physical reads 0, read-ahead reads 0.

Hints & EIM (II) WITH HINT:
Table 'S_CONTACT'. Scan count 1142, logical reads 8008, physical reads 0, read-ahead reads 0. Table ‘EIM_CONTACT'. Scan count 1, logical reads 3162, physical reads 0, read-ahead reads 0. WITHOUT HINT: Table ‘EIM_CONTACT'. Scan count 1, logical reads 231, physical reads 0, read-ahead reads 0. Table 'S_APPLD_CVRG'. Scan count 1, logical reads , physical reads 0, read-ahead reads Table ‘EIM_ASSET5_FN'. Scan count 1, logical reads 366, physical reads 0, read-ahead reads 0. Table 'S_APPLD_CVRG'. Scan count 1268, logical reads 10203, physical reads 697, read-ahead reads 0.

IFB File (I)

IFB File (II) [Siebel Interface Manager] USER NAME = "SADMIN"
PASSWORD = "SADMIN" PROCESS = Import ISS ; FIXES USE INDEX HINTS = FALSE USE ESSENTIAL HINTS = FALSE ; ; This group of processes provides samples for import data through ; all the interface tables, broken up into logical groups. Note ; that the order of import is often significant. [IMPORT ISS] TYPE = IMPORT TABLE = EIM_ISS BATCH = 1 ONLY BASE TABLES = S_ISS ONLY BASE COLUMNS = S_ISS.NAME,S_ISS.LANG_ID,S_ISS.BU_ID

EIM, Hints & Errors 2003-10-20 17:52:45 UPDATE dbo.EIM_ISS
SET T_ISS__RID = '1-239-' + CONVERT(varchar, MS_IDENT - 1), T_ISS__UNQ = 'Y' FROM dbo.EIM_ISS T1 WHERE (T_ISS__EXS = 'N' AND ROW_ID = (SELECT MIN(ROW_ID) FROM dbo.EIM_ISS T2 (INDEX (EIM_ISS_T05)) WHERE ((T2.ISS_LANG_ID = T1.ISS_LANG_ID OR (ISS_LANG_ID IS NULL AND T1.ISS_LANG_ID IS NULL)) AND T2.T_ISS_BU_ID = T1.T_ISS_BU_ID AND T2.ISS_NAME = T1.ISS_NAME AND T2.IF_ROW_BATCH_NUM = 1 AND T2.IF_ROW_STAT_NUM = 0 )) AND IF_ROW_BATCH_NUM = 1 AND IF_ROW_STAT_NUM = 0 AND T_ISS__STA = 0) GenericLog GenericError :52:45 [Microsoft][ODBC SQL Server Driver][SQL Server]Index 'EIM_ISS_T05' on table 'dbo.EIM_ISS' (specified in the FROM clause) does not exist. GenericLog GenericError :52:45 (compmain.cpp 16(1555) err= sys=0) SBL-EIM-00101: Invalid arguments to function. GenericLog GenericError :52:45 (smisched.cpp 17(821) err= sys=0) SBL-EIM-00101: Invalid arguments to function.

Server Level Disabling of Hints
Sometimes hints are hard coded and you cannot take them out. As a last resort, if putting commands in IFB File do not work, the DBA can turn on Trace Flag 8602 at the database level. (SQL Server Magazine Nov 2002) This causes SQL Server to ignore ALL hints passed to the optimizer even if they are hard coded.

8602 Can turn on from Query Analyzer… Or… at the command line level…
DBCC TRACEON(-1, 8602) DBCC TRACEOFF(-1,8602) Or… at the command line level… sqlservr –d"C:\Program Files\Microsoft SQL Server\MSSQL\Data\master.mdf" T8602

IFB File (III) Limit the Siebel metadata read Configure IFB using:
ONLY BASE TABLES, IGNORE BASE TABLES, ONLY BASE COLUMNS, IGNORE BASE COLUMNS. Will see time consumed in EIM log file Read Siebel Technote #409.

Stale Statistics What happens when stats get old…
Bad plans. A query that normally runs in a few seconds can take 30 minutes. How do the stats get stale? EIM updates every row in the EIM_* table. The process that auto updates stats doesn’t “wake up” in time between runs. Correct this by running UPDATE STATISTICS between runs or a SQL AGENT job that wakes up and runs. Consider turning off auto update stats for the data load. It’s all about getting predictable performance. 100 runs… 97 run in 5 minutes… 3 run in 4 hours… each… at midnight when no one is around to kill it. Strategy: Hard to pick a bad plan with only a clustered index.

Fragmentation Happens…
During big data loads Run DBCC REINDEX to correct Think about: Fill Factor Pad Index Check with Perfmon: SQL Server: Access Methods -> Page Splits / sec Trend over time Defrag & update stats between large runs DBCC INDEXDEFRAG, REINDEX, and drop recreate Samples and run times Pro’s and con’s Possibly use file groups for initial load. For example, put the EIM tables in their own FG… or put the indexes on their own FG.

Recovery Models in EIM Run a full or differential backup frequently during the day. Coalesce database during backup. Known point to restart jobs if needed. Weigh the issues of recovery vs. lost data. Note: Switching from FULL to SIMPLE will break the log chain and have recovery consequences. Always make a full backup after switching to SIMPLE.

Indexes & EIM In general, indexes degrade performance on data loading.
Some indexes are needed: Clustered and Unique. Some are needed for child table references. Add indexes back on after loads done. Remember to re-enable parallelism for the index create, then turn off for production. Work with ES/TAM to prune off 100% NULL indexes

Reality (I) Out of Box… sp_helpindex EIM_CONTACT3 EIM_CONTACT3_T01
EIM_CONTACT3_U1 A Large Customer… sp_helpindex EIM_CONTACT3 EIM_CONTACT3_T01 EIM_CONTACT3_U1

Reality (II) S_CONTACT_EI S_CONTACT_F6_X S_CONTACT_II S_CONTACT_M1
S_CONTACT_P1 S_CONTACT_U1 S_CONTACT_U2 S_CONTACT_V1 S_CONTACT_V2 S_CONTACT_V3 S_CONTACT_V5 S_CONTACT_EI S_CONTACT_F6_X S_CONTACT_II S_CONTACT_M1 S_CONTACT_M50 S_CONTACT_M8 S_CONTACT_ML1_X S_CONTACT_ML2_X S_CONTACT_ML3_X S_CONTACT_ML4_X S_CONTACT_ML5_X S_CONTACT_ML6_X S_CONTACT_P1 S_CONTACT_PREM01_X S_CONTACT_PREM02_X S_CONTACT_U1 S_CONTACT_U2 S_CONTACT_V3 Indexes in RED were custom.

Look at SYSINDEXES Query sysindexes and look for very uniform values.
What are the chances that all indexes will be the same size… 100% NULL or 100% ‘Y’, etc… select name, dpages, reserved, reserved * * 8. 'bytes used' from sysindexes (nolock) where id = object_id('S_OPTY') sp_helpindex S_OPTY S_OPTY_F10 CAMP_CON_ID select count(distinct(CAMP_CON_ID)) from S_OPTY (nolock) 1 Distinct values for over 4 million rows (1 row(s) affected) 33 Indexes (at least) that are poor and take up 3.7G of disk.

name dpage reserved bytes used S_OPTY_P1 446628 521967 4,275,953,664 S_OPTY_U1 26615 53675 439,705,600 S_OPTY_II 18272 36766 301,187,072 S_OPTY_U2 26612 26816 219,676,672 S_OPTY_M56 21670 21784 178,454,528 S_OPTY_M57 S_OPTY_M52 20849 20952 171,638,784 S_OPTY_M53 19554 19648 160,956,416 S_OPTY_M55 19135 19224 157,483,008 S_OPTY_M50 18923 19016 155,779,072 S_OPTY_M51 S_OPTY_M54 S_OPTY_F5 18491 18576 152,174,592 S_OPTY_F4 17955 18040 147,783,680 S_OPTY_M1 17825 17904 146,669,568 S_OPTY_M8 15918 15984 130,940,928 S_OPTY_V2 19020 14128 115,736,576 S_OPTY_F1 14039 14096 115,474,432 S_OPTY_F10 S_OPTY_F3 S_OPTY_F50 S_OPTY_F51 S_OPTY_F52 S_OPTY_F53 14039 14096 115,474,432 S_OPTY_F54 S_OPTY_F57 S_OPTY_F58 S_OPTY_F59 S_OPTY_F6 S_OPTY_F60 S_OPTY_F7 S_OPTY_F8 S_OPTY_F9 S_OPTY_M2 S_OPTY_M3 S_OPTY_M4 S_OPTY_M9 21369 S_OPTY_V1 S_OPTY_V3 S_OPTY_V4 S_OPTY_V5 S_OPTY_V51 S_OPTY_V52 18980 S_OPTY_V53 S_OPTY_M61_X S_OPTY_F61_X S_OPTY_M6 13377 13424 109,969,408

Summary (I) Optimize your EIM Batch Size
Hint Removal: Siebel and SQL Server Turn off docking replication Get rid of workflow triggers Only load up the tables needed from the Siebel meta data. Loading the whole catalog can represent 25% the time of your whole batch run. Run batches in parallel Exclude validation of non-used data. If you know something is never NULL (ex. EmpID), then don’t check for it.

Summary (II) Update Stats Stale stats can cause problems
Investigate turning autostats off Defrag between large runs Defrag both EIM_ and base tables On large initial loads, put fill factor and pad index to 50%. Cut down on page splits. Default is 100% full. Use minimally logged operations to load and scrub. Bulk Insert, SELECT/INTO Recovery Models Run all data loading locally Scrub data inside SQL Server. No cursors. Make the right indexes Try monster clustered index only Get rid of unused indexes Add them back after runs Work with ES to resolve support issues.

Questions?

Reference (I) SQL Server 2000 Failover Clustering
Products Designed for the Microsoft Windows Operating Systems Microsoft Windows Clustering: Storage Area Networks Recommended Private "Heartbeat" Configuration on a Cluster Serverhttp://support.microsoft.com/default.aspx?scid=kb;en-us;258750 Cluster Network Role Changes Automatically Network Failure Detection and Recovery in a Two-Node Server Cluster Cluster Resources Quorum Log Size Defaults to 64 KB Removing the HBA Cable on a Server Cluster Support for Multiple Clusters Attached to the Same SAN Device How the Cluster Service Takes Ownership of a Disk on the Shared Bus The Microsoft support policy for server clusters, the Hardware Compatibility List, and the Windows Server Catalog Microsoft Support for Server Clusters with 3rd Party System Components

Reference (II) Supported Fibre Channel Configurations
Troubleshooting Multiple Cluster Symptoms on the Same SAN Explanation of Why Server Clusters Do Not Verify that Resources will Work Properly on All Nodes HOWTO: Rebuild or Move MSDTC Used with a SQL Failover Cluster Clusrest.exe: Cluster Quorum Restore Utility Recovering from a Lost or Corrupted Quorum Log How to Properly Restore Cluster Information

Reference (III) Related articles – SQL Server Configuration
HOW TO: Configure memory for more than 2 GB in SQL Server Related articles – SQL Server Security SQL Server Security Center Microsoft Security Bulletin Search Implementing Security SQL Server Security How-to Guide

Reference (IV) SQL Server 2000 Auditing
Securing SQL Server 2000 Resource Guide Securing Windows 2000 Terminal Services Related articles – SQL Server Disaster Recovery Benchmark: High Transaction Throughput During Online Database Backup Benchmark: High Performance Online Database Backup for Very Large Databases Database Backup and Recovery Backing Up and Restoring Databases Restoring System and User Databases Restoring a Database Related articles – SQL Server Log Shipping Log Shipping in SQL Server Part 1 Log Shipping in SQL Server Part 2 HOW TO: Transfer Logins and Passwords Between Instances of SQL Server

Reference (V) Related articles – SQL Server Performance
HOW TO: Troubleshoot Application Performance with SQL Server INF: How to Monitor SQL Server 2000 Blocking INF: How to Monitor SQL Server 7.0 Blocking INF: Definition of Sysprocesses Waittype and Lastwaittype Fields for SQL Server 7.0 INF: Troubleshooting Stored Procedure Recompilation HOW TO: Identify the Cause of Recompilation in an SP:Recompile Event FIX: Concurrency Enhancements for the Tempdb Database HOW TO: Troubleshoot Slow-Running Queries on SQL Server 7.0 or Later HOW TO: Use the SQLIOStress Utility to Stress a Disk Subsystem Such As SQL Server IOMETER: Intel's I/O Performance Analysis Tool

Case Study (I) Problem: EIM initial processing 30,000 – 40,000 rows per hour Final Results: EIM initial processing 1,300,000 rows per hour Process: Standard Siebel Expert Services methodology Tools: Siebel SQL Tracing for EIM to show 10 most expensive queries Window System Monitor to measure I/O, CPU and memory SQL Server Profiler to capture database events Steps: Spot problematic queries – long running insert Memory, SQL settings all fine, I/O was poor due to HW problems Use Query Analyzer (Show Execution Plan) to determine whether additional index will help – in this case: No Minimize I/O by dropping indexes on Base Tables (S_CONTACT) – 400,000+ Separate data into batches by key ranges to avoid deadlocks and run 4 EIM processes in parallel – 1.3 Million rows / hour Recreate indexes, update stats, defrag table and indexes Investigate I/O subsystem Hardware / SAN

Reference Siebel Note 409

Appendix (II) Configuration Specs Real World
sp_configure go name minimum maximum config_value run_value affinity mask affinity64 mask allow updates awe enabled c2 audit mode cost threshold for parallelism Cross DB Ownership Chaining cursor threshold default full-text language default language fill factor (%) index create memory (KB) lightweight pooling locks max degree of parallelism max server memory (MB) max text repl size (B) max worker threads media retention min memory per query (KB) min server memory (MB) nested triggers network packet size (B) open objects priority boost query governor cost limit query wait (s) recovery interval (min) remote access remote login timeout (s) remote proc trans remote query timeout (s) scan for startup procs set working set size show advanced options two digit year cutoff user connections user options exec master..xp_msver Go Index Name Internal_Value Character_Value ProductName NULL Microsoft SQL Server ProductVersion Language English (United States) Platform NULL NT INTEL IA64 Comments NULL NT INTEL IA64 CompanyName NULL Microsoft Corporation FileDescription NULL SQL Server Windows NT FileVersion NULL InternalName NULL SQLSERVR LegalCopyright NULL © Microsoft Corp. All r LegalTrademarks NULL Microsoft® is a registered tradem OriginalFilename NULL SQLSERVR.EXE PrivateBuild NULL NULL SpecialBuild NULL WindowsVersion (3790) ProcessorCount ProcessorActiveMask ff ProcessorType PROCESSOR_INTEL_IA64 PhysicalMemory ( ) Product ID NULL NULL (20 row(s) affected)

Cursor vs. Set Wise The following is based on a 9.5 million row table. 2 CPU. 2G RAM. DECLARE scrub_cursor CURSOR FOR SELECT y FROM table_x order by y OPEN scrub_cursor FETCH NEXT FROM scrub_cursor WHILE = 0 BEGIN … then update table_x

Cursor DECLARE scrub_cursor CURSOR <FAST_FORWARD> FOR SELECT y
FROM table_x <nolock> order by y -- 4:23 minutes -- normal: :48:26.560 :52:49.230 -- 3:50 minutes -- fast forward: :56:16.140 :00:05.780

Set Wise Just as much “work” for SQL Server to crunch 1 row as 10K rows… select getdate() go update table_x set ROW_ID = '0' -- 26 seconds :44:14.263 ( row(s) affected) :44:40.030

Another Strategy Drop all indexes
Let the Index Tuning Wizard tell you what you need. On both the EIM_* and Base Tables Use SQL Server Profiler to find scans (ex. > Reads). Add those indexes back on. Use SP_INDEXOPTION and enable row level ONLY locking (disable page level locking). This facilitates scaling through multiple parallel jobs. Explain how IF_BATCH_NUM impacts parallel jobs. Know your data!

Log Shipping Log Shipping Jobs How to setup
Failing over and syncing users

SQL Server & Siebel In About One Day

Similar presentations

Presentation on theme: "SQL Server & Siebel In About One Day"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

SQL Server & Siebel In About One Day

Similar presentations

Presentation on theme: "SQL Server & Siebel In About One Day"— Presentation transcript:

Similar presentations

About project

Feedback