Practical Use of MDA Tables

Practical Use of MDA Tables
A Case-Study in the Practical Use of MDA Tables from Sybase's Internal IT/DBA Team David Burgess Sybase Staff DBA

Agenda DBA Nightmares ASE Performance and Tuning Problems
The ASE is slow, fix it! How MDA Tables can help How MDA Collector can help Performance and Tuning with MDA Collector MDA Solutions: Practical applications of MDA Tables and Collector Common Problems with MDA Tables Q&A

Presentation Goal: Fixing DBA Nightmares
Black Box ASE Black Box Applications connecting to the ASE Hung query Hung ASE Query hogging 100% CPU Query crashing ASE Query resulting in massive blocking Problem logins running problem SQL Runaway SQL consuming log

MDA Quick Review MDA tables were introduced in ASE 12.5.0.3
MDA = Monitoring and Diagnostic Access Also called “monitoring tables” 41 tables installed ‘master’ database (15.0.2) Table names begin with “mon” (e.g., monProcessActivity) Can be accessed with standard SQL commands Table data is generated from in-memory structures No data is stored in master database Tables are proxy tables No license needed: included in ASE base product MDA tables provide access to detailed monitoring data Process Activity CPU usage, IO activity, resource usage Resource usage Data cache, procedure cache, engines Object usage Tables, partitions, indexes, stored procedures Query history SQL text, statement metrics, query plans, errors

How MDA Tables can help Identify, track and fix:
Root cause of many ASE P&T issues Bad SQL that impacts performance Rogue processes hogging resources (CPU, IO, etc.) Bad stored procedures Bad indexes Problem applications and logins Unused objects (tables, indexes) MDA Tables provide a lot of data Up until recently, ASE did not provide many ready-to-run sprocs to sort that data into information sysqueryplans dbo.sp_mda dbo.sp_mda_help dbo.sp_mda_io dbo.sp_mda_wait

How MDA Collector can help
Why Setup MDA Collector? Collector tables can be used as ASE baseline Problems always happen when DBA is away from desk; Collector is always working Collector should be setup to gather enough info to catch the problem when it happens Use the Collector reports to review errors Review last few “collections” just before ASE problem Archive Collector reports for root cause analysis

How MDA Collector can help
MDA Collector Reports MDA Collector has two dozen ready-to-run reports to help ‘translate’ MDA data into information Use the Collector reports to review errors Review last few “collections” just before ASE crash or other error Archive Collector reports for root cause analysis Run Collector reports to generate a new baseline New Baseline required after any change Compare new baseline with old baseline to measure performance improvements

How MDA Collector works
Can be configured to collect on three options Application SPID running longer than 15 seconds System info every 60 seconds: Engine Counters, Device, syslogs, contention... Historical SQL, Errorlog, Deadlocks... every 15 seconds Can be set to collect ‘snapshot’ for any frequency May set to archive for any number of days Example: Prune everything older than 7 days Techwave 2006 presentation on MDA collector reference:

How to setup MDA Collector
Enable MDA tables and grant ‘mon_role’ permission Create Collector database Install ASE sprocs and schema from zip-files Copy start_scripts from zip-files onto ASE host Run start_script and purge_script

How to startup MDA Collector
START_MDA to call all 3 'start' sprocs Makes 3 user connections to ASE as the mda_collect user dbadb_metrics..sp_dba_MON_SYSTEM_control 'start‘ dbadb_metrics..sp_dba_MON_APP_control 'start' dbadb_metrics..sp_dba_MON_HIST_control 'start' Those sprocs then wait in infinite loop until 'stop‘ A snapshot is collected at every specified control_table interval How to make it work for your version Edit control table : …to collect what? …and how often? Edit Collector sprocs to add additional “filters” sp_dba_MON_SYSTEM_col_v1252 sp_dba_MON_APP_col_v1252 sp_dba_MON_HIST_col_v1252 WHERE ml.SPID = mp.SPID AND ml.KPID = mp.KPID AND mp.SPID --AND mp.DBID <> db_id("dbadb_metrics") --AND mp.DBID <> db_id("sybsystemprocs") --AND mp.Login <> "mda_collect"

How to use MDA Collector
Example Collector Reports: HIST & APP show minutes, SYS is hour HIST PROCS SYS PROCS APP PROCS dbasp_MONR_HIST_DeadLock Captures all deadlocks and can go back in history dbasp_MONR_HIST_ErrorLog All errors caused by SPIDS. dbasp_MONR_HIST_SQLSmt All statements and associated CPU, Read, Writes, Network activity etc. dbasp_MONR_HIST_SQLText Recreate all the SQL executed, can be limited to a particular SPID dbasp_MONR_SYS_Engine_Stats Collected every 180 Seconds, allows us to identify peak times at a granular level. dbasp_MONR_SYS_OpenDB_Stats DB activity, logs contention .. tune syslogs and tempdb, could be used to justify multiple tempdbs. dbasp_MONR_SYS_SysWaits_Stats Context Switches,. Allows us to identify what kind of resources are limiting the ASE. dbasp_MONR_SYS_CachedObj_Stats Used to identity if a user defined cached is being used or sized right ? dbatb_MON_App_Locks Similar to lock history, will identify which spid are holding locks on which objects during the snapshot. dbatb_MON_App_ProcessActivity Will identify the source of the SPID (IP Address) dbasp_MONR_APP_Process_Waits Shows what the SPID waiting for dbasp_MONR_APP_Process_Smt Information on Statements executing, e.g. CPU, Reads/Writes, source of statement etc., Network activity. dbasp_MONR_APP_Process_SQLText Catches all SQL text, for queries running longer that interval (15 seconds currently) dbasp_MONR_APP_Process_Procs Procedures running longer than 15 secs, and currently executing during the snapshot. Other Options to view MDA Tables sp_monitor select mon_tables into temp_tables and bcp out Additional options in ASE 15.x: sysquerymetrics , sysqueryplans, QPTune DB XRay DB Expert Improving Server Performance Index Advisor Optimize SQL Statement SQL Inspector and collecting SQL performance stats

MDA Collector : Tips & Tricks
Consolidating MDA Collector START/STOP scripts to one central host Consolidating MDA Collector data to one central Reporting ASE Set RPC or loopback to the reporting ASE Run Collector reports on Reporting rather than OLTP Future Direction of DBA team for MDA Collector

ASE Performance and Tuning: ASE is slow, fix it!
DBA Solutions : Slower than what? Compare against a baseline Any recent system changes : upgrades, new hardware Identify the problem: system resource …or… sql and query plan Rogue transaction hogging resources or blocking users Users blocked on tempdb Make one change at a time and compare against previous baseline

P & T with MDA Collector : Where to Start?
Gather baseline of ‘normal’ performance Advise smaller intervals of about 15 – 20 minutes Get baseline for peak times and non-peak times Keep archive of baseline samples dbasp_MONR_SYS_SysWaits_Stats 1 WaitEventID SUM_WaitTime SUM_Waits AVG_WaitTime AVG_Waits Description waiting for incoming network data waiting while no network read or write is required waiting for date or time in waitfor command hk: pause for some time xact coord: pause during idle loop checkpoint process idle loop wait until an engine has been offlined waiting while allocating new client socket waiting on run queue after yield waiting on run queue after sleep waiting for network send to complete waiting for last i/o on MASS to complete waiting for regular buffer read to complete waiting for message in worker thread mailbox waiting for buf write to complete before writing wait to acquire latch wait for i/o to finish after writing last log page waiting for a lock waiting for MASS to finish writing before changing waiting for i/o on MASS initated by another task wait for mass read to finish when getting page waiting for a pipe buffer to read

P & T with MDA Collector : Where to Start?
Reading that Baseline Start with monProcessWaits/monSysWaits Info on whether the next step is query related, client software, hardware or contention in ASE If you know this to be SQL query related, you may be able to skip monProcessWaits and go directly to monProcessActivity/ monProcessStatement/monSysStatement Most closely approximates sp_sysmon context switching section …but gives you the details not available with sysmon …and lets you focus down to the process detail level Unfortunately, the “WaitEvents” require explanation (see later slide) Comparing Baselines Compare Apples-to-Apples : Remember to compare an equivalent timeframe Sample interval, peak periods vs. peak periods

P & T with MDA Collector : What to look for?
Focus on the "Waits" Log, Tempdb, data IO, WaitEvents Use MS Excel or OpenOffice to plot Requests vs. Waits Look at monOpenObjectActivity for explanation Sort report output Sort by : CPUTime, WaitTime, PhysicalReads, LogicalReads, PagesRead, PhysicalWrites, PagesWritten, MemUsageKB Remember that SPIDs report total of every and any sql every executed by that spid. If same spid logged in for weeks, then those report numbers will be very high

P & T with MDA Collector : Good wait events
dbasp_MONR_SYS_SysWaits_Stats 1 WaitEventID SUM_WaitTime SUM_Waits AVG_WaitTime AVG_Waits Description waiting for incoming network data waiting while no network read or write is required waiting for date or time in waitfor command hk: pause for some time xact coord: pause during idle loop checkpoint process idle loop wait until an engine has been offlined waiting while allocating new client socket waiting on run queue after yield waiting on run queue after sleep waiting for network send to complete waiting for last i/o on MASS to complete waiting for regular buffer read to complete waiting for message in worker thread mailbox waiting for buf write to complete before writing wait to acquire latch wait for i/o to finish after writing last log page waiting for a lock waiting for MASS to finish writing before changing waiting for i/o on MASS initated by another task wait for mass read to finish when getting page waiting for a pipe buffer to read waiting for write of the last log page to complete waiting for Lava pipe buffer for write waiting for buffer validation to complete wait for MASS to finish changing before changing waiting for MASS to finish changing to start i/o wait for DES state is changing wait for flusher to queue full DFLPIECE waiting for disk buffer manager i/o to complete waiting for incoming network data - Equivalent to ‘awaiting command’ xact coord: pause during idle loop waiting while no network read or write is required waiting for date or time in waitfor command hk: pause for some time checkpoint process idle loop wait until an engine has been offlined waiting while allocating new client socket replication agent sleeping during flush

P & T with MDA Collector : More wait events to watch for…
dbasp_MONR_SYS_SysWaits_Stats 1 WaitEventID SUM_WaitTime SUM_Waits AVG_WaitTime AVG_Waits Description … waiting for last i/o on MASS to complete waiting for regular buffer read to complete waiting for message in worker thread mailbox waiting for buf write to complete before writing wait to acquire latch wait for i/o to finish after writing last log page waiting for a lock waiting for MASS to finish writing before changing waiting for i/o on MASS initated by another task wait for mass read to finish when getting page waiting for a pipe buffer to read waiting for write of the last log page to complete waiting for Lava pipe buffer for write waiting for buffer validation to complete wait for MASS to finish changing before changing waiting for MASS to finish changing to start i/o wait for DES state is changing wait for flusher to queue full DFLPIECE waiting for disk buffer manager i/o to complete waiting for synchronous disk buffer manager i/o wait for data from client pause to synchronise with site manager wait for object to be returned to pool wait for message waiting for read to complete in parallel dbcc waiting for page reads in parallel dbcc waiting for disk read in parallel dbcc waiting to re-read page in parallel dbcc waiting on MASS_READING bit in parallel dbcc waiting on TPT lock in parallel dbcc waiting sending fault msg to parent in PLL dbcc waiting for site handler to complete setup waiting to be scheduled (cpu) waiting for a disk read to complete (read) waiting for a disk write to complete (write) waiting to acquire the log semaphore (log contention) waiting to take a lock (lock contention) waiting for memory or a buffer (address contention) waiting for input from the network (client speed) waiting to output to the network (client fetch/net sat) waiting for internal system event (PLC, index balance) waiting on another thread (contention) waiting for semaphore (ASE is waiting for a logical lock) waiting for CTLIB event to complete (Client Related S/W Issues: non-data related: i.e. waiting for TDS tokens such as ACK for packets sent, or waiting on next command to be sent (i.e. gap between ct_command() and ct_send())… if CIS is involved, it is waiting on ct_fetch()/result set materialization at remote server Next move is to look at the client code waiting for network send to complete Client Related S/W Issues: This is data stream related – outbound commands (RPC’s, RepAgent, etc.) will be ‘waiting for CTLIB event to complete’ due to waiting for ct_sendpassthru(), etc. to execute. ...Next table to check out is monProcessNetIO – probably going to be a change to fetch block size in program and/or packet size wait in bufwrite for mass to finish changing before writing buffer wait for mass write to complete before setting change flag wait for mass to finish changing before setting change flag waiting in writedes for mass to finish changing before writing buffer wait in DBCC delbuf for mass to finish changing before removing buffer Write was restarted because previous attempt failed – if you see this check sys error log waiting for last MASS on which i/o was issued waiting for last MASS on which i/o was issued by some other task waiting to write of the last page of the log waiting after write of the last page of the log waiting until last chance threshold is cleared Transaction Log Delays:: Transaction log keeps filling and crossing the lct – you need to add a threshold to dump earlier, or make the log bigger ...Something to watch if tempdb is filling waiting for semaphore Transaction Log Delays:: Check monOpenDatabases and compare appendLogRequests to appendLogWaits Disk I/O wait events 54 & 55 you are waiting to write to the last log page you are waiting for the last log page you wrote to flush ..You don’t commit until page is flushed to disk Wait to acquire latch Address locking contention (tran log) DOL index contention (last index page – ASE 15 partition table/local index) Waiting for semaphore Typically normal row/pg lock, but could be log semaphore or spinlock contention Wait for someone else to finish reading in mass Memory access contention May show up with Wait Event 52 – "waiting for last MASS on which i/o was issued by some other task" Possible causes: Tempdb in same data cache as primary tables user does select/into (bulk I/O) The last mass in use will be appended to with the new logical pages being written But the previous user is still reading the previous pages Most likely cause – two nearly concurrent select/into's in tempdb See above progession – think about it – select/into tempdb and then you immediately read out Next task has to wait to access memory Most Likely Answer: multiple tempdb's waiting on run queue after yield H/W Issues: CPU contention : Task reached timeslice - No I/O wait, so task is cpu-intensive in memory scan, join operations, sorting, looping logic in proc, etc. waiting on run queue after sleep Could also indicate high write activity i.e. BCP, or other write intensive process will sleep while waiting I/O… Remember, log writes also mean SPID sleeps – Slow cpu's could result in higher waits on log semaphore and disk writes 54 & 55 Either one could be due to a cpu pig next step is to look at monProcessActivity.CPUtime If no obvious cpu hogs, you may need to add cpu's/online additional engines wait for buffer read to complete H/W issues: Device I/O related: Logical read or network read wait for buffer write to complete H/W issues: Device I/O related: Logical write (update in cache before disk flush)/network send waiting for disk write to complete H/W issues: Device I/O related: Exceeded disk i/o structures and delayed for pending i/o queue??? waiting while no network read or write is required Netserver checked and no network read/write pending Server level – shouldn’t see this in monProcessWaits Check "i/o polling process count" If CPU & IO bound – reduce "i/o polling process count" For – look at the following in monEngine: DiskIOChecks, DiskIOPolled, DiskIOCompleted

P & T with MDA Collector : Wait events to watch for…
Wait event descriptions Where to look “waiting to be scheduled” (cpu) “waiting for a disk read to complete” (read) “waiting for a disk write to complete” (write) “waiting to acquire the log semaphore” (log contention) “waiting to take a lock” (lock contention) “waiting for memory or a buffer” (address contention) “waiting for input from the network” (client speed) “waiting to output to the network” (client fetch/net stat) “waiting for internal system event” (PLC, index balance) “waiting on another thread” (contention)

P & T with MDA Collector : Wait events to watch for…
Client Related S/W Issues 171 : Waiting for CTLIB event to complete -waiting on next Client command to be sent 251 : Waiting for network send to complete -waiting for ct_sendpassthru(), 250 : Waiting for incoming network data -Equivalent to ‘awaiting command’ Transaction Log Delays 259 : Waiting until last chance threshold is cleared -Transaction log keeps filling and crossing the LCT 150 : Waiting for semaphore 54 : waiting for write of the last log page to complete -you are waiting to write to the last log page 55 : wait for i/o to finish after writing last log page -you are waiting for the last log page you wrote to flush Contention 41 : Wait to acquire latch -Address locking contention (tran log) 70 : Waiting for semaphore -Typically normal row/pg lock, but could be log semaphore or spinlock contention 52 : "waiting for last MASS on which i/o was issued by some other task" -Two nearly concurrent select/into's in tempdb H/W Issues: CPU contention 214 : Waiting on run queue after yield -in memory scan, join operations, sorting, looping logic in proc, etc. 215 : Waiting on run queue after sleep -Slow cpu's could result in higher waits on log semaphore and disk writes 33 : Wait for buffer read to complete -Logical read or network read 34 : Wait for buffer write to complete -Logical write (update in cache before disk flush)/network send 179 : waiting while no network read or write is required -Netserver checked and no network read/write pending Waiting for disk write to complete -Exceeded disk i/o structures and delayed for pending i/o queue Client Related S/W Issues 171 : Waiting for CTLIB event to complete -look at the client code 251 : Waiting for network send to complete -monProcessNetIO –change to fetch block size and/or packet size 250 : Waiting for incoming network data -try (ct_dynamic) or BLOB processing Transaction Log Delays 259 : Waiting until last chance threshold is cleared - add a threshold to dump earlier, or make the log bigger 150 : Waiting for semaphore -monOpenDatabases and compare appendLogRequests to appendLogWaits 54 : waiting for write of the last log page to complete - You don’t commit until page is flushed to disk 55 : wait for i/o to finish after writing last log page -You don’t commit until page is flushed to disk Contention 41 : Wait to acquire latch -Address locking contention (tran log) 70 : Waiting for semaphore -Typically normal row/pg lock, but could be log semaphore or spinlock contention 52 : "waiting for last MASS on which i/o was issued by some other task" -Memory access contention H/W Issues: CPU contention 214 : Waiting on run queue after yield -Task reached timeslice - No I/O wait, so task is cpu-intensive 215 : Waiting on run queue after sleep -Could also indicate high write activity 33 : Wait for buffer read to complete -Logical read or network read 34 : Wait for buffer write to complete -Logical write (update in cache before disk flush)/network send 179 : waiting while no network read or write is required -Netserver checked and no network read/write pending Waiting for disk write to complete -Exceeded disk i/o structures and delayed for pending i/o queue

P & T with MDA Collector : Where to look?
monCachedObject - dbasp_MONR_SYS_CachedObj_Stats Provides statistics for all objects and indexes that currently have pages cached within a data cache monCachePool - dbasp_MONR_SYS_CachePool_Stats Provides statistics for all pools allocated for all caches monOpenObjectActivity - dbasp_MONR_SYS_OpenObj_Stats Provides statistics for all open objects monIOQueue - dbasp_MONR_SYS_IOQueue_Stats Provides device IO statistics broken down into data and log IO, for normal and temporary databases on each device. monDeviceIO - dbasp_MONR_SYS_DeviceIO_Stats Provides statistical information about devices monSysWaits - dbasp_MONR_SYS_SysWaits_Stats Provides a server-wide view of events that processes are waiting for monProcessActivity - dbasp_MONR_APP_Proc_Activity Provides statistics about process activity monSysStatement - dbasp_MONR_HIST_SQLSmt Provides statistics for the most recently executed statements. monCachedProcedures - dbasp_MONR_APP_Process_Procs Provides statistics about all procedures currently stored in procedure cache monEngine - dbasp_MONR_SYS_Engine_Stats Provides statistics regarding ASE engines monProcedureCacheModuleUsage Provides procedure cache usage statistics by ASE modules

P & T with MDA Collector : What next?
Step 1: Gather current statement statistics Check monProcessStatement : dbasp_MONR_APP_Process_Smt Check monProcessSQLtext : dbasp_MONR_APP_Process_SQLText Step 2: Get SPID Resource Consumption Check monProcessActivity : dbasp_MONR_APP_Proc_Activity Step 3: If High Wait Time – Find cause Check monProcessWaits : dbasp_MONR_APP_Process_Waits Step 4: If High I/O Write waits or Tempdb is suspect Check monProcessObject : dbasp_MONR_APP_Process_Object Check monOpenObjectActivity : dbasp_MONR_SYS_OpenObj_Stats Step 5: If Contention Check monOpenObjectActivity to find table(s) with most contention (LockWaits) Check monProcess for Blocking : dbasp_MONR_APP_Proc_Activity Check monLocks, monDeadLocks : dbasp_MONR_APP_Lock_History Step 6: If Proc (somewhere in proc is slow) monProcessStatement only gives metrics on current statement within the current batch/context/line monSysStatement – historical view of the query tree : dbasp_MONR_HIST_SQLSmt

MDA Collector Solutions Case Study 1: Tempdb full, ASE sprocs hang
<SERVER.master.1> select * from master..monProcessSQLText <SERVER.master.2> go SPID KPID ServerUserID BatchID LineNumber SequenceInLine SQLText select bug_id, ( select min(a.action_ts) from sy_resolution r, sy_resolution_activity a where r.resolution_id = a.resolution_id and a.fieldname = 'res_status' and a.newval = 'Closed' and bug_id = bugs.bug_id ) as report_dt into #bug_closed from bugs where bug_id in ( select sy_addl_case.bug_id from sy_addl_case join clarify..table_case on left(sy_customer_case,8)=id_number join bugs b on b.bug_id = sy_addl_case.bug_id where sy_customer_case is not null and sy_customer_case <> '10001' and active_flag <> 2 and cure_code = '1' and b.sy_product_family in ( 'SQL Server','Docs: SQL Server') and b.bug_type in ('Regression', 'Defect') and b.bug_severity in (0, 1) )

MDA Collector Solutions Case Study 2: ASE stacktrace and crash
ASE stacktrace and die on spid 287 Use Collector to find out what was spid 287 doing ? Turns out that the sp_configure 'allow nested triggers' option was enabled on ASE. This results in a trigger + sproc infinite loop = stacktrace and crash. 00:00000:00287:2008/06/13 14:19:03.61 kernel Stack overflow detected: limit: 0x , sp: 0x 00:00000:00287:2008/06/13 14:19:03.62 kernel pc: 0x f84ea0 pcstkwalk+0x24(0x , 0x , 0x f, 0x , 0x ) 00:00000:00287:2008/06/13 14:19:03.62 kernel pc: 0x f84d04 ucstkgentrace+0x1c0(0x be2ff18, 0x , 0x f, 0x , 0x ) dbasp_MONR_HIST_SQLText , null, null , 287 KeyCounter SampleTime SPID KPID BatchID SequenceInBatch SQLText /13/08 2:18 PM select * from x_debug_on /13/08 2:18 PM select name from master.dbo.sysdatabases /13/08 2:18 PM UPDATE table_subcase SET x_eng_prty_rating = "40"\t\t WHERE id_number

MDA Collector Solutions Case Study 3: Bad Sprocs
Use MDA to find problem sprocs dbasp_MONR_APP_Process_Object dbasp_MONR_APP_Process_Procs Use MDA to find stored Procedure Performance Averages Use Collector to find sproc averages over historical timeline… SPID KPID Login Application Command MasterTranID ObjectID OBJ_NAME IndexID LogicalReads tools_user tools UPDATE $upd [NULL] 0 0 tools_user tools UPDATE $upd [NULL] 0 1 tools_user tools UPDATE $upd work_nav_names tools_user tools UPDATE $upd work_nav_trees SPID KPID Login Application Command MasterTranID OwnerUID ObjectID OBJ_NAME tools_user tools UPDATE $upd p_tools_get_authorized_path

MDA Collector Solutions Case Study 4: Bad Cache
Use MDA to show key columns in monStatementCache TotalSizeKB, NumStatements, NumSearches, HitCount, NumRemovals Use Collector to show cache stats over time… dbasp_MONR_SYS_DataCache_Stats 1 , null, null, null, 'summary' 1> select * from monStatementCache 2> go TotalSizeKB UsedSizeKB NumStatements NumSearches HitCount NumInserts NumRemovals NumRecompilesSchemaChanges NumRecompilesPlanFlushes KeyCounter SampleTime CacheID RelaxedReplacement BufferPools CacheSearches PhysicalReads LogicalReads PhysicalWrites Stalls CachePartitions CacheName Accurate SamplePeriod Hit Ratio 3793 7/7/08 4:10 PM app_cache Y 97 0 3794 7/7/08 4:12 PM app_cache Y 96 0 3795 7/7/08 4:13 PM app_cache Y 93 0 3796 7/7/08 4:15 PM app_cache Y 93 0 3797 7/7/08 4:16 PM app_cache Y 94 0 3798 7/7/08 4:18 PM app_cache Y 93 0 3799 7/7/08 4:19 PM app_cache Y 92 0

MDA Collector Solutions Case Study 5: Baseline for setting resource limits
Use Collector to find “high-watermark” of sql Can then use this information to set realistic resource limits dbasp_MONR_APP_Proc_Activity, dbasp_MONR_APP_Process_Object LogicalReads ~ resource limit io_cost SPID Application ObjectID OwnerUserID LogicalReads 590 isql sp_help_resource_limit name appname rangename rangeid limitid limitvalue enforced action scope NULL isql at all times

MDA Collector Solutions Case Study 6: Bad SQL
Use MDA to find most I/O intensive statement Use Collector to find most I/O intensive statement in last 2 hours… select * into #ts from master..monSysStatement select KPID, BatchID, LineNumber, LogicalReads, Elapsed = datediff(ms, StartTime, EndTime) from #ts where LogicalReads > 100 order by 4 desc KPID BatchID LineNumber LogicalReads Elapsed dbasp_MONR_APP_Proc_Activity 120 , null, null, null, 'ANALYZE'

MDA Collector Solutions Case Study 7: Archiving unused objects
Use MDA to find old unused tables to be archived Unused indexes that be dropped or impacting performance Use Collector to find old unused tables over last few months…. Archive Collector reports for a few Quarters Find Indexes that have not been used since the server was started select DB = convert(char(20), db_name()), TableName = convert(char(20), object_name(i.id, db_id())), IndexName = convert(char(20),i.name), IndID = i.indid from master..monOpenObjectActivity a, sysindexes i where a.ObjectID =* i.id and a.IndexID =* i.indid and (a.UsedCount = 0 or a.UsedCount is NULL) and i.indid > 0 and i.id > No system tables order by 2, 4 asc

MDA Collector Solutions Case Study 8: Hot tables
Use MDA to find Hot Tables and Indexes Use Collector to find tables usage over time… dbasp_MONR_SYS_OpenObj_Stats select db_name = db_name(DBID), table_name = object_name(ObjectID, DBID), IndexID, LogicalReads, PhysicalReads, Operations, LockWaits from master..monOpenObjectActivity order by LogicalReads desc DBName TableName IndexID LogicalReads PhysicalReads Operations qts_db bugs qts_db sy_resolution qts_db e2_cset qts_db x_bugs_nc qts_db sy_product_name qts_db sy_resolution_activity qts_db sn_activity_details qts_db doc_xref

Common Problems with MDA Tables
Overconfiguration of MDA Table options could result in problems in the master database. Keep the MDA Table options realistic Turn on only what you need Another benefit with Collector; smaller pipe config is enough exec sp_configure "sql text pipe max messages",100 exec sp_configure "plan text pipe max messages",100 exec sp_configure "statement pipe max messages",100 exec sp_configure "errorlog pipe max messages",100 exec sp_configure "deadlock pipe max messages",100 exec sp_configure "max SQL text monitored",2048

Common Problems with MDA Tables (Cont.)
Excessive Polling E.g. sampling every second If more than every minute, you'd better have a real good reason Drives cpu & network I/O artificially high Collecting Everything for Everybody Instead of using MDA parameters (especially SPID & KPID) "turn it all on and wait for magic to happen” Joining MDA tables (or subqueries) Accuracy problems if self-joins, subqueries – even normal joins Enabling pipe tables too early Determine that you have a bad query before looking for it

Performance problems with the following options Enable only if requried for specific problem exec sp_configure "plan text pipe active" exec sp_configure "plan text pipe max messages“ exec sp_configure "sql text pipe active" exec sp_configure "sql text pipe max messages” exec sp_configure "statement pipe max messages” exec sp_configure "statement statistics active” exec sp_configure "per object statistics active”

MDA table changes between ASE versions Install script after ASE upgrade Internal Ct-Lib/Cs-Lib error : 'cs_convert: cslib user api layer: external error: Conversion between 8 and 12 datatypes is not supported.‘ Must re-run requried $SYBASE/$SYBASE_ASE/scripts/installmontables ASE , this script is part of 'installmaster' Do NOT run this script with 'sqsh‘. 'sqsh' sees a '$' as the start of a sqsh variable, and this messes up the native RPC names, since these RPC names start with a '$' as well.

Arithmetic overflow for multiple engines while using MDA monEngine table - ContextSwitches field. "select sum(convert(numeric(18,0),ContextSwitches)) from master.dbo.monEngine“ Invalid MDA table data Incorrect DBID Invalid columns : wait time

CR , , : Fixed in ASE Querying the monCachedObject table can cause other processes to timeslice resulting in performance hit. CR : Fixed in "ubo_object_from_slot:Pss Invalid slot id 0" … "current process infected with 11" in module "lock__print_deadlockchain" after ASE encounters a deadlock when "print deadlock information" is set or the "deadlock pipe active" MDA option is in use. CR : Fixed in ESD#5 Enabling the 'per object statistics active' Tables configuration option may cause degradation in query performance.

CR# : Fixed in ESD#3 CR# : Fixed in ESD#2 , ESD#5 Under rare conditions querying the monOpenObjectActivity table may result in error 14108, or stacktrace, or crash Workaround to not collect from monOpenObjectActivity table update dbadb_metrics..dbatb_mon_control set Record = 'N‘ where KeyType = 'openobjectactivity‘ sp_sysmon with MDA Must use sp_sysmon option "noclear" to preserve the counters sp_sysmon "00:01:00", noclear

Learn more about MDA ISUG Rob V’s 2004 Techwave Presentation
MDA Tables in ASE – Tips and Tricks Rob V’s 2004 Techwave Presentation Source for the MDA Collector

Q & A and Collector Demo Latest events from Sybase DBA team…
MDA Collector Demo Examples of configuring Collector to Collect sp_configure parameters dbo.dbatb_mon_control table settings Collector table schema Collector sprocs Example Collector setup in our Dev environment Examples of running Collector Reports

Practical Use of MDA Tables

Similar presentations

Presentation on theme: "Practical Use of MDA Tables"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Practical Use of MDA Tables

Similar presentations

Presentation on theme: "Practical Use of MDA Tables"— Presentation transcript:

Similar presentations

About project

Feedback