Performance SeNTry: Development Status Report Demand Technology Software A DataCore Software company 1020 Eighth Avenue South, Suite 6, Naples, FL 34102 phone: (941) 261-8945 fax: (941) 261-5456 e-mail:support@demandtech.com http://www.demandtech.com Demand Technology, Inc.

Agenda Performance SeNTry version 2.4.2 highlights Version 3 roadmap NTSMF Collector SeNTry Administration Version 3 roadmap Alerts and Traps File System Collector Improved Systems Console for real-time notification, management, and control Demand Technology, Inc.

Performance SeNTry version 2.4.2 Topics Windows 2000 ‘features’ NTSMF features SeNTry Administration features Event log use New Enhancements – SQL Server 2000 New Enhancements - Exchange 2000 support New collection service architecture – Threaded collector Improved Exception handling

Performance SeNTry version 2.4.2 Windows 2000 ‘features’ System level objects now in visible performance libraries System object ‘missing’ some NT4 counters: %Total Processor Time, % Total User Time, % Total Privileged Time, % Total DPC Time, % Total Interrupt Time, Total Interrupts/sec Delay in some objects reporting data at system start going through RegQueryValueEx() New values in Performance Registry keys Timeouts DisablePerformanceCounters Discuss backwards compatibility and efforts at getting around these features Demand Technology, Inc.

Performance SeNTry version 2. 4 Performance SeNTry version 2.4.2 New fields in Registry: HKLM\…\Services\PerfDisk\Performance Timeouts are in milliseconds – Collect is 2 secs, Open is 5 secs Demand Technology, Inc.

Performance SeNTry version 2.4.2 Additional NTSMF features Install option to run under a user account -install -account <account name> -password <password> Program defaults with filters active - NT4 and Win2K have specific sets Memory size field in header fixed for Win 2000 systems with 4GB or more of physical memory No fix for Windows NT4 systems with 4GB or more of physical memory since the GlobalMemoryStatus api call returns only a 32 bit number

SeNTry Administration enhancements New default DCS for Win2K and Performance Gallery Gold templates SQL Server 2000 object support MS Exchange 2000 objects support Tool to compact the Performance SeNTry Database ('Tools' menu item in the 'Network Browser' window) More filters More options

SeNTry Administration enhancements Collection Filters - Process New Process Filter on Process Working Set Size Always write named processes – availability reporting Threads Add 'Ready Threads' counter in the 'Process' object No longer need to write Thread objects into .smf Terminal Server suppress Idle User record write User and Session records (automatic) only when activity occurs Demand Technology, Inc.

Performance SeNTry version 2.4.2 Collection filters, Thread Top filter triggers writing the ‘Ready Threads’ ntsmf counter that removes need to write any thread records Demand Technology, Inc.

SeNTry Administration enhancements Parameters Administration, Collection Cycle On Error Command capability for launching commands when errors are encountered Same rules as Cycle End commands ‘poor man’s’ alert – something as simple as ‘Net Send user-name message’

SeNTry Administration enhancements Parameters Administration, Scheduling Flexible DCS chaining specifying Next DCS in chain (e.g., prime shift.DCS chains to nighttimeDCS, which chains back) Collection Period to specify the period that the collector is active and reporting data Provide basic scheduling options without requiring a 3rd party package

Performance SeNTry version 2. 4 Performance SeNTry version 2.4.2 Parameters Administration, File Contents Support for summarization program Collect Inventory data from Registry keys New option – “Write denominator counters” Demand Technology, Inc.

Performance SeNTry version 2. 4 Performance SeNTry version 2.4.2 Parameters Administration, File Contents New option – “Write denominator counters” Used by .smf Summarization (DMSumSMF.exe) uses in calculations to compute correct average values of compound Counters e.g., Copy Read Hits % One more option to manage the volume of data that can be collected effectively “Base” appended to denominator counter name strings for identification in .smf file

Performance SeNTry version 2. 4 Performance SeNTry version 2.4.2 Parameters Administration, File Contents Summarization utility (based on Counter Type) Requires Type 6 records! Designed to run at Cycle End, but can be run standalone Creates sum.smf files Identical in format to detail records aged like any other .smf file in Current/Previous/Archive folders Calculates the arithmetic mean for most Counters Calculates interval peak (min, max) where appropriate Some counters simply save the last observed value (e.g. elapsed time)

SeNTry Administration enhancements Parameters Administration, File Contents ‘Include Registry Key Values’ to get registry key value information into configuration records in the SMF data file. Used to collect hardware and software inventory, versioning, etc. No restrictions: inventory as many values as you need Must specify the full Registry key name and field name Use in conjunction with regedit Copy Key Name function Key Not Found earns a Warning message, but no other side effect Invalid spec earns a Warning message, but no other side effect

SeNTry Administration enhancements Parameters Administration, File Management PKZIP compression option at Cycle End Another way to manage data volume

Performance SeNTry version 2.4.2 Event Log messages Informational, Warning and Error messages are placed in the Application Event log Also in <system>.ntsmf.log (plain text) Date, time User, System, source filled in by event facility Type, category, event id, description defined by developers I and W will not Point out that ntsmf.log is most valuable, and we size and age according to parameters Demand Technology, Inc.

Performance SeNTry version 2.4.2 Event Log messages - Warning Warning is something that Performance SeNTry detects that does not impact ability to collect critical counters. Demand Technology, Inc.

Performance SeNTry version 2. 4. 2 Event Log messages – ntsmf Performance SeNTry version 2.4.2 Event Log messages – ntsmf.log example 10/24/01-00:00:01 - Event ID: 210, Category: Discovery, Severity: Warning The Performance Library, "C:\Program Files\Exchsrvr\bin\exprfdll.dll", for service "ExIPC" (ExIPC), failed to load. The system error was "The specified module could not be found (126)". Performance data for this service will not be available. The Event Viewer (source Perflib) may have more details Date, time, category, severity as in an Event Log entry. Text (description in an Event Log example) comes from GetlLastError() and FormatMEssage() api calls Final line – the Perflib can and sometimes does put out information – but not always. Demand Technology, Inc.

Performance SeNTry version 2. 4 Performance SeNTry version 2.4.2 Event Log messages – unhandled exception 10/13/01-00:00:02 - Event ID: 96, Category: N/A, Severity: Error Unhandled exception encountered 10/13/01-00:00:02 - Ended Performance SeNTry service 10/13/01-00:00:02 - Event ID: 104, Category: Initialization, Severity: Info The Performance SeNTry service stopped with error code, -1073741728. Version=2.3.6 The stop error code is a result of a formatted 32 bit value. The severity, couple of flags and category are in the high order 16 bits. An assigned code is in the low order 16. The method implemented in Event log handling makes these difficult to deal with in automated systems. The event id, severity and text are available in both log forms to Event scrapers. Demand Technology, Inc.

Performance SeNTry version 2.4.2 New Enhancements – SQL Server 2000 Lots of new SQL Server 2000 records A single machine can be run multiple instances of SQL Server No instance names for separate SQL Server processes, as delivered by MS MSSQL$db-name:object-name e.g. MSSQL$SUP:Buffer Manager,… MSSQL$SUP1:Buffer Manager,…) Anticipate object selection and reporting problems We write a MSSQL:Buffer Manager Object instance where the db-name token becomes an Instance name 2.2.2,0,5,2001,4,17,20,36,45,859,0,MSSQL:Buffer Manager,1,18,22,1,SQL Server DB,Buffer cache hit ratio,… Instances definition problem - Demand Technology, Inc.

Performance SeNTry version 2. 4 Performance SeNTry version 2.4.2 New Enhancements - Exchange 2000 support New objects visible when optional components installed (e.g. Instant Messaging) Cases where a perflib loads another .dll exprfdll.dll loads exchmem.dll Objects have many counters: MSExchange Web Mail … 75 counters MSExchangeIMAP4 … 96 counters MSExchangeIS ……… 107 counters (actual) Database …………….. 127 Counters (many of ‘no data’ type) Discuss the development effort a bit, comment on the number of counters and point out that we/they have to figure out what Exchange 2000 stuff matters – I vaguely remember seeing a reference, can’t recall if it was an article in NT magazine or an MSDN thing on what to measure – should find a reference if we can before CMG Demand Technology, Inc.

Performance SeNTry version 2.4.2 New Architecture - Threaded collector Separate thread for each performance library found in the Registry 3rd party Perflib DLLs perform the actual performance data gathering Windows 2000, XP expose the ‘system’ perflibs, including perfos.dll, perfdisk.dll, perfproc.dll, perfnet.dll Improved error handling, especially in 3rd party code Additional collection service threads manage dispatching, file writes, other tasks Demand Technology, Inc.

NT Performance monitoring API Perflib Dlls are loaded into the dmperfss process address space when opened 3rd party Perflib Dlls might need to synchronize with a separate monitored address space

NT Performance monitoring API Open and Collect Timeout values exceeded trigger DisablePerformanceCounters setting, which must be disabled manually! HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Perflib ExtCounterTestLevel can be used to verify integrity of 3rd party Perflib dlls.

Performance SeNTry version 2.4.2 New Architecture - Exception handling Try to catch exceptions that occur in a thread, i.e., inside code in a Perflib DLL The failing thread and perflib (and the performance Objects it supplies) are deactivated, while other Objects continue to be collected Problem perflib is automatically retried every new cycle or on demand (Notify Command) No manual intervention required Unless the Object is on the critical list: System, Memory, Cache, Process, Thread, Processor Critical missing means log and stop Explain why we thought to stop is better – data will be meaningless without ‘system’ stuff, also want to get User’s attention asap to address the system problem Demand Technology, Inc.

Performance SeNTry version 2.4.2 New Architecture - Exception handling On exception: system error text message the loaded modules with addresses and the call stack Helps to identify ‘rogue Perflib dlls’ Very compact and accurate

Performance SeNTry version 2.4.2 Log message and module list 10/26/01-10:20:05 - Event ID: 1300, Category: PerfData, Severity: Warning Could not collect performance data for object, "Database ==> Instances (L28698, G3370), counter, Pages Converted/sec (L28700, G3372)". The system error was "Win32 exception, 0xC0000005, encountered at location 0x0043FBB1 Read attempt at location 0x00000020 Module List: (0x00400000-0x00480000) D:\NTSMFv24224\DMPerfss.exe (0x01000000-0x01017000) F:\WINNT\System32\BEPerfDll.dll (0x01140000-0x0114E000) D:\Program Files\Exchsrvr\bin\davexpc.dll (0x01630000-0x0163B000) D:\Program Files\Exchsrvr\bin\eseperf.dll (0x02000000-0x02010000) F:\WINNT\system32\dssdata.dll (0x02C40000-0x02C46000) D:\Program Files\Exchsrvr\bin\srsperf.dll …. (lots more of these) Note that this was induced during testing – maybe find and example where it occurs I a perflib – Dave has test lib to work with. Point out content of .dll entries. Don’t talk too long to this – make the point and move on Demand Technology, Inc.

Performance SeNTry version 2.4.2 Stack Trace Stack Trace (ThreadId 4540): DMPerfss.exe: PSSPerfLocalCounter::RealizeCache + 111 bytes E:\DmPerfss 2.4.2\Collector\PSSPerfLocalCounter.cpp, Line 372 + 15 bytes DMPerfss.exe: PSSPerfObjectCache::RealizeCache + 821 bytes E:\DmPerfss 2.4.2\Collector\PSSPerfObjectCache.cpp, Line 247 + 11 bytes DMPerfss.exe: PSSPerfObjectCache::OnCollectionComplete + 12 bytes E:\DmPerfss 2.4.2\Collector\PSSPerfObjectCache.cpp, Line 408 + 9 bytes DMPerfss.exe: PSSPerfDataSink::Collect + 597 bytes E:\DmPerfss 2.4.2\Collector\PSSPerfDataSink.cpp, Line 395 + 14 bytes DMPerfss.exe: PSSPerfDataSink::Collect + 15 bytes E:\DmPerfss 2.4.2\Collector\PSSPerfDataSink.cpp, Line 463 + 15 bytes DMPerfss.exe: PSSPerfObject::_Collect + 75 bytes E:\DmPerfss 2.4.2\Collector\PSSPerfObject.cpp, Line 1688 + 26 bytes DMPerfss.exe: PSSPerfObject::Collect + 79 bytes E:\DmPerfss 2.4.2\Collector\PSSPerfObject.cpp, Line 1748 + 20 bytes Explain that the symbols give us this trace back of the code – this is why we need symbols! Demand Technology, Inc.

Performance SeNTry version 2.4.2 Stack Trace DMPerfss.exe: PSSCollectionCycle::Start + 1483 bytes E:\DmPerfss 2.4.2\Collector\PSSCollectionCycle.cpp, Line 1500 + 45 bytes DMPerfss.exe: PSSPerformanceSeNTry::StartCollection + 248 bytes E:\DmPerfss 2.4.2\Collector\PSSPerformanceSeNTry.cpp, Line 1571 + 11 bytes DMPerfss.exe: PSSPerformanceSeNTry::Run + 109 bytes E:\DmPerfss 2.4.2\Collector\PSSPerformanceSeNTry.cpp, Line 1254 + 22 bytes"

Performance SeNTry version 2.4.2 New Architecture - Exception handling New dmperfss.exe ships with debug symbols included for extended error diagnostics report Size of the image file is now approximately 5.8 MB The larger image file installs by default A dmperfss.exe without symbols is also on the CD E.g., a floppy disk distribution/install dmperfss.exe with symbols is preferred, especially if new applications or new versions are discovered

Performance SeNTry Version 3 BREAK

Performance SeNTry Version 3 Roadmap Real-time Alerts Event log & SNMP support File System Collector Revamped Systems Console These aren’t complete yet, need to review design and discuss with Dave and Phil before filling out rest of slides Demand Technology, Inc.

Performance SeNTry Version 3 Alerts Alert is written into the Application event log Triggered by system condition (e.g. cpu busy threshold) Frequency of alerts is also thresholded; won’t fill up the Event Log unnecessarily Trigger, Release mechanism Alerts are cleared when condition is absent (based on the ‘lower threshold’)

Performance SeNTry Version 3 SNMP - Traps Like Alerts, defined by same rules SNMP extension agent notified of the ‘alert’ by dmperfss.exe SNMP extension agent generates a Trap message Trap is sent to management entities by SNMP facility Management entities would be OpenView, Tivoli, CA-Unicenter, others Add Demand Technology, Inc.

Performance SeNTry Version 3 File System Collector At prototype stage now… Probably another thread in the collector Designed to run infrequently, say once/day (don’t know how long it takes yet) Report files that are above threshold (e.g. file size > 2% Logical disk size) Report folders that are above threshold when files in them are totaled (don’t also report parents, just siblings) Demand Technology, Inc.

Performance SeNTry Version 3 File System Collector Summary records – size of files remaining NTSMF header record format Pseudo object id Records are instances where the UNC path is the instance name Probably optioned - allow include in .SMF file or write to a file of its own

Performance SeNTry Version 3 System Console Processes real-time alerts Can examine objects and counters related to the alerts Can change the alert definition, collection interval, etc., as needed for the display Display results graphically Capture the data and graphs for later review Demand Technology, Inc.

Performance SeNTry Version 3 Exchange 2000 references http://www.exchange-mail.org/whitepapers.html http://www.microsoft.com/technet/treeview/default.asp?url=/technet/prodtechnol/exchange/reskit/ex00res/resguide/c29monit.asp (Microsoft's online version of the Exchange Resource Kit Documentation) http://www.netiq.com/Downloads/Library/White_Papers/NetIQ_Monitoring_and_Managing_Exchange.pdf

Performance SeNTry reach us at: Demand Technology Software A DataCore Software company Address: 1020 Eighth Avenue South, Suite 6, Naples, FL 34102 Phone: (941) 261-8945 fax: (941) 261-5456 E-mail: support@demandtech.com Web site: http://www.demandtech.com

Performance SeNTry Version 3 Roadmap Questions?