Presentation on theme: "Performance SeNTry: Development Status Report"— Presentation transcript:
1Performance SeNTry: Development Status Report Demand Technology SoftwareA DataCore Software company1020 Eighth Avenue South, Suite 6, Naples, FL 34102phone: (941) fax: (941)Demand Technology, Inc.
2Agenda Performance SeNTry version 2.4.2 highlights Version 3 roadmap NTSMF CollectorSeNTry AdministrationVersion 3 roadmapAlerts and TrapsFile System CollectorImproved Systems Console for real-time notification, management, and controlDemand Technology, Inc.
3Performance SeNTry version 2.4.2 Topics Windows 2000 ‘features’NTSMF featuresSeNTry Administration featuresEvent log useNew Enhancements – SQL Server 2000New Enhancements - Exchange 2000 supportNew collection service architecture –Threaded collectorImproved Exception handling
4Performance SeNTry version 2.4.2 Windows 2000 ‘features’ System level objects now in visible performance librariesSystem object ‘missing’ some NT4 counters:%Total Processor Time, % Total User Time, % Total Privileged Time, % Total DPC Time, % Total Interrupt Time, Total Interrupts/secDelay in some objects reporting data at system start going through RegQueryValueEx()New values in Performance Registry keysTimeoutsDisablePerformanceCountersDiscuss backwards compatibility and efforts at getting around these featuresDemand Technology, Inc.
5Performance SeNTry version 2. 4 Performance SeNTry version New fields in Registry: HKLM\…\Services\PerfDisk\PerformanceTimeouts are in milliseconds – Collect is 2 secs, Open is 5 secsDemand Technology, Inc.
6Performance SeNTry version 2.4.2 Additional NTSMF features Install option to run under a user account-install -account <account name> -password <password>Program defaults with filters active - NT4 and Win2K have specific setsMemory size field in header fixed for Win 2000 systems with 4GB or more of physical memoryNo fix for Windows NT4 systems with 4GB or more of physical memory since the GlobalMemoryStatus api call returns only a 32 bit number
7SeNTry Administration enhancements New default DCS for Win2K and Performance Gallery Gold templatesSQL Server 2000 object supportMS Exchange 2000 objects supportTool to compact the Performance SeNTry Database ('Tools' menu item in the 'Network Browser' window)More filtersMore options
8SeNTry Administration enhancements Collection Filters - Process New Process Filter on Process Working Set SizeAlways write named processes – availability reportingThreadsAdd 'Ready Threads' counter in the 'Process' objectNo longer need to write Thread objects into .smfTerminal Serversuppress Idle User recordwrite User and Session records (automatic) only when activity occursDemand Technology, Inc.
9Performance SeNTry version 2.4.2 Collection filters, Thread Top filter triggers writing the ‘Ready Threads’ ntsmf counter that removes need to write any thread recordsDemand Technology, Inc.
10SeNTry Administration enhancements Parameters Administration, Collection Cycle On Error Command capability for launching commands when errors are encounteredSame rules as Cycle End commands‘poor man’s’ alert – something as simple as ‘Net Send user-name message’
11SeNTry Administration enhancements Parameters Administration, Scheduling Flexible DCS chaining specifying Next DCS in chain(e.g., prime shift.DCS chains to nighttimeDCS, which chains back)Collection Period to specify the period that the collector is active and reporting dataProvide basic scheduling options without requiring a 3rd party package
12Performance SeNTry version 2. 4 Performance SeNTry version Parameters Administration, File ContentsSupport for summarization programCollect Inventory data from Registry keysNew option – “Write denominator counters”Demand Technology, Inc.
13Performance SeNTry version 2. 4 Performance SeNTry version Parameters Administration, File ContentsNew option – “Write denominator counters”Used by .smf Summarization (DMSumSMF.exe) uses in calculations to compute correct average values of compound Counterse.g., Copy Read Hits %One more option to manage the volume of data that can be collected effectively“Base” appended to denominator counter name strings for identification in .smf file
14Performance SeNTry version 2. 4 Performance SeNTry version Parameters Administration, File ContentsSummarization utility (based on Counter Type)Requires Type 6 records!Designed to run at Cycle End, but can be run standaloneCreates sum.smf filesIdentical in format to detail recordsaged like any other .smf file in Current/Previous/Archive foldersCalculates the arithmetic mean for most CountersCalculates interval peak (min, max) where appropriateSome counters simply save the last observed value(e.g. elapsed time)
15SeNTry Administration enhancements Parameters Administration, File Contents ‘Include Registry Key Values’ to get registry key value information into configuration records in the SMF data file.Used to collect hardware and software inventory, versioning, etc.No restrictions: inventory as many values as you needMust specify the full Registry key name and field nameUse in conjunction with regedit Copy Key Name functionKey Not Found earns a Warning message, but no other side effectInvalid spec earns a Warning message, but no other side effect
16SeNTry Administration enhancements Parameters Administration, File Management PKZIP compression option at Cycle EndAnother way to manage data volume
17Performance SeNTry version 2.4.2 Event Log messages Informational, Warning and Error messages are placed in the Application Event logAlso in <system>.ntsmf.log (plain text)Date, time User, System, source filled in by event facilityType, category, event id, description defined by developersI and W will notPoint out that ntsmf.log is most valuable, and we size and age according to parametersDemand Technology, Inc.
18Performance SeNTry version 2.4.2 Event Log messages - Warning Warning is something that Performance SeNTry detects that does not impact ability to collect critical counters.Demand Technology, Inc.
19Performance SeNTry version 2. 4. 2 Event Log messages – ntsmf Performance SeNTry version Event Log messages – ntsmf.log example10/24/01-00:00:01 - Event ID: 210, Category: Discovery, Severity: WarningThe Performance Library, "C:\Program Files\Exchsrvr\bin\exprfdll.dll", for service "ExIPC" (ExIPC), failed to load.The system error was "The specified module could not be found (126)".Performance data for this service will not be available.The Event Viewer (source Perflib) may have more detailsDate, time, category, severity as in an Event Log entry. Text (description in an Event Log example) comes from GetlLastError() and FormatMEssage() api callsFinal line – the Perflib can and sometimes does put out information – but not always.Demand Technology, Inc.
20Performance SeNTry version 2. 4 Performance SeNTry version Event Log messages – unhandled exception10/13/01-00:00:02 - Event ID: 96, Category: N/A, Severity: ErrorUnhandled exception encountered10/13/01-00:00:02 - Ended Performance SeNTry service10/13/01-00:00:02 - Event ID: 104, Category: Initialization, Severity: InfoThe Performance SeNTry service stopped with error code,Version=2.3.6The stop error code is a result of a formatted 32 bit value.The severity, couple of flags and category are in the high order 16 bits. An assigned code is in the low order 16.The method implemented in Event log handling makes these difficult to deal with in automated systems. The event id, severity and text are available in both log forms to Event scrapers.Demand Technology, Inc.
21Performance SeNTry version 2.4.2 New Enhancements – SQL Server 2000 Lots of new SQL Server 2000 recordsA single machine can be run multiple instances of SQL ServerNo instance names for separate SQL Server processes, as delivered by MSMSSQL$db-name:object-namee.g. MSSQL$SUP:Buffer Manager,…MSSQL$SUP1:Buffer Manager,…)Anticipate object selection and reporting problemsWe write a MSSQL:Buffer Manager Object instance where the db-name token becomes an Instance name2.2.2,0,5,2001,4,17,20,36,45,859,0,MSSQL:Buffer Manager,1,18,22,1,SQL Server DB,Buffer cache hit ratio,…Instances definition problem -Demand Technology, Inc.
22Performance SeNTry version 2. 4 Performance SeNTry version New Enhancements - Exchange 2000 supportNew objects visible when optional components installed (e.g. Instant Messaging)Cases where a perflib loads another .dllexprfdll.dll loads exchmem.dllObjects have many counters:MSExchange Web Mail … 75 countersMSExchangeIMAP4 … countersMSExchangeIS ……… counters (actual)Database …………… Counters (many of ‘no data’ type)Discuss the development effort a bit, comment on the number of counters and point out that we/they have to figure out what Exchange 2000 stuff matters – I vaguely remember seeing a reference, can’t recall if it was an article in NT magazine or an MSDN thing on what to measure – should find a reference if we can before CMGDemand Technology, Inc.
23Performance SeNTry version 2.4.2 New Architecture - Threaded collector Separate thread for each performance library found in the Registry3rd party Perflib DLLs perform the actual performance data gatheringWindows 2000, XP expose the ‘system’ perflibs, including perfos.dll, perfdisk.dll, perfproc.dll, perfnet.dllImproved error handling, especially in 3rd party codeAdditional collection service threads manage dispatching, file writes, other tasksDemand Technology, Inc.
24NT Performance monitoring API Perflib Dlls are loaded into the dmperfss process address space when opened3rd party Perflib Dlls might need to synchronize with a separate monitored address space
25NT Performance monitoring API Open and Collect Timeout values exceeded trigger DisablePerformanceCounters setting, which must be disabled manually!HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Perflib ExtCounterTestLevel can be used to verify integrity of 3rd party Perflib dlls.
26Performance SeNTry version 2.4.2 New Architecture - Exception handling Try to catch exceptions that occur in a thread, i.e., inside code in a Perflib DLLThe failing thread and perflib (and the performance Objects it supplies) are deactivated, while other Objects continue to be collectedProblem perflib is automatically retried every new cycle or on demand (Notify Command)No manual intervention requiredUnless the Object is on the critical list:System, Memory, Cache, Process, Thread, ProcessorCritical missing means log and stopExplain why we thought to stop is better – data will be meaningless without ‘system’ stuff, also want to get User’s attention asap to address the system problemDemand Technology, Inc.
27Performance SeNTry version 2.4.2 New Architecture - Exception handling On exception:system error text messagethe loaded modules with addressesand the call stackHelps to identify ‘rogue Perflib dlls’Very compact and accurate
28Performance SeNTry version 2.4.2 Log message and module list 10/26/01-10:20:05 - Event ID: 1300, Category: PerfData, Severity: Warning Could not collect performance data for object, "Database ==> Instances (L28698, G3370), counter, Pages Converted/sec (L28700, G3372)". The system error was "Win32 exception, 0xC , encountered at location 0x0043FBB1 Read attempt at location 0x Module List: (0x x ) D:\NTSMFv24224\DMPerfss.exe (0x x ) F:\WINNT\System32\BEPerfDll.dll (0x x0114E000) D:\Program Files\Exchsrvr\bin\davexpc.dll (0x x0163B000) D:\Program Files\Exchsrvr\bin\eseperf.dll (0x x ) F:\WINNT\system32\dssdata.dll (0x02C x02C46000) D:\Program Files\Exchsrvr\bin\srsperf.dll….(lots more of these)Note that this was induced during testing – maybe find and example where it occurs I a perflib – Dave has test lib to work with.Point out content of .dll entries. Don’t talk too long to this – make the point and move onDemand Technology, Inc.
29Performance SeNTry version 2.4.2 Stack Trace Stack Trace (ThreadId 4540):DMPerfss.exe: PSSPerfLocalCounter::RealizeCache bytesE:\DmPerfss 2.4.2\Collector\PSSPerfLocalCounter.cpp, Line bytesDMPerfss.exe: PSSPerfObjectCache::RealizeCache bytesE:\DmPerfss 2.4.2\Collector\PSSPerfObjectCache.cpp, Line bytesDMPerfss.exe: PSSPerfObjectCache::OnCollectionComplete + 12 bytesE:\DmPerfss 2.4.2\Collector\PSSPerfObjectCache.cpp, Line bytesDMPerfss.exe: PSSPerfDataSink::Collect bytesE:\DmPerfss 2.4.2\Collector\PSSPerfDataSink.cpp, Line bytesDMPerfss.exe: PSSPerfDataSink::Collect + 15 bytesE:\DmPerfss 2.4.2\Collector\PSSPerfDataSink.cpp, Line bytesDMPerfss.exe: PSSPerfObject::_Collect + 75 bytesE:\DmPerfss 2.4.2\Collector\PSSPerfObject.cpp, Line bytesDMPerfss.exe: PSSPerfObject::Collect + 79 bytesE:\DmPerfss 2.4.2\Collector\PSSPerfObject.cpp, Line bytesExplain that the symbols give us this trace back of the code – this is why we need symbols!Demand Technology, Inc.
30Performance SeNTry version 2.4.2 Stack Trace DMPerfss.exe: PSSCollectionCycle::Start bytesE:\DmPerfss 2.4.2\Collector\PSSCollectionCycle.cpp, Line bytesDMPerfss.exe: PSSPerformanceSeNTry::StartCollection bytesE:\DmPerfss 2.4.2\Collector\PSSPerformanceSeNTry.cpp, Line bytesDMPerfss.exe: PSSPerformanceSeNTry::Run bytesE:\DmPerfss 2.4.2\Collector\PSSPerformanceSeNTry.cpp, Line bytes"
31Performance SeNTry version 2.4.2 New Architecture - Exception handling New dmperfss.exe ships with debug symbols included for extended error diagnostics reportSize of the image file is now approximately 5.8 MBThe larger image file installs by defaultA dmperfss.exe without symbols is also on the CDE.g., a floppy disk distribution/installdmperfss.exe with symbols is preferred, especially if new applications or new versions are discovered
33Performance SeNTry Version 3 Roadmap Real-time AlertsEvent log & SNMP supportFile System CollectorRevamped Systems ConsoleThese aren’t complete yet, need to review design and discuss with Dave and Phil before filling out rest of slidesDemand Technology, Inc.
34Performance SeNTry Version 3 Alerts Alert is written into the Application event logTriggered by system condition (e.g. cpu busy threshold)Frequency of alerts is also thresholded; won’t fill up the Event Log unnecessarilyTrigger, Release mechanismAlerts are cleared when condition is absent(based on the ‘lower threshold’)
35Performance SeNTry Version 3 SNMP - Traps Like Alerts, defined by same rulesSNMP extension agent notified of the ‘alert’ by dmperfss.exeSNMP extension agent generates a Trap messageTrap is sent to management entities by SNMP facilityManagement entities would be OpenView, Tivoli,CA-Unicenter, othersAddDemand Technology, Inc.
36Performance SeNTry Version 3 File System Collector At prototype stage now…Probably another thread in the collectorDesigned to run infrequently, say once/day(don’t know how long it takes yet)Report files that are above threshold(e.g. file size > 2% Logical disk size)Report folders that are above threshold when files in them are totaled(don’t also report parents, just siblings)Demand Technology, Inc.
37Performance SeNTry Version 3 File System Collector Summary records – size of files remainingNTSMF header record formatPseudo object idRecords are instances where the UNC path is the instance nameProbably optioned - allow include in .SMF file or write to a file of its own
38Performance SeNTry Version 3 System Console Processes real-time alertsCan examine objects and counters related to the alertsCan change the alert definition, collection interval, etc., as needed for the displayDisplay results graphicallyCapture the data and graphs for later reviewDemand Technology, Inc.
39Performance SeNTry Version 3 Exchange 2000 references (Microsoft's online version of the Exchange Resource Kit Documentation)