1. Performance SeNTry: Development Status Report
Demand Technology Software
A DataCore Software company
1020 Eighth Avenue South, Suite 6, Naples, FL 34102
phone: (941) fax: (941)
Demand Technology, Inc.

2. Agenda Performance SeNTry version 2.4.2 highlights Version 3 roadmap
NTSMF Collector
SeNTry Administration
Version 3 roadmap
Alerts and Traps
File System Collector
Improved Systems Console for real-time notification, management, and control
Demand Technology, Inc.

3. Performance SeNTry version 2.4.2 Topics
Windows 2000 ‘features’
NTSMF features
SeNTry Administration features
Event log use
New Enhancements – SQL Server 2000
New Enhancements - Exchange 2000 support
New collection service architecture –
Threaded collector
Improved Exception handling

4. Performance SeNTry version 2.4.2 Windows 2000 ‘features’
System level objects now in visible performance libraries
System object ‘missing’ some NT4 counters:
%Total Processor Time, % Total User Time, % Total Privileged Time, % Total DPC Time, % Total Interrupt Time, Total Interrupts/sec
Delay in some objects reporting data at system start going through RegQueryValueEx()
New values in Performance Registry keys
Timeouts
DisablePerformanceCounters
Discuss backwards compatibility and efforts at getting around these features
Demand Technology, Inc.

5. Performance SeNTry version 2. 4
Performance SeNTry version New fields in Registry: HKLM\…\Services\PerfDisk\Performance
Timeouts are in milliseconds – Collect is 2 secs, Open is 5 secs
Demand Technology, Inc.

6. Performance SeNTry version 2.4.2 Additional NTSMF features
Install option to run under a user account
-install -account <account name> -password <password>
Program defaults with filters active - NT4 and Win2K have specific sets
Memory size field in header fixed for Win 2000 systems with 4GB or more of physical memory
No fix for Windows NT4 systems with 4GB or more of physical memory since the GlobalMemoryStatus api call returns only a 32 bit number

7. SeNTry Administration enhancements
New default DCS for Win2K and Performance Gallery Gold templates
SQL Server 2000 object support
MS Exchange 2000 objects support
Tool to compact the Performance SeNTry Database ('Tools' menu item in the 'Network Browser' window)
More filters
More options

8. SeNTry Administration enhancements Collection Filters - Process
New Process Filter on Process Working Set Size
Always write named processes – availability reporting
Threads
Add 'Ready Threads' counter in the 'Process' object
No longer need to write Thread objects into .smf
Terminal Server
suppress Idle User record
write User and Session records (automatic) only when activity occurs
Demand Technology, Inc.

9. Performance SeNTry version 2.4.2 Collection filters, Thread
Top filter triggers writing the ‘Ready Threads’ ntsmf counter that removes need to write any thread records
Demand Technology, Inc.

10. SeNTry Administration enhancements Parameters Administration, Collection Cycle
On Error Command capability for launching commands when errors are encountered
Same rules as Cycle End commands
‘poor man’s’ alert – something as simple as ‘Net Send user-name message’

11. SeNTry Administration enhancements Parameters Administration, Scheduling
Flexible DCS chaining specifying Next DCS in chain
(e.g., prime shift.DCS chains to nighttimeDCS, which chains back)
Collection Period to specify the period that the collector is active and reporting data
Provide basic scheduling options without requiring a 3rd party package

12. Performance SeNTry version 2. 4
Performance SeNTry version Parameters Administration, File Contents
Support for summarization program
Collect Inventory data from Registry keys
New option – “Write denominator counters”
Demand Technology, Inc.

13. Performance SeNTry version 2. 4
Performance SeNTry version Parameters Administration, File Contents
New option – “Write denominator counters”
Used by .smf Summarization (DMSumSMF.exe) uses in calculations to compute correct average values of compound Counters
e.g., Copy Read Hits %
One more option to manage the volume of data that can be collected effectively
“Base” appended to denominator counter name strings for identification in .smf file

14. Performance SeNTry version 2. 4
Performance SeNTry version Parameters Administration, File Contents
Summarization utility (based on Counter Type)
Requires Type 6 records!
Designed to run at Cycle End, but can be run standalone
Creates sum.smf files
Identical in format to detail records
aged like any other .smf file in Current/Previous/Archive folders
Calculates the arithmetic mean for most Counters
Calculates interval peak (min, max) where appropriate
Some counters simply save the last observed value
(e.g. elapsed time)

15. SeNTry Administration enhancements Parameters Administration, File Contents
‘Include Registry Key Values’ to get registry key value information into configuration records in the SMF data file.
Used to collect hardware and software inventory, versioning, etc.
No restrictions: inventory as many values as you need
Must specify the full Registry key name and field name
Use in conjunction with regedit Copy Key Name function
Key Not Found earns a Warning message, but no other side effect
Invalid spec earns a Warning message, but no other side effect

16. SeNTry Administration enhancements Parameters Administration, File Management
PKZIP compression option at Cycle End
Another way to manage data volume

17. Performance SeNTry version 2.4.2 Event Log messages
Informational, Warning and Error messages are placed in the Application Event log
Also in <system>.ntsmf.log (plain text)
Date, time User, System, source filled in by event facility
Type, category, event id, description defined by developers
I and W will not
Point out that ntsmf.log is most valuable, and we size and age according to parameters
Demand Technology, Inc.

18. Performance SeNTry version 2.4.2 Event Log messages - Warning
Warning is something that Performance SeNTry detects that does not impact ability to collect critical counters.
Demand Technology, Inc.

19. Performance SeNTry version 2. 4. 2 Event Log messages – ntsmf
Performance SeNTry version Event Log messages – ntsmf.log example
10/24/01-00:00:01 - Event ID: 210, Category: Discovery, Severity: Warning
The Performance Library, "C:\Program Files\Exchsrvr\bin\exprfdll.dll", for service "ExIPC" (ExIPC), failed to load.
The system error was "The specified module could not be found (126)".
Performance data for this service will not be available.
The Event Viewer (source Perflib) may have more details
Date, time, category, severity as in an Event Log entry. Text (description in an Event Log example) comes from GetlLastError() and FormatMEssage() api calls
Final line – the Perflib can and sometimes does put out information – but not always.
Demand Technology, Inc.

20. Performance SeNTry version 2. 4
Performance SeNTry version Event Log messages – unhandled exception
10/13/01-00:00:02 - Event ID: 96, Category: N/A, Severity: Error
Unhandled exception encountered
10/13/01-00:00:02 - Ended Performance SeNTry service
10/13/01-00:00:02 - Event ID: 104, Category: Initialization, Severity: Info
The Performance SeNTry service stopped with error code,
Version=2.3.6
The stop error code is a result of a formatted 32 bit value.
The severity, couple of flags and category are in the high order 16 bits. An assigned code is in the low order 16.
The method implemented in Event log handling makes these difficult to deal with in automated systems. The event id, severity and text are available in both log forms to Event scrapers.
Demand Technology, Inc.

21. Performance SeNTry version 2.4.2 New Enhancements – SQL Server 2000
Lots of new SQL Server 2000 records
A single machine can be run multiple instances of SQL Server
No instance names for separate SQL Server processes, as delivered by MS
MSSQL$db-name:object-name
e.g. MSSQL$SUP:Buffer Manager,…
MSSQL$SUP1:Buffer Manager,…)
Anticipate object selection and reporting problems
We write a MSSQL:Buffer Manager Object instance where the db-name token becomes an Instance name
2.2.2,0,5,2001,4,17,20,36,45,859,0,MSSQL:Buffer Manager,1,18,22,1,SQL Server DB,Buffer cache hit ratio,…
Instances definition problem -
Demand Technology, Inc.

22. Performance SeNTry version 2. 4
Performance SeNTry version New Enhancements - Exchange 2000 support
New objects visible when optional components installed (e.g. Instant Messaging)
Cases where a perflib loads another .dll
exprfdll.dll loads exchmem.dll
Objects have many counters:
MSExchange Web Mail … 75 counters
MSExchangeIMAP4 … counters
MSExchangeIS ……… counters (actual)
Database …………… Counters (many of ‘no data’ type)
Discuss the development effort a bit, comment on the number of counters and point out that we/they have to figure out what Exchange 2000 stuff matters – I vaguely remember seeing a reference, can’t recall if it was an article in NT magazine or an MSDN thing on what to measure – should find a reference if we can before CMG
Demand Technology, Inc.

23. Performance SeNTry version 2.4.2 New Architecture - Threaded collector
Separate thread for each performance library found in the Registry
3rd party Perflib DLLs perform the actual performance data gathering
Windows 2000, XP expose the ‘system’ perflibs, including perfos.dll, perfdisk.dll, perfproc.dll, perfnet.dll
Improved error handling, especially in 3rd party code
Additional collection service threads manage dispatching, file writes, other tasks
Demand Technology, Inc.

24. NT Performance monitoring API
Perflib Dlls are loaded into the dmperfss process address space when opened
3rd party Perflib Dlls might need to synchronize with a separate monitored address space

25. NT Performance monitoring API
Open and Collect Timeout values exceeded trigger DisablePerformanceCounters setting, which must be disabled manually!
HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Perflib ExtCounterTestLevel can be used to verify integrity of 3rd party Perflib dlls.

26. Performance SeNTry version 2.4.2 New Architecture - Exception handling
Try to catch exceptions that occur in a thread, i.e., inside code in a Perflib DLL
The failing thread and perflib (and the performance Objects it supplies) are deactivated, while other Objects continue to be collected
Problem perflib is automatically retried every new cycle or on demand (Notify Command)
No manual intervention required
Unless the Object is on the critical list:
System, Memory, Cache, Process, Thread, Processor
Critical missing means log and stop
Explain why we thought to stop is better – data will be meaningless without ‘system’ stuff, also want to get User’s attention asap to address the system problem
Demand Technology, Inc.

27. Performance SeNTry version 2.4.2 New Architecture - Exception handling
On exception:
system error text message
the loaded modules with addresses
and the call stack
Helps to identify ‘rogue Perflib dlls’
Very compact and accurate

28. Performance SeNTry version 2.4.2 Log message and module list
10/26/01-10:20:05 - Event ID: 1300, Category: PerfData, Severity: Warning Could not collect performance data for object, "Database ==> Instances (L28698, G3370), counter, Pages Converted/sec (L28700, G3372)". The system error was "Win32 exception, 0xC , encountered at location 0x0043FBB1 Read attempt at location 0x Module List: (0x x ) D:\NTSMFv24224\DMPerfss.exe (0x x ) F:\WINNT\System32\BEPerfDll.dll (0x x0114E000) D:\Program Files\Exchsrvr\bin\davexpc.dll (0x x0163B000) D:\Program Files\Exchsrvr\bin\eseperf.dll (0x x ) F:\WINNT\system32\dssdata.dll (0x02C x02C46000) D:\Program Files\Exchsrvr\bin\srsperf.dll ….
(lots more of these)
Note that this was induced during testing – maybe find and example where it occurs I a perflib – Dave has test lib to work with.
Point out content of .dll entries. Don’t talk too long to this – make the point and move on
Demand Technology, Inc.

29. Performance SeNTry version 2.4.2 Stack Trace
Stack Trace (ThreadId 4540):
DMPerfss.exe: PSSPerfLocalCounter::RealizeCache bytes
E:\DmPerfss 2.4.2\Collector\PSSPerfLocalCounter.cpp, Line bytes
DMPerfss.exe: PSSPerfObjectCache::RealizeCache bytes
E:\DmPerfss 2.4.2\Collector\PSSPerfObjectCache.cpp, Line bytes
DMPerfss.exe: PSSPerfObjectCache::OnCollectionComplete + 12 bytes
E:\DmPerfss 2.4.2\Collector\PSSPerfObjectCache.cpp, Line bytes
DMPerfss.exe: PSSPerfDataSink::Collect bytes
E:\DmPerfss 2.4.2\Collector\PSSPerfDataSink.cpp, Line bytes
DMPerfss.exe: PSSPerfDataSink::Collect + 15 bytes
E:\DmPerfss 2.4.2\Collector\PSSPerfDataSink.cpp, Line bytes
DMPerfss.exe: PSSPerfObject::_Collect + 75 bytes
E:\DmPerfss 2.4.2\Collector\PSSPerfObject.cpp, Line bytes
DMPerfss.exe: PSSPerfObject::Collect + 79 bytes
E:\DmPerfss 2.4.2\Collector\PSSPerfObject.cpp, Line bytes
Explain that the symbols give us this trace back of the code – this is why we need symbols!
Demand Technology, Inc.

30. Performance SeNTry version 2.4.2 Stack Trace
DMPerfss.exe: PSSCollectionCycle::Start bytes
E:\DmPerfss 2.4.2\Collector\PSSCollectionCycle.cpp, Line bytes
DMPerfss.exe: PSSPerformanceSeNTry::StartCollection bytes
E:\DmPerfss 2.4.2\Collector\PSSPerformanceSeNTry.cpp, Line bytes
DMPerfss.exe: PSSPerformanceSeNTry::Run bytes
E:\DmPerfss 2.4.2\Collector\PSSPerformanceSeNTry.cpp, Line bytes"

31. Performance SeNTry version 2.4.2 New Architecture - Exception handling
New dmperfss.exe ships with debug symbols included for extended error diagnostics report
Size of the image file is now approximately 5.8 MB
The larger image file installs by default
A dmperfss.exe without symbols is also on the CD
E.g., a floppy disk distribution/install
dmperfss.exe with symbols is preferred, especially if new applications or new versions are discovered

32. Performance SeNTry Version 3
BREAK

33. Performance SeNTry Version 3 Roadmap
Real-time Alerts
Event log & SNMP support
File System Collector
Revamped Systems Console
These aren’t complete yet, need to review design and discuss with Dave and Phil before filling out rest of slides
Demand Technology, Inc.

34. Performance SeNTry Version 3 Alerts
Alert is written into the Application event log
Triggered by system condition (e.g. cpu busy threshold)
Frequency of alerts is also thresholded; won’t fill up the Event Log unnecessarily
Trigger, Release mechanism
Alerts are cleared when condition is absent
(based on the ‘lower threshold’)

35. Performance SeNTry Version 3 SNMP - Traps
Like Alerts, defined by same rules
SNMP extension agent notified of the ‘alert’ by dmperfss.exe
SNMP extension agent generates a Trap message
Trap is sent to management entities by SNMP facility
Management entities would be OpenView, Tivoli,
CA-Unicenter, others
Add
Demand Technology, Inc.

36. Performance SeNTry Version 3 File System Collector
At prototype stage now…
Probably another thread in the collector
Designed to run infrequently, say once/day
(don’t know how long it takes yet)
Report files that are above threshold
(e.g. file size > 2% Logical disk size)
Report folders that are above threshold when files in them are totaled
(don’t also report parents, just siblings)
Demand Technology, Inc.

37. Performance SeNTry Version 3 File System Collector
Summary records – size of files remaining
NTSMF header record format
Pseudo object id
Records are instances where the UNC path is the instance name
Probably optioned - allow include in .SMF file or write to a file of its own

38. Performance SeNTry Version 3 System Console
Processes real-time alerts
Can examine objects and counters related to the alerts
Can change the alert definition, collection interval, etc., as needed for the display
Display results graphically
Capture the data and graphs for later review
Demand Technology, Inc.

39. Performance SeNTry Version 3 Exchange 2000 references
(Microsoft's online version of the Exchange Resource Kit Documentation)

40. Performance SeNTry reach us at:
Demand Technology Software
A DataCore Software company
Address: 1020 Eighth Avenue South, Suite 6, Naples, FL 34102
Phone: (941) fax: (941)
Web site:

41. Performance SeNTry Version 3 Roadmap
Questions?