Paul Jaffray Technical Services Manager Promedica Health System, Inc. Wednesday March 6th, 2002 Session 5823: Working with Windows NT Performance Data

Abstract “Real world” look at working with NT performance data Application of MVS techniques to NT High level overview of NT performance data Tools used for collection and analysis What data do we look at Discussion – What does everyone else do?

Intro I am not an expert on NT or Performance I took my experience from managing SMF data on MVS and creating various performance reports for the mainframe and applied those processes to NT I am still learning how NT works and what the various counters represent Still developing ROTs for various counters

What Data Is Available? Various Performance Objects –Each object contains related Performance Counters Processor:% Processor Time Processor:Interrupts/Sec Processor:DPC Rate Lots and lots of counters –But how good is the data?

Performance Objects Processor System Memory Network Interface Physical Disk Process Print Queue Server Thread MSExchangeDB MSExchangeDS MSExchangeMTA Lotus.Notes.MTA SQLServer SQLServer:Databases AppleTalk ADSM Client Performance ColdFusion Server

What’s Missing No response time counters –Exchange Delivery times are available –SQL Transactions per Second

How Can I View the Data? Task Manager –Real Time –Local machine only –Provides summary performance data –Not customizable –How to start: Right-click the task bar and select Task Manager Crtl-Alt-Del and select Task Manager

How Can I View the data? Performance Monitor –Real time or historical via logs –Also has alerting facility –Local machine or remote –You can select which objects and counters to view –How to start: Start:Run:PERFMON Start:Programs:Administrative Tools:Performance

So Now What? I wanted a way to consolidate performance data from every server to a single point and combine into a daily file for processing. This file should then be saved where it can be combined into a monthly file. A number of daily files and all monthly files should be archived so they can be retrieved and reported against in the future.

Challenges No SMF facility standard with Windows No IFASMFDP or DFSORT No JCL/TSO/ISPF How to automate How to locate and copy the data How to handle and report errors How to analyze data and create reports

Solution (so far) NTSMF PrimalScript VBScript code Scheduled Tasks Text file with a list of servers with data location SAS MXG

Creating the “SMF” Files On each server NTSMF gets data from standard NT interfaces, builds records and writes them to a “current” file NTSMF Directory Structure –C:\NTSMF\Data\Current Holds active file –C:\NTSMF\Data\Previous Yesterday’s data file(s) –C:\NTSMF\Data\Archive Settable number of days worth of data

Gathering The Data A script, driven by a list of servers to collect from, looks for files on each of those servers where the date in the file name is equal to yesterday’s date and copies those files to a directory on the system the script is running on All files are combined together in a single daily file that is available locally for SAS/MXG processing and is copied to a server for archival and later consolidation

Error Logging Status messages are written to the NT Application Log –Start and Stop times for the script –Servers not found –Data not found Messages are also gathered together and ed to me as the last step in the script

Reporting On the Data Processing can be done with SAS on MVS or Windows MXG provides SAS Macros to process the raw NTSMF data into SAS observations SAS can then be used to summarize and report on the data

Some Gotcha’s Cross domain trusts –Must be in place to copy files from servers in other domains Permissions –File access –Registry updates

Some Gotcha’s A crashed server can dump several K worth of nulls onto the end of a.smf file These nulls cause SAS to stop processing the file If they exist in the daily file they have to found and removed

Some Gotcha’s Disk performance numbers must be turned on by issuing the diskperf command from the command prompt –C:\> diskperf –yv Some network numbers require Network Monitoring Agent to be installed

Some Gotcha’s Memory:Page Faults/Sec –Can be very misleading as it includes Transition faults –Should use Memory:Page Reads/Sec Accurate reflection of demand paging

Some Gotcha’s Microsoft’s Outlook security patch in response to the “ILOVEYOU” virus broke the part of my initial script –New version of script requires the SMTP service to be loaded on the machine that runs the script

Some Gotcha’s Manually have to maintain the Server List file –Can be coordinated with the setting of the NTSMF options that must be set once the collector software has been loaded

What Numbers Do We Look At? Daily – 3 reports –Configuration report Shows all servers we collect data from and their configuration –Disk Numbers Free space shortages and high queuing –System Hourly Summaries Hourly report for each server of key values

What Numbers Do We Look At? Processor:% Processor Time –Looking for busy processors –SQL based applications seem to suffer at 60% –Others? System:Processor Queue Length –Indicates work waiting for CPU Doesn’t seem to work like I would expect

What Numbers Do We Look At? Memory:Available Bytes –Shows physical memory installed but not used by operating system or application –Large number indicates an over-configured server Memory:Page Reads/sec –Actual page faults resulting in a page in from disk –Page Faults/sec counter is misleading –Should be less than 200/sec?

What Numbers Do We Look At? PhysicalDisk:Avg. Disk Queue Length –How backed up is the IO subsystem –Shouldn’t be greater than 3 or 4? PhysicalDisk:Disk Bytes/sec –How much data is being moved –How much is too much depends on the interface

What Numbers Do We Look At? PhysicalDisk:Avg. Disk Sec/Read –Average number of seconds per read –Less than 20ms – good, more – bad PhysicalDisk:Avg. Disk Sec/Write –Same as above except for writes

Discussion Comments Questions What are others doing?

Tools – NTSMF Windows NT/2000 performance data collector Creates “SMF” files containing records of selected performance counters Demand Technology Software

Tools – PrimalScript “The professional scripting environment of choice for developers working in any scripting language” Useful for working with VBScript SAPIEN Technologies Inc.

Tools – SAS Data Access/Management/Analysis/Presentation Tool Allows processing and summerization of raw data SAS Institute Inc.

Tools – MXG “SAS-based software package that processes the "SMF" data records created by computer operating systems” Turns raw data into observations in SAS datasets Merrill Consultants

Some Books Microsoft Windows 2000 Performance Tuning Technical Reference –ISBN Windows 2000 Performance Guide –ISBN Tuning Netfinity Servers for Performance –IBM Redbook / ISBN