Performance Analysis and Tuning Presented by Carol ThornborrowPresented by Carol Thornborrow

System Performance User “perceived performance”

System Performance System performance is the efficiency which system resources respond to current demands on the system.System performance is the efficiency which system resources respond to current demands on the system. Service Level Agreement (SLA) - Agreement between management, the user community and the information technology staff.

System Performance What is “normal”?What is “normal”? Create a baseline for comparison.Create a baseline for comparison.

Monitoring Resources which affect system performanceResources which affect system performance –CPU –Memory –Disk I/O –Network I/O Resources affect on system performanceResources affect on system performance –Not enough of the resource –Allocation of the resource is being mismanaged

Monitoring Identifying bottlenecksIdentifying bottlenecks –What is the problem? –Identify the Cause –What will it take to fix? –Is the problem fixed? –Reevaluate, start over Monitoring ToolsMonitoring Tools –OS utilities vmstat, netstatvmstat, netstat –Vendor products GlancePlus, SarCheckGlancePlus, SarCheck

Monitoring - CPU Overall utilizationOverall utilization –vmstat on most systems –sar –u Solaris and HP-UX Load AverageLoad Average –uptime Per process CPU timePer process CPU time –ps –elf on HP-UX, Solaris –ps –aux on Red Hat, Free BSD –top - real time –pstree init-+-alarmdinit-+-alarmd |-7*[kdeinit] |-7*[kdeinit]

Monitoring - CPU vmstat 5 4vmstat 5 4 procs cpu r b w us sy id r > 3-6/CPU -us + sy > 80% sar –u 5 4 %usr %sys %wio %idle Average

Tuning - CPU Manually change priorityManually change priority –System V, Free BSD renice new nice# pidrenice new nice# pid –Solaris also has priocntl –s –p new-pri –i pid proc-idpriocntl –s –p new-pri –i pid proc-id Replace CPUReplace CPU Add CPUAdd CPU Redistribute loadRedistribute load –At – run process at specific time –Batch – run processes sequentially –Idle CPU

Monitoring - Memory Active virtual memoryActive virtual memory –Solaris swap –l or sar -r –HP-UX swapinfo –Free BSDpstat -s –Red Hatswapon –s Paging ratePaging rate –vmstat # of running processes# of running processes –vmstat SwapsSwaps –vmstat

Monitoring - Memory vmstat 5 4vmstat 5 4 r w swap free re po sar –r 5 4sar –r :03:06 freemem freeswap 16:03: :03: :03: :03:

Tuning – Memory Add more memoryAdd more memory Distribute swap spaceDistribute swap space Distribute jobs – at, batchDistribute jobs – at, batch Virtual memory and pagingVirtual memory and paging –Free BSD sysctl Increase vm.v_cache_min and/or vm.v_cache_max to favor fs buffer cache over user processesIncrease vm.v_cache_min and/or vm.v_cache_max to favor fs buffer cache over user processes

Monitoring – Disk I/O Avg size of transfersAvg size of transfers iostat iostat dad0 dad0 kps tps kps tps kps/tps = Kb/transfer kps/tps = Kb/transfer % of disk utilization% of disk utilization iostat –D iostat –D ssd0 ssd0 rps wps util rps wps util System V –sar –d Need accounting turned on May affect performance

Tuning – Disk I/O Spread active files over several disks and controllersSpread active files over several disks and controllers Sequential files are contiguous and near outer edge of diskSequential files are contiguous and near outer edge of disk Place randomly accessed files near center of diskPlace randomly accessed files near center of disk Place root system on own disk if possiblePlace root system on own disk if possible Disk stripingDisk striping Purchase fast disks w fast controllersPurchase fast disks w fast controllers

Monitoring and Tuning Network I/O netstat –inetstat –i –Collisions > 5% of total calls pingping nfsstat –cnfsstat –c –retrans > 5% of calls badxid = retrans - > serverbadxid = retrans - > server badxid networkbadxid network Spread the network load between servers Add more bandwidth