1. Copyright © 2003, SAS Institute Inc. All rights reserved. Where's Waldo Uncovering Hard-to-Find Application Killers Claire Cates SAS Institute, Inc Claire.Cates@sas.com

2. Copyright © 2003, SAS Institute Inc. All rights reserved. 2

3. 3 Always something different  Just like the Waldo pictures, performance is always different and not easy to find. Performance based on the test that was run Performance based on the data that was used Performance based on the hardware used Performance based on the number of users

4. Copyright © 2003, SAS Institute Inc. All rights reserved. 4 What is a Performance Problem  When jobs cannot execute within a required time period.  Something that makes the system run slower than it should  When a system cannot sustain expected or unexpected growth (scalability) More users More data More processors

5. Copyright © 2003, SAS Institute Inc. All rights reserved. 5 Causes of Performance Problems  Hardware  Software  Integration of software systems  Network  Web

6. Copyright © 2003, SAS Institute Inc. All rights reserved. 6 How Do You Find Performance Problems Vendor profilers Looking at system variables (cache hits, IO timings,…) Profilers that analyze the code Code reviews Network sniffers Profiles that profile the entire machine Code instrumentation

7. Copyright © 2003, SAS Institute Inc. All rights reserved. 7 Common Software Performance Problems Antipatterns  Excessive Memory Allocation  The Ramp  Unnecessary Processing  One Lane Bridge  How Many Times Do I have to tell you.  Unbalanced Processing  More is Less

8. Copyright © 2003, SAS Institute Inc. All rights reserved. 8 Excessive Memory Allocation  Memory Allocations/Frees are expensive OS calls If your system is multi-threaded, locking may need to occur  Often times memory is allocated temporarily within a function  Often times structures are allocated and freed and not reused  Memory allocations in loops.  Possible Fixes Use a stack variable instead of temporary memory allocations Maintain a free chain for structures that are used often Make one large allocation outside loop – split it within the loop

9. Copyright © 2003, SAS Institute Inc. All rights reserved. 9 The Ramp  When processing slows down over time  Can be caused by long lists that grow and need to be searched  Can be caused by too many resources being used  Another kind of Ramp happens when the system doesn’t scale well.  Possible fixes Use a different algorithm when you know large amounts of data will be required Make sure all resources (especially memory) are returned.

10. Copyright © 2003, SAS Institute Inc. All rights reserved. 10 Unnecessary Processing  Loops that contain statements that do not need to execute on each iteration of the loop  Macros that are inefficient because they are too general  Zeroing of allocated memory that does not need to be zeroed.  Inefficient algorithms  Possible fixes Rework algorithm Don’t use macros

11. Copyright © 2003, SAS Institute Inc. All rights reserved. 11 One Lane Bridge  Resource contention – every thread waiting on the same resource  Eventually causes a traffic jam  Possible fixes Change the algorithm to use less resources that are shared

12. Copyright © 2003, SAS Institute Inc. All rights reserved. 12 How Many Times Do I Have To Tell You  Calling a routine too often with the same parameters  Sometimes a routine (foo) is the hot spot yet, (foo) is fine, it is just that foo is called to often. Most common occurrence of this that I see occurs when (afoo) (bfoo) (cfoo) (foo) So cfoo is called twice and foo is called 4 times all because afoo does not know what bfoo and cfoo actually call

13. Copyright © 2003, SAS Institute Inc. All rights reserved. 13 Unbalanced Processing  Occurs in a multi-threaded environment running on a multi-processor machine.  When one or two threads do all the processing while other threads are idle  Possible fixes Rebalance the work in the threads so that all threads process a similar amount of work

14. Copyright © 2003, SAS Institute Inc. All rights reserved. 14 More is Less or Thundering Herd  Adding threads does not make a system run faster  The system spends its time thrashing  Thrashing can occur because of Excessive thread switching Destroying the system memory or I/O caches when each thread runs  Possible fixes Don’t create more threads than there is work to be done

15. Copyright © 2003, SAS Institute Inc. All rights reserved. 15 Common Profilers in Use at SAS  InTune – z/OS  VTune – Intel – PCs  Caliper – HP  Rational Quantify – Windows, Solaris, HP, Linux  Jprobe  OS tools – not really profilers but gives information on −System calls −IO bottlenecks −Caching problems

16. Copyright © 2003, SAS Institute Inc. All rights reserved. 16 SAS’s use of Quantify  Other Profilers we use Event / timer based Based more on the function level, not the source level Network traffic & other processes on the machine distorted the results  Great results from all profilers  Tips I’m using with Quantify do apply to these other tools

17. Copyright © 2003, SAS Institute Inc. All rights reserved. 17 Quantify differences…  Quantify Counts instruction cycles for the user code Times the system calls (can be turned off) Does not have to run on a clean machine (a machine with no other activity) – developer machine Provides performance information down to the source line level Performance can be broken down by threads

18. Copyright © 2003, SAS Institute Inc. All rights reserved. 18 A Little about Quantify  Quantify integrates itself with your system. As your system executes, Quantify will accumulate a mass of data. Function counts for every function used The amount of time spent in every function The amount of time spent in every function plus the amount of time spent in all its descendent routines. Full Call tree information

19. Copyright © 2003, SAS Institute Inc. All rights reserved. 19 Quantify Quick Overview

26. Copyright © 2003, SAS Institute Inc. All rights reserved. 26 Excessive Memory Allocation

36. Copyright © 2003, SAS Institute Inc. All rights reserved. 36 How Many Times Do I Have to Tell You?

43. Copyright © 2003, SAS Institute Inc. All rights reserved. 43 More Is Less

47. Copyright © 2003, SAS Institute Inc. All rights reserved. 47

52. Copyright © 2003, SAS Institute Inc. All rights reserved. 52 Unnecessary Processing

55. Copyright © 2003, SAS Institute Inc. All rights reserved. 55 One Lane Bridge

59. Copyright © 2003, SAS Institute Inc. All rights reserved. 59 Unbalanced Processing

64. Copyright © 2003, SAS Institute Inc. All rights reserved. 64 Ramp

70. Copyright © 2003, SAS Institute Inc. All rights reserved. 70 Another Useful Hint

72. Copyright © 2003, SAS Institute Inc. All rights reserved. 72 Summary  Profilers are useful in helping to uncover performance problems in a software application  By being patient and looking at the mass of data provided by the tools you can find out a tremendous amount about what your system is doing and how to optimize the performance.

73. Copyright © 2003, SAS Institute Inc. All rights reserved. 73 References  AntiPatterns – Refactoring Software, Architectures, and Projects in Crisis - Brown, Malveau, McCormick & Mowbray ISBN 0-471-19713-0  New Software Performance AntiPatterns: More Ways to Shoot Yourself in the Foot. Smith & Williams CMG 2002 Paper Presentation  Computer Measurement Group www.cmg.org

74. Copyright © 2003, SAS Institute Inc. All rights reserved. 74