Introduction Andy Wang CIS Computer Systems Performance Analysis

2 Overview of Analysis of Software Systems Introduction Common mistakes made in system software analysis –And how to avoid them Selection of techniques and metrics

3 Introduction Why do we care? Why is it hard? What tools can we use to understand system performance?

4 Why Do We Care? Performance is almost always a key issue in software Everyone wants best possible performance Cost of achieving performance also key Reporting performance is necessary in many publication venues –Both academic and industry

5 Importance of Performance in Research Performance is key in CS research A solution that doesn’t perform well isn’t a solution at all Must prove performance characteristics to a skeptical community

6 State of Performance Evaluation in the Field Generally regarded as poor Many systems have little performance data presented –Measured by improper criteria –Experiments poorly designed –Results badly or incorrectly presented –Replication not generally respected

7 What’s the Result? Can’t always trust what you read in a research paper Authors may have accidentally or intentionally misled you –Overstating performance –Hiding problems –Not answering the important questions

8 Where Does This Problem Come From? Mostly from ignorance of proper methods of measuring and presenting performance Abetted by reader’s ignorance of what questions they should be asking

9 But the Field Is Improving People are taking performance measurement more seriously Quality of published experiments is increasing Yours had better be of high quality, too –Publishing is tough –Be at the top of the heap of papers

10 What Do You Need To Know? Selection of proper experiment characteristics Proper performance measurement techniques Proper statistical techniques Proper data presentation techniques

11 Selecting Appropriate Experiment Characteristics Evaluation techniques Performance metrics Workloads

12 Evaluation Techniques What’s the best technique for getting an answer to a performance question? Modeling? Simulation? Experimentation?

13 The Role of Modeling and Simulation Measurement not always best tool Modeling might be better Simulation might be better But that’s not what this class is about We will discuss when to use those techniques, though

14 Performance Metrics The criteria used to evaluate the performance of a system E.g., response time, transactions per second, bandwidth delivered, etc. Choosing the proper metrics is key to a real understanding of system performance

15 Workloads The requests users make on a system Without a proper workload, you aren’t measuring what users will experience Typical workloads: –Types of queries –Jobs submitted to an OS –Messages sent through a protocol

16 Proper Performance Measurement Techniques You need at least two components to measure performance: 1. A load generator To apply a workload to the system 2. A monitor To find out what happened

17 Proper Statistical Techniques Most computer performance measurements not deterministic Most performance evaluations weigh effects of different alternatives How to separate meaningless variations from vital data in measurements? Requires proper statistical techniques

18 Minimizing Your Work Unless you design carefully, you’ll measure a lot more than you need to A careful design can save you from doing lots of measurements –Should identify critical factors –And determine smallest number of experiments that gives sufficiently accurate answer

19 Proper Data Presentation Techniques You’ve got pertinent, statistically accurate data that describes your system Now what? How to present it –Honestly –Clearly –Convincingly

20 Why Is Performance Analysis Difficult? Because it’s an art - it’s not mechanical You can’t just apply a handful of principles and expect good results You’ve got to understand your system You’ve got to select your measurement techniques and tools properly You’ve got to be careful and honest

Bad Example: Ratio Game Throughput in transactions per second SystemWorkload 1Workload 2Average A B102015

Bad Example: Ratio Game Throughput normalized to system B Throughput normalized to system A SystemWorkload 1Workload 2Average A B111 SystemWorkload 1Workload 2Average A111 B

23 Example Suppose you’ve built an OS for a special-purpose Internet browsing box How well does it perform? Indeed, how do you even begin to answer that question?

24 Starting on an Answer What’s the OS supposed to do? What demands will be put on it? What HW will it work with, and what are that hardware’s characteristics? What performance metrics are most important? –Response time? –Delivered bandwidth?

25 Some Common Mistakes in Performance Evaluation List is long (nearly infinite) We’ll cover the most common ones …and how to avoid them

26 No Goals If it’s not clear what you’re trying to find out, you won’t find out anything useful It’s very hard to design good general- purpose experiments and frameworks So know what you want to discover Think before you start

27 Biased Goals Don’t set out to show OUR system is better than THEIR system –Doing so biases you towards using certain metrics, workloads, and techniques –Which may not be the right ones Try not to let your prejudices dictate the way you measure a system Try to disprove your hypotheses

28 Unsystematic Approach Avoid scattershot approaches Work from a plan Follow through on the plan Otherwise, you’re likely to miss something –And everything will take longer, in the end

29 Analysis Without Understanding the Problem Properly defining what you’re trying to do is a major component of the performance evaluation Don’t rush into gathering data till you understand what’s important This is the most common mistake in this class

30 Incorrect Performance Metrics If you don’t measure what’s important, your results don’t shed much light –Example: instruction rates of CPUs with different architectures Avoid choosing the metric that’s easy to measure (but isn’t helpful) over the one that’s hard to measure (but captures the performance)

31 Unrepresentative Workload If workload isn’t like what normally happens, results aren’t useful E.g., for Web browser, it’s wrong to measure its performance against –Just text pages –Just pages stored on the local server

32 Wrong Evaluation Technique Not all performance problems are properly understood by measurement –E.g., issues of scaling, or testing for rare cases Measurement is labor-intensive Sometimes hard to measure peak or unusual conditions Decide whether you should model/simulate before designing a measurement experiment

33 Overlooking Important Parameters Try to make a complete list of system/workload characteristics that affect performance Don’t just guess at a couple of interesting parameters Despite your best efforts, you may miss one anyway –But the better you understand the system, the less likely you will

34 Ignoring Significant Factors Factor: parameter you vary Not all parameters equally important More factors  more experiments –But don’t ignore significant ones –Give weight to those that users can vary

35 Inappropriate Experiment Design Too few test runs Wrong values for parameters Interacting factors can complicate proper design of experiments

36 Inappropriate Level of Detail Be sure you’re investigating what’s important Examining at too high a level may oversimplify or miss important factors Going too low wastes time and may cause you to miss forest for trees

37 No Analysis Raw data isn’t too helpful Remember, final result is analysis that describes performance Preferably in compact form easily understood by others –Especially for non-technical audiences Common mistake in this class (trying to satisfy page minimum in final report)

38 Erroneous Analysis Often based on misunderstanding of how statistics should be handled Or on not understanding transient effects in experiments Or in careless handling of the data Many other possible problems

39 No Sensitivity Analysis Rarely is one number or one curve fully descriptive of a system’s performance How different will things be if parameters are varied? Sensitivity analysis addresses this problem

40 Ignoring Errors in Input Particularly a problem when you have limited control over the parameters If you can’t measure an input parameter directly, need to understand any bias in indirect measurement

41 Improper Treatment of Outliers Sometimes particular data points are far outside range of all others Sometimes they’re purely statistical Sometimes they indicate a true, different behavior of system You must determine which are which

42 Assuming Static Systems A previous experiment may be useless if workload changes Or if key system parameters change Don’t rely blindly on old results –E.g. Ousterhout’s old study of file servers –Or old results suggesting superiority of diskless workstations

43 Ignoring Variability Often, showing only mean does not paint true picture of a system If variability is high, analysis and data presentation must make that clear

44 Overly Complex Analysis Choose simple explanations over complicated ones Choose simple experiments over complicated ones Choose simple explanations of phenomena Choose simple presentations of data

45 Improper Presentation of Results Real performance experiments are run to convince others –Or help others make decisions If your results don’t convince or assist, you haven’t done any good Often, manner of presenting results makes the difference

46 Ignoring Social Aspects Raw graphs, tables, and numbers aren’t always enough for many people Need clear explanations Present in terms your audience understands Be especially careful if going against audience’s prejudices!

47 Omitting Assumptions and Limitations Tell your audience about assumptions and limitations Assumptions and limitations are usually necessary in real performance study But: –Recognize them –Be honest about them, –Try to understand how they affect results

48 A Systematic Approach To Performance Evaluation 1. State goals and define the system 2. List services and outcomes 3. Select metrics 4. List parameters 5. Select factors to study 6. Select evaluation technique 7. Select workload

49 Systematic Approach, Cont. 8. Design experiments 9. Analyze and interpret data 10. Present results Usually, it is necessary to repeat some steps to get necessary answers

50 State Goals and Define the System Decide what you want to find out Put boundaries around what you’re going to consider This is a critical step, since everything afterwards is built on this foundation

51 List Services and Outcomes What services does system provide? What are possible results of requesting each service? Understanding these helps determine metrics

52 Select Metrics Metrics: criteria for determining performance Usually related to speed, accuracy, availability Studying inappropriate metrics makes a performance evaluation useless

53 List Parameters What system characteristics affect performance? System parameters Workload parameters Try to make it complete But expect you’ll add to list later

54 Select Factors To Study Factors: parameters that will be varied during study Best to choose factors most likely to be key to performance Also select levels for each factor Keep list and number of levels short –Because it directly affects work involved

55 Select Evaluation Technique Analytic modeling? Simulation? Measurement? Or perhaps some combination Right choice depends on many issues

56 Select Workload List of service requests to be presented to system For measurement, usually workload is list of actual requests to system Must be representative of real workloads!

57 Design Experiments How to apply chosen workload? How to vary factors to their various levels? How to measure metrics? All with minimal effort Pay attention to interaction of factors

58 Analyze and Interpret Data Data gathered from measurement tend to have a random component Must extract the real information from the random noise Interpretation is key to getting value from experiment –And is not at all mechanical –Do this well in your project

59 Present Results Make them easy to understand –For the broadest possible audience Focus on most important results Use graphics Give maximum information with minimum presentation Make sure you explain actual implications