1. 1 PERF EVAL (CONT’D) H There are many other “tools of the trade” used in performance evaluation H Only a few will be mentioned here: –queueing theory –verification and validation –statistical analysis –multi-variate analysis –presentation of results

2. 2 Queueing Theory H A mathematical technique that specializes in the analysis of queues (e.g., customer arrivals at a bank, jobs arriving at CPU, I/O requests arriving at a disk subsystem) H General diagram: Customer Arrivals Departures Buffer Server

3. 3 Queueing Theory (cont’d) H The queueing system is characterized by: –Arrival process (M, G) –Service time (M, D, G) –Number of servers (1 to infinity) –Number of buffers (infinite or finite) H Example notation: M/M/1, M/D/1 H Example notation: M/M/, M/G/1/k 8

4. 4 Queueing Theory (cont’d) H There are well-known mathematical results for the mean waiting time and the number of customers in the system for several simple queueing models H E.g., M/M/1, M/D/1, M/G/1 H Example: M/M/1 –q = rho/ (1 - rho) where rho = lambda/mu < 1

5. 5 Queueing Theory (cont’d) H These simple models can be cascaded in series and in parallel to create arbitrarily large complicated queueing network models H Two main types: –closed queueing network model (finite pop.) –open queueing network model (infinite pop.) H Software packages exist for solving these types of models to determine steady-state performance (e.g., delay, throughput, util.)

6. 6 Verification and Validation H An important step in any modeling work (simulation or analytical) is convincing others that the model is “correct” H Verification: develop simple test cases with known inputs; compare to expected outputs H Validation: the “reality check” to see if model predictions agree with real world H Sanity checks (e.g., Little’s Law: N = T) H This V&V process is often overlooked!!!

7. 7 Statistical Analysis H “Math and stats can be your friends!!!” CW H There are lots of “standard” techniques from mathematics, probability, and statistics that are of immense value in performance work: –confidence intervals, null hypotheses, F-tests, T-tests, linear regression, least-squares fit, maximum likelihood estimation, correlation, time series analysis, transforms, Q-Q, EM... –working knowledge of commonly-observed statistical distributions

8. 8 Multi-Variate Analysis H For in-depth and really messy data analysis, there are multi-variate techniques that can be immensely helpful H In many cases, good data visualization tools will tell you a lot (e.g., plotting graphs), but in other cases you might try things like: –multi-variate regression: find out which parameters are relevant or not for curve fitting –ANOVA: analysis of variance can show the parameters with greatest impact on results

9. 9 Presentation of Results H Graphs and tables are the two most common ways of illustrating and/or summarizing data –graphs can show you the trends –tables provide the details H There are good ways and bad ways to do each of these H Again, it is a bit of an “art”, but there are lots of good tips and guidelines as well

10. 10 Table Tips H Decide if a table is really needed; if so, should it be part of main paper, or just an appendix? H Choose formatting software with which you are familiar; easy to import data, export tables H Table caption goes at the top H Clearly delineate rows and columns (lines) H Logically organize rows and columns H Report results to several significant digits H Be consistent in formatting wherever possible

11. 11 Graphing Tips H Choose a good software package, preferably one with which you are familiar, and one for which it is easy to import data, export graphs H Title at top; caption below (informative) H Labels on each axis, including units H Logical step sizes along axes (10’s, 100’s…) H Make sure choice of scale is clear for each axis (linear, log-linear, log-log) H Graph should start from origin (zero) unless there is a compelling reason not to do so

12. 12 Graphing Tips (cont’d) H Make judicious choice of type of plot –scatter plot, line graph, bar chart, histogram H Make judicious choice of line types –solid, dashed, dotted, lines and points, colours H If multiple lines on a plot, then use a key, which should be well-placed and informative H If graph is “well-behaved”, then organize the key to match the lines on the graph (try it!) H Be consistent from one graph to the next wherever possible (size, scale, key, colours)