1. 10-1 ©2006 Raj Jain www.rajjain.com The Art of Data Presentation

2. 10-2 ©2006 Raj Jain www.rajjain.com Overview  Types of Variables  Guidelines for Preparing Good Charts  Common Mistakes in Preparing Charts  Pictorial Games  Special Charts for Computer Performance  Gantt Charts  Kiviat Graphs  Schumacher Charts  Decision Maker’s Games

3. 10-3 ©2006 Raj Jain www.rajjain.com Types of Variables  Type of computer: Super computer, minicomputer, microcomputer  Type of Workload: Scientific, engineering, educational  Number of processors  Response time of system

4. 10-4 ©2006 Raj Jain www.rajjain.com Guidelines for Preparing Good Charts  Require minimum effort from the reader Direct labeling vs. legend box  Maximize Information: Words in place of symbols Cleary label the axes

5. 10-5 ©2006 Raj Jain www.rajjain.com Guidelines (cont)  Minimize Ink: No grid lines, more details  Use Commonly accepted practices: origin at (0,0) Independent variable (cause) along x axis, linear scales, increasing scales, equal divisions  Avoid ambiguity: Show coordinate axes, scale divisions, origin. Identify individual curves and bars.  See checklist in Box 10.1

6. 10-6 ©2006 Raj Jain www.rajjain.com Common Mistakes in Preparing Charts  Presenting too many alternatives on a single chart Max 5 to 7 messages => Max 6 curves in a line charts, no more than 10 bars in a bar chart, max 8 components in a pie chart  Presenting many y variables on a single chart

7. 10-7 ©2006 Raj Jain www.rajjain.com Common Mistakes in Charts (Cont)  Using symbols in place of text  Placing extraneous information on the chart: grid lines, granularity of the grid lines  Selecting scale ranges improperly: automatic selection by programs may not be appropriate

8. 10-8 ©2006 Raj Jain www.rajjain.com Common Mistakes in Charts (Cont)  Using a line chart in place of column chart: line => Continuity CPU Type 8000810083008200 MIPS

9. 10-9 ©2006 Raj Jain www.rajjain.com Pictorial Games  Using non-zero origins to emphasize the difference Three quarter high-rule => height/width > 3/4

10. 10-10 ©2006 Raj Jain www.rajjain.com Pictorial Games (Cont)  Using double-whammy graph for dramatization Using related metrics

11. 10-11 ©2006 Raj Jain www.rajjain.com Pictorial Games (Cont)  Plotting random quantities without showing confidence intervals

12. 10-12 ©2006 Raj Jain www.rajjain.com Pictorial Games (Cont)  Pictograms scaled by height Mine Performance = 2 Yours Performance = 1

13. 10-13 ©2006 Raj Jain www.rajjain.com Pictorial Games (Cont)  Using inappropriate cell size in histograms [0,2)[2,4)[4,6)[6,8)[8,10)[10,12)[0,6)[6,12) Response Time Frequency 2 4 6 8 10 12 0 2 4 6 8 10 12 0

14. 10-14 ©2006 Raj Jain www.rajjain.com Pictorial Games (Cont)  Using broken scales in column charts A System Resp. Time 2 4 6 8 10 12 0 BCDE F A System Resp. Time 0 BCDE F 9 10 11 12

15. 10-15 ©2006 Raj Jain www.rajjain.com Special Charts for Computer Performance  Gantt charts  Kiviat Graphs  Schumacher's charts

16. 10-16 ©2006 Raj Jain www.rajjain.com Gantt Charts  Shows relative duration of a number of conditions CPU IO Channel Network 20%40%60%80%100%0% 60 20 3010515 Utilization

17. 10-17 ©2006 Raj Jain www.rajjain.com Example: Data for Gantt Chart

18. 10-18 ©2006 Raj Jain www.rajjain.com Draft of the Gantt Chart

19. 10-19 ©2006 Raj Jain www.rajjain.com Final Gantt Chart

20. 10-20 ©2006 Raj Jain www.rajjain.com Kiviat Graphs  Radial chart with even number of metrics  HB and LB metrics alternate  Ideal shape: star CPU Busy CPU in Supervisor State CPU in Problem State CPU Wait Any Channel Busy Channel only Busy CPU/Channel Overlap CPU Only Busy

21. 10-21 ©2006 Raj Jain www.rajjain.com Kiviat Graph for a Balanced System  Problem: Inter-related metrics CPU busy = problem state + Supervisor state CPU wait = 100 – CPU busy Channel only – any channel –CPU/channel overlap CPU only = CPU busy – CPU/channel overlap CPU Busy CPU in Supervisor State CPU in Problem State CPU Wait Any Channel Busy Channel only Busy CPU/Channel Overlap CPU Only Busy

22. 10-22 ©2006 Raj Jain www.rajjain.com Shapes of Kiviat Graphs CPU Keel boatI/O WedgeI/O Arrow

23. 10-23 ©2006 Raj Jain www.rajjain.com Merrill’s Figure of Merit (FoM)  Performance = {x 1, x 2, x 3, …, x 2n } Odd values are HB and even values are LB  x 2n+1 is the same as x 1  Average FOM = 50%

24. 10-24 ©2006 Raj Jain www.rajjain.com Example: FoM  System A:

25. 10-25 ©2006 Raj Jain www.rajjain.com FoM Example (Cont)  System B: System B has a higher figure of merit and it is better.

26. 10-26 ©2006 Raj Jain www.rajjain.com Figure of Merit: Known Problems  All axes are considered equal  Extreme values are assumed to be better  Utility is not a linear function of FoM  Two systems with the same FoM are not equally good.  System with slightly lower FoM may be better

27. 10-27 ©2006 Raj Jain www.rajjain.com Kiviat Graphs For Other Systems  Networks: Application Throughput Packets With Error Implicit Acks Duplicate Packets Link Utilization Link Overhead

28. 10-28 ©2006 Raj Jain www.rajjain.com Schumacher Charts  Performance matrix are plotted in a tabular manner  Values are normalized with respect to long term means and standard deviations  Any observations that are beyond mean  one standard deviation need to be explained  See Figure 10.25 in the book

29. 10-29 ©2006 Raj Jain www.rajjain.com Performance Analysis Rat Holes ConfigurationWorkloadMetricsDetails

30. 10-30 ©2006 Raj Jain www.rajjain.com Reasons for not Accepting an Analysis  This needs more analysis.  You need a better understanding of the workload.  It improves performance only for long IOs/packets/jobs/files, and most of the IOs/packets/jobs/files are short.  It improves performance only for short IOs/packets/jobs/files, but who cares for the performance of short IOs/packets/jobs/files, its the long ones that impact the system.  It needs too much memory/CPU/bandwidth and memory/CPU/bandwidth isn't free.  It only saves us memory/CPU/bandwidth and memory/CPU/bandwidth is cheap. See Box 10.2 on page 162 of the book for a complete list

31. 10-31 ©2006 Raj Jain www.rajjain.com Summary 1.Qualitative/quantitative, ordered/unordered, discrete/continuous variables 2.Good charts should require minimum effort from the reader and provide maximum information with minimum ink 3.Use no more than 5-6 curves, select ranges properly, Three- quarter high rule 4.Gantt Charts show utilizations of various components 5.Kiviat Graphs show HB and LB metrics alternatively on a circular graph 6.Schumacher Charts show mean and standard deviations 7.Workload, metrics, configuration, and details can always be challenged. Should be carefully selected.

32. 10-32 ©2006 Raj Jain www.rajjain.com Ratio Games

33. 10-33 ©2006 Raj Jain www.rajjain.com Overview  Ratio Game Examples  Using an Appropriate Ratio Metric  Using Relative Performance Enhancement  Ratio Games with Percentages  Ratio Games Guidelines  Numerical Conditions for Ratio Games

34. 10-34 ©2006 Raj Jain www.rajjain.com Using an Appropriate Ratio Metric 1.Throughput: A is better 2.Response Time: A is worse 3.Power (Ratio): A is better  could be a contradictory conclusion Example:

35. 10-35 ©2006 Raj Jain www.rajjain.com Using Relative Performance Enhancement  Example: Two floating point accelerators  Problem: Incomparable bases. Need to try both on the same machine

36. 10-36 ©2006 Raj Jain www.rajjain.com Ratio Games with Percentages  Example: Tests on two systems 1. System B is better on both systems 2. System A is better overall. System A: System B:

37. 10-37 ©2006 Raj Jain www.rajjain.com Ratio Games Guidelines 1. If one system is better on all benchmarks, contradicting conclusions can not be drawn by any ratio game technique

38. 10-38 ©2006 Raj Jain www.rajjain.com Guidelines (cont) 2. Even if one system is better than the other on all benchmarks, a better relative performance can be shown by selecting appropriate base.  In the previous example, System A is 40% better than System B using raw data, 43% better using system A as a base, and 42% better using System B as a base. 3.If a system is better on some benchmarks and worse on others, contracting conclusions can be drawn in some cases. Not in all cases. 4.If the performance metric is an LB metric, it is better to use your system as the base 5.If the performance metric is an HB metric, it is better to use your opponent as the base 6.Those benchmarks that perform better on your system should be elongated and those that perform worse should be shortened

39. 10-39 ©2006 Raj Jain www.rajjain.com Numerical Conditions for Ratio Games  A is better than B iff  Raw Data:  With A as the Base:

40. 10-40 ©2006 Raj Jain www.rajjain.com Numerical Conditions (Cont) Ratio of B/A response on benchmark i Ratio of B/A response on benchmark j 1 2 0 11231 Raw Data Base B Base A A is better using all 3 B is better using all 3

41. 10-41 ©2006 Raj Jain www.rajjain.com Summary  Ratio games arise from use of incomparable bases  Ratios may be part of the metric  Relative performance enhancements  Percentages are ratios  For HB metrics, it is better to use opponent as the base