CS533 Performance Evaluation Projects Overview CS533 Performance Evaluation Projects Overview

Your Projects This Semester Performance Measurement Warm-ups A series of small tasks to get you in the mode for measuring and reporting Follow closely the material in the text – “Network Performance” The Main Performance A large project following your own interests. Contains a series of steps to ensure success. These steps include: Project Pre-Proposal Project Proposal Design Proposal Project Report Projects Overview

Warm-Ups Students taking this course have usually had a strong interest in network performance. As such, these warm-up exercises give you a chance to practice measurement on networks. There are three warm-ups to be accomplished in the next few weeks: Using a network snoop device; Wireshark or tcpdump Measuring network performance Measuring application performance Warning – these warm-ups are undefined and open ended. Ground rules for all warm-up reports. Note – the measurement and the report both matter. The report should state clearly what’s been done. It should state the environment used. “Wireshark was used to measure the round-trip ping time as a function of physical distance between sites.” The report should include measurements where one thing depends on another – there MUST BE a graph in your report. There should be an indication of the accuracy of your measurements. What’s the uncertain in your values? Are they repeatable? If someone else were to measure the same thing, would they get the same result? Plan on delivering your report to the class; should be no more than 5 minutes in length. I will critique your results on the fly. Everyone will learn. Plan also on handing in your report – it can be a few slides for instance or it can be on paper. Projects Overview

Warm-Ups Does Correlation Imply Causation? Examples: There's a strong correlation between foot size and ability to do arithmetic. (Of course, it's not causation; both measured variables are strongly correlated with the age of the children. Older ones have bigger feet and better math ability.) People who drink diet soft drinks don't lose weight. In fact, they gain weight, a new study shows. (Best Theory: When we offer our bodies the sweet taste of diet drinks, but give them no calories, it makes us crave more calories.) The more firemen fighting a fire, the bigger the fire is observed to be. (Therefore firemen cause an increase in the size of a fire? ) As ice cream sales increase, the rate of drowning deaths increases sharply. (Therefore, ice cream consumption causes drowning.) Projects Overview

Warm-Ups Use of Performance Nomenclature: It is expected that by the time you finish each of these warm-ups, you will have made use of Performance concepts. It is SOOO easy to talk only about throughput, for instance, and ignore all the other nice concepts. But I’m expecting that you will talk about: Response Time Utilization Throughput Capacity/Bandwidth Projects Overview

Warm-Ups Ground rules for all warm-up reports. The WPI Acceptable Use Policy http://www.wpi.edu/+AUP clearly prohibits the use of sniffers on the network. This rule is in the Acceptable Use Policy to prevent denial of service conditions resulting from 'active' sniffing, which uses gratuitous ARP traffic to redirect packet flow, and to preserve privacy and uphold Federal law by not allowing students to accidentally wiretap a communications channel. There is a way to perform the measurement tests as described in these projects that does not violate the AUP. Students need to run their sniffers in non-promiscuous mode. This allows for collection of data in the sniffer that they would be processing anyways, without seeing any other traffic on the network. This solution avoids both potential problems outlined above. Here is how to use non-promiscuous mode for your tests. In Wireshark, that setting is a checkbox on the capture configuration screen. In tcpdump, that setting is the '-p' flag. Projects Overview

Environments and Tools In doing these warmups, we’re exploring three different “levels” of environments. Warmup #1 focusses on very LOW level environments – where we look at individual actions – for instance, individual network packets, individual disk accesses, individual process scheduling. We can study only ONE action – and get as much possible information as we can. Or we can look at multiple instances of this simple action, with a goal of getting more information about its performance. Warmup #2 looks at the next “higher” level of environment. An aggregate action – a stream of data made up of many packets, disk accesses over a span of time, and so on. Warmup #3 in intended to look at a complex environment. For instance, the disk activities that occur to a “user” environment, the network traffic coming from multiple sources, the car and truck traffic associated with an intersection, etc. These do not lend themselves to simple analysis, and we get only aggregate results. Once we’ve identified the environment, then we can select the tool. Low level environments require tools that can measure individual actions – Wireshark to watch an individual network packet, timing of individual left turns at an intersection. Medium level experiments probably won’t measure individual actions, but require tools that can measure throughput, response time, queueing time for a series of actions. High level, complex environments will perhaps need tools that understand the environment. For instance, a database in the cloud will support transactions made up of a number of smaller actions – it may involve network, disk, and other resources. Projects Overview

Warm-Up #1 Use a software snoop device to measure network traffic on a machine. What you need to know: Chapters 1 – 2 in “Network Performance” Class notes from the first lecture on measurement. In this project, you will install a sniffer on a machine of your choice and use it to measure low level network performance. A sniffer is a program running on a machine that is able to capture packets and packet information on that machine. In considering what machine to use, be aware that you will need generally administrative access to that hardware. Chapter 2 of “Network Performance” talks about tcpdump, windump, and Wireshark (formerly ethereal). I’m a Wireshark fan myself, but it doesn’t really matter which you use. Here’s your assignment: Measure some aspect of network traffic where you measure the same thing but change some one or two variables. Organize your results so they are easily understandable (presentation is everything!) including a plot or graph. Here are several possible examples – but I encourage you to think up your own: While Wireshark is running, connect to a website with your browser. How long does it take to complete the handshaking required to establish the connection. Now vary the distance to the site you’re connecting to. Does this change the connection time? Does the connection time depend on other things – like time of day, number of hops, etc. ( Remember that correlation does not imply correlation! What is the real cause of a dependency you find?) While Wireshark is running, ping a number of sites and determine the exact time it takes for that ping to complete (rather than the very general number reported by ping.) The ping time depends on what – distance, hops, etc??? Projects Overview

Warm-Up #1 Name______________________ Use a software snoop device to measure network traffic as seen by a node. Does the measurement accomplish the assignment? Uses a sniffer of some kind. Measures network traffic at a low level (and not via some mega-very-complicated-application high level interface). Uses a simple tool (browser, ping, or traceroute, for instance) to drive the traffic. Measurement Criteria: Is the experiment conducted such that there’s only one independent variable at a time? And that one independent variable leads to a change in the dependent variable – i.e., the dependent variable is really dependent? Are there several measurements under the same conditions – do we have the ability to assess the uncertainty in the results obtained? Is there discussion of significant figures, standard deviation and variation so we know how much we trust the numbers? Is there an analysis of why the variations seen in the numbers might occur? Is there a sanity check that the information presented makes sense? Is there an analysis that the dependence is due to causation and not just correlation? Is there a physical model presented that explains why the observed behavior occurs? The presentation: Is well organized. There’s a plot or graph presented. The visual information accurately represented what was measured. There was an in-class delivery and material given to me for further evaluation. Completed within the time limit.

Mid-Level Tools In Warm-Up #1, there was a low-level tool you used; Wireshark is an excellent example – it gives lots of very detailed information. Warm-Up #2 will use tools at the next higher level. On Windows these might be the Task Manager (fairly simple statistics) or the Performance Monitor that gives you detailed information. Neither of these give packet level details, but they provide information about aggregate network behavior. On LINUX, there are not such nice tools that come with the OS (at least, not that I’ve found.) There are numerous tools out there that you can download. Netstat and traceroute come with LINUX, but the command line interface isn’t much – you can get free GUI interfaces for them. Also take a look at: http://www.uperf.org/ http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html and many others. Note that there are two flavors of measurement here – and either is appropriate for your experiment: The measuring device is independent of the generator of traffic; it’s essentially standing “outside” of the generator and gathers what’s happening. The tool that generates the traffic also incorporates a means of measuring that traffic The network generation tool is usually designed to max out the connection between A and B. It’s a measurement of capacity, and will find a bottleneck somewhere along the path. This is more of a network measuring tool (fine for this Warmup) than a system or application tool. Projects Overview

Warm-Up #2 Measure Network Performance - throughput What you need to know: Chapters 4 – 12 in “Network Performance” Class notes from the first lecture on measurement. In this project, you will measure the performance of a network. This network might be a local LAN consisting of machines on the same switch, it could be a metropolitan network of machines near us, or it might span the globe – your choice. Here’s your assignment: Use several of the tools described on the previous slide to measure network traffic. As in Warm-Up #1, vary the properties of what you are measuring in order to get a variety of measurements, and again display and explain your results. Here is one possible example – but I encourage you to think up your own: 1. Determine the throughput between your machine and an off-site location as a function of the time of day. When is the best and worst time to use your machine as a source across the WPI network? Are your numbers consistent from day to day? What matters in getting good throughput? Your report should include the data analysis as you did in Warm-Up #1, (see Ground Rules) but in addition you should report on the relative advantages and disadvantages of the several tools you used.. Projects Overview

Measuring a Mid-Level Environment Warm-Up #2 Name______________________ Measuring a Mid-Level Environment Does the measurement accomplish the assignment? Uses a tool to measure traffic. Measures network traffic at a medium level (and not via some mega-very-complicated-application high level interface). Uses a simple tool to generate the traffic – perhaps same as measuring tool. Measurement Criteria: Is the experiment conducted such that there’s only one independent variable at a time? And that one independent variable leads to a change in the dependent variable – i.e., the dependent variable is really dependent? Are there several measurements under the same conditions – do we have the ability to assess the uncertainty in the results obtained? Is there discussion of significant figures, standard deviation and variation so we know how much we trust the numbers? Is there an analysis of why the variations seen in the numbers might occur? Is there a sanity check that the information presented makes sense? Is there an analysis that the dependence is due to causation and not just correlation? Is there a physical model presented that explains why the observed behavior occurs? The presentation: Is at the right level; just the right amount of detail – enough to explain what you did, but not so much you are giving TOO much data. There’s a plot or graph presented. The visual information accurately represented what was measured. There was an in-class delivery and material given to me for further evaluation. Completed within the time limit.

Warm-Up #3 Measure Application Performance What you need to know: Chapters 14 – 18 in “Network Performance” Class notes on simulation. In this project, you will determine the performance of an application. You can do that in one of two ways – with a traffic generator/benchmark/workload (Chpt. 16) or with a simulator (the other chapters) Here’s your assignment: Use a tool of your choosing to emulate the behavior a system would have. This “system” is more complex than the simple traffic patterns that you would get from the network measurements. This tool can be one you build, it can be one you’ve previously used (but “enhanced” to emulate a whole system) or could be a new tool. You are free to come up with an environment of interest to you. But as before, measurement and reporting are essential. Please try to DESIGN YOUR EXPERIMENT: There are a series of steps in Section 12 of the “Measurement and Statistics” Slides. Use the steps from there that are applicable. That section asks questions that will help you break down your project into do-able steps. Too many of us do an experiment and then say “that’s what I was planning on measuring all along.” This design process is extremely important. Your report should include the data analysis as you did in Warm-Up #2, (see Ground Rules) but in addition you should report on the relative advantages and disadvantages of the tool you used. Projects Overview

Measuring an Application Environment Warm-Up #3 Name______________________ Measuring an Application Environment Does the Design and Measurement accomplish the assignment? Shows a use of the 12-step design process. Uses a tool to measure traffic. Measures at an application level – a fairly complex environment. Show that the benchmark you are using is representative of a real environment. Measurement Criteria: Is the experiment conducted such that there’s only one independent variable at a time? And that one independent variable leads to a change in the dependent variable – i.e., the dependent variable is really dependent? Are there several measurements under the same conditions – do we have the ability to assess the uncertainty in the results obtained? Is there discussion of significant figures, standard deviation and variation so we know how much we trust the numbers? Is there an analysis of why the variations seen in the numbers might occur? Is there a sanity check that the information presented makes sense? Is there an analysis that the dependence is due to causation and not just correlation? Is there a physical model presented that explains why the observed behavior occurs? The presentation: The presentation is mailed to me by noon on the date it is due. Includes a discussion of the environment being studied. Is at the right level; Not too much or too little detail. There’s a plot or graph presented. The visual information represents what was measured. There was an in-class delivery and material given to me for further evaluation. Presentation Completed within the time limit.

The Main Performance Simulations: Choosing a Project. There are of course many great ideas for projects. Here’s my overview of these ideas. Simulations: I am delighted if you want to do a simulation.  This can be a simulation of computer behavior (of course), or it can be of any other phenomenon.    There are two conditions for the simulations I will accept: a)  A simulation based on some REAL data of a physical phenomenon.  You can use some already-available simulation tool for this.  BUT the tool must be able to predict a result that you didn't measure and you need to be able to verify this prediction. It is not acceptable to use an existing model with data you get from the web or anywhere else; you must measure it. EXAMPLE:  You measure a traffic intersection.  You know the timing of all the lights, you know traffic patterns; you know for each minute of a two-hour period how many cars are queued waiting for a left turn, etc.  You know EVERYTHING about that intersection.  You put that data into an existing simulation package and tune it to macth your intersection.  THEN you give it the light patterns and basic properties for a second intersection - now the model is able to successfully predict the traffic patterns of this second intersection.  That's how you know the model is correct. Projects Overview

The Main Performance Simulations (Continued): Choosing a Project. a)  You build the simulation model.  You need to collect some data for this, but I’m much less concerned about the extend of that data – you’ve doing a lot of hard work building the model, so that makes up for the lesser time spent measuring. EXAMPLE:  On the following pages, I give two examples of models built by former students; one build a monopoly game – it had a mechanism for moving the pieces around the board, included buying and selling of properties, for Chance and Community Chest, etc. Another student built a network model including several routers and nodes and determined what happens as traffic increases in this environment. Projects Overview

The Main Performance Measurements: Choosing a Project. People almost always measure something that’s computer or network related; it’s so much easier to determine the performance characteristics of such things. You get to use all the things you’ve learned this semester. Talking about the “performance of weather” or the “performanceof responses to a questionnaire” while theoretically possible, I suppose, are pretty hard to do in practice. Usually in this case your project would be a bigger example of the kind of thing you did for your warm-ups. It would follow the twelve-step process, and would be of a bigger scope and in more detail than you did for those warm-ups. Here’s a list of measurement tools that people have used in the past: Tracing and Tools Profiling Project in Eclipse, Iometer, Perfmon, Tcpdump, Wireshark, Iperf Simgrid  simgrid.gforge.inria.fr, NetworkTrafficGenerator (ntg), Network Simulator (ns) Projects Overview

The Main Performance Choosing a Project. Projects From Last Year Ray Tracing Performance: A study of the factors affecting the performance of a program rendering a three dimensional object.  Evaluating Virtual Machine Software Performance: Compares the performance of two virtual machines as well as native OS running a number of benchmarks on Linux and Windows. Performance Analysis of Common Security Tools: Looks at the hardware and network performance of two products. Slingbox: Analyzing the performance of the video streaming of SlingBox under different network settings. Traffic Simulation: Wrote a simulation model (from scratch) and used it to understand automobile traffic patterns. Monopoly Simulation: Wrote a simulation model (from scratch) for the game of monopoly. This did not use much real data, but presented a major challenge in modeling the various aspects of the game. Projects Overview

The Main Performance Choosing a Project. Projects From Last Year Web Server Performance: What factors affect the performance of Apache Tomcat. Home WLAN Performance: Studies factors affecting the performance of home networks. Mesh Networks: Measured and analyzed the performance of networks of computers. Hadoop: A study of the measured performance of a “big data” database. Data Base Behavior: Performance factors affecting the measured performance of data base updates. Mobile Hard Drive: Measured the performance of a hard drive under a large number of conditions. Projects Overview

The Main Performance Project Pre-Proposal It is assumed that you will choose a topic an aspect of performance that is of interest to you or that you can use in your job. The project contains several components as shown below. The due dates for these components are given on the main syllabus page. Project Pre-Proposal Goal: To get agreement on a topic in an informal way. Deliverable: Here you and I have a conversation/e-mail/brief write-up that contains a very short statement of what you want to do for a project. I Owe You: Very rapid feedback that the direction you have chosen is OK. What To Look For: Major issues I will check for include: Scope/size - does the project you are proposing contain the appropriate amount of work? Diversity – does the project contain several components such as measurement, simulation, etc. from the topics we are discussing in class. Projects Overview

The Main Performance Project Proposal Design Proposal Goal: To allow you to write up your project and present it to the class. Deliverable: You will prepare and deliver a 10 minute class presentation. You will also have a written proposal. It’s the written proposal that will be evaluated for a grade. What’s Included: There are a number of items that should be here. They include: A description of the performance project you will do. What performance components are included in this project. An explanation of how this fits into some larger context. A justification of why this is important and/or useful. Design Proposal Goal: To give a formal progress report on your project. By this time, you have gone down a number of dead-ends, figured out a number of details, and now have a project that’s quite a bit different from when you made your proposal. We want to hear where you’re at. Deliverable: You will prepare and deliver a 10 minute class presentation. You will also have a written proposal. It’s the written proposal that will be evaluated for a grade. What’s Included: There are a number of items that should be here. They include: A brief reminder of the performance project you are doing. What performance components you have used thus far. How your project has changed, and why (ways we have gone wrong can be very educational.) Difficulties ahead. Projects Overview

Design Proposal What you should report during this presentation: Goal: To give a status report on your project. We want to hear what you’ve done so far. . Deliverable: You will prepare and deliver a 15 (FIFTEEN) minute class presentation. What’s Included: There are a number of items that should be here. They include: A description of the performance project you are doing. Focus on the PERFORMANCE aspects and less on the network, etc. aspects. Please avoid “ratholes.” A justification that this is a PERFORMANCE project and not some warmed over project from another course. The Performance Engineering techniques/practices you are using in this project. Current progress: a) Portion of the goal achieved so far. b) Portion still to be done. 5. A description of: a) The performance environment (what was happening on the machine in terms of disk accesses, CPU usage, etc.) b) Performance tools used. c) Performance models employed. Projects Overview

Project Report Goal: To give a formal report on your project. We want to hear what you’ve done and how you’ve done it. Deliverable: You will prepare and deliver a 30 (thirty) minute class presentation. You will also have a written report. It’s the written report that will be evaluated for a grade. What’s Included: There are a number of items that should be here: A brief reminder of the performance project you are doing. Functionality (how your product works) ONLY to the extent absolutely necessary to explain the performance work you've done. Try to avoid rat-holes please. Items from previous reports you’ve done that help us understand the context of your project. The Performance Engineering techniques/practices you used in completing your project. The best way to do this is to fit your methodology into the 12-step process we’ve discussed. Show the performance measurements or performance practices that you used. This includes models, tools and techniques. A description of how much you trust your results. Show IN DETAIL the statistical analysis you perform to show your trust. What did you learn from this project? Your report may well have additional topics. Projects Overview