1. Workloads Experimental environment prototype real sys exec- driven sim trace- driven sim stochastic sim Live workload Benchmark applications Micro- benchmark programs Synthetic benchmark programs Traces Distributions & other statistics monitor analysis generator Synthetic traces “Real” workloads Made-up © 2003, Carla Ellis Data sets You are here

2. © 2003, Carla Ellis Using Traces The issue of feedback: Is the request stream invariant, regardless of timing of responses? –http requests – will user just move on to some other page and not complete the viewing of the requested page if the response is slowed down? –Will requests be repeated in response to slower replies? S.U.T. Request Generator Request stream Responses Missing or delayed Client (User) Web server

3. © 2003, Carla Ellis Using Traces The issue of feedback: Is the request stream invariant, regardless of timing of responses? –What if cache behavior changes and different memory requests are hits/misses than before? S.U.T. Request Generator Request stream Responses reordered Out- of- Order Processor Cache & Memory System

4. © 2003, Carla Ellis Using Traces The issue of feedback: Is the request stream invariant, regardless of timing of responses? –What if cache behavior changes and different memory requests are hits/misses than before? –What if messages return in a different order from various distributed sources? S.U.T. Request Generator Request stream Responses reordered Client Message-based distributed system

5. © 2003, Carla Ellis Using Traces The issue of feedback: Is the request stream invariant, regardless of timing of responses? –What if block size is halved and twice as many requests are needed to fetch same data stream? S.U.T. Request Generator Request stream Responses File System Disk Storage System Finer grain

6. © 2003, Carla Ellis Using Traces The issue of feedback: Is the request stream invariant, regardless of timing of responses? –Merging concurrent streams into one. What if there are data dependencies between them? All streams may depend on relative interleaving. S.U.T. Request Generator Request stream Responses Reordered or values changed Client Server

7. © 2003, Carla Ellis Using Traces Mismatch of traces and S.U.T. S.U.T. Request Generator Request Generator Client browser Web server Potential trace collection points

8. © 2003, Carla Ellis Using Traces Mismatch of traces and S.U.T. Which traces fit new scenario better? S.U.T. Request Generator Request Generator Client browser Web server Proxy cache

9. © 2003, Carla Ellis Advantages of Traces Repeatable –Interactive usage is one tricky area Availability and community acceptance (convenience) Representative of real behavior even when you can’t identify and isolate the programs/processes generating that behavior –Message traffic in the middle of a network But even full traces aren’t perfectly predictive.

10. © 2003, Carla Ellis Dealing with Huge Traces Compression, filtering, sampling Filtered caches: eliminating “redundant” entries –Intuitive idea is that if a reference hits in a small direct mapped cache then it will hit in a larger cache that subsumes the smaller one. The number of misses will be the same with the filtered trace (while potentially discarding ~90% of references) compared to the full trace. –But what happens if you change, say, the line size, context switching behavior?

11. © 2003, Carla Ellis Filtered Traces A B C C C C D C D C D D E F A B C D E F

12. © 2003, Carla Ellis Dealing with Huge Traces Sampled traces: sequence of “samples” – each of which is a string of n consecutive references where n is sample size – separated by a fixed number of references that are ignored. Sampling ratio is % used. Provide only a statistical estimate of properties of full trace. What is the state of the system (caches) at the start of each interval (unknown refs)? –stitch – assume it’s the same as at the end of last interval –Use some number of refs at beginning of each interval to prime the cache –INITMR -- estimate misses via probability at beginning of each interval

13. © 2003, Carla Ellis Sampled Traces A B C C C C D C D C D D E F A B C | C D C For stitch, hit, miss

14. © 2003, Carla Ellis Justifying Use of Traces Changes you are going to make in the S.U.T do not change the request stream in any substantial way –Data dependencies (e.g, from concurrency) –Timing dependencies –Order dependencies The traces are appropriate to the S.U.T. Reductions applied to traces don’t introduce significant errors in the results. Depends on choice of metrics (relative trends vs. absolute metrics) and S.U.T. variations.

15. Workloads Experimental environment prototype real sys exec- driven sim trace- driven sim stochastic sim Live workload Benchmark applications Micro- benchmark programs Synthetic benchmark programs Traces Distributions & other statistics monitor analysis generator Synthetic traces “Real” workloads Made-up © 2003, Carla Ellis Data sets Fake user generator

16. Vague idea “groping around” experiences Hypothesis Initial observations Discussion: Destination Initial Hypothesis Pre-proposal 1: Sketch out what information you would need to collect (or have already gathered) in a “groping around” phase to get from a vague idea to the hypothesis stage for your planned project © 2003, Carla Ellis