1. http://dblab.usc.edu Announcements URL for the class is http://dblab.usc.edu/csci599 URL for the class is http://dblab.usc.edu/csci599 http://dblab.usc.edu/csci599 I will be away from September 5 to 16. There will be lectures: I will be away from September 5 to 16. There will be lectures:  Video-taped presentation on automatic system tuning.  Use these presentations to complete homework 1 (posted on the web site).  Design your project. Prof. Roger Zimmermann is our Guest Lecturer on September 28 th. Prof. Roger Zimmermann is our Guest Lecturer on September 28 th. Today’s paper: Today’s paper:  T. R. Andel and A. Yasinac. On the Credibility of Manet Simulations. IEEE Computer, July 2006.

2. 10 Commandments of Simulation Studies Shahram Ghandeharizadeh Computer Science Department University of Southern California

3. http://dblab.usc.edu Outline Why simulation studies? Why simulation studies? 10 Commandments of simulation studies. 10 Commandments of simulation studies. Review of each commandment. Review of each commandment. Conclusions Conclusions References References

4. http://dblab.usc.edu Why simulation models? To understand tradeoffs between alternative ways of implementing a functionality when: To understand tradeoffs between alternative ways of implementing a functionality when:  A realistic implementation is too expensive/time- consuming. Quantify the complexity of alternative implementations prior to a real system prototyping effort. Make sure it is not over- engineered: Many fold increase in software to get a 2% benefit? Quantify the complexity of alternative implementations prior to a real system prototyping effort. Make sure it is not over- engineered: Many fold increase in software to get a 2% benefit? Quantify tradeoffs with alternative strategies and choose the most appropriate one. Quantify tradeoffs with alternative strategies and choose the most appropriate one. Gain insight to develop better strategies. Gain insight to develop better strategies. Byte-hit versus Frequenc y-based. Identical Complexity. Byte-hit outperforms Frequency- based with larger configurations. Halo-clip and Halo-block.

5. http://dblab.usc.edu Why simulation models? There are other reasons for developing a simulation model that falls beyond the scope of this presentation: There are other reasons for developing a simulation model that falls beyond the scope of this presentation:  To describe and understand systems too complex to explain, e.g., the human brain.  To identify factors/conditions with most impact on a performance metric, e.g., airplane crash in Singapore during a hurricane. Typically conducted using traces gathered from a real environment.  Capacity planning and risk analysis, e.g., design of power grids in anticipation of 20% increase in peak load in 2007, war-games.

6. http://dblab.usc.edu Inherent limitation A simulation is abstraction of a real system that may either exist or foreseen to exist in the future. A simulation is abstraction of a real system that may either exist or foreseen to exist in the future. How do you know the abstraction level is correct? How do you know the abstraction level is correct?

7. http://dblab.usc.edu Inherent limitation A simulation is abstraction of a real system that may either exist or foreseen to exist in the future. A simulation is abstraction of a real system that may either exist or foreseen to exist in the future. How do you know the abstraction level is correct? How do you know the abstraction level is correct? 1. We cannot know what we do not know. 2. We can never be sure that we have accounted for all aspects that could affect a simulation model’s ability to provide meaningful results. 3. Items 1 and 2 are specially true for future foreseen applications that do not exist today.

8. http://dblab.usc.edu Alternatives to a simulation Real Prototypes/Implementation Simulation Study Analytical Models

9. http://dblab.usc.edu Alternatives to a simulation Real Prototypes/Implementation Simulation Study Analytical Models An implementation is specific and most often static. A simulation study can model a variety of possible system parameters. An implementation is specific and most often static. A simulation study can model a variety of possible system parameters. More Abstract More Specific

10. http://dblab.usc.edu Alternatives to a simulation Real Prototypes/Implementation Simulation Study Analytical Models A real prototype is more expensive to design, implement, test, and maintain. A real prototype is more expensive to design, implement, test, and maintain. Cheaper Expensive

11. http://dblab.usc.edu Alternatives to a simulation Real Prototypes/Implementation Simulation Study Analytical Models Analytical models are more general, providing insights that might be overlooked by a narrow implementation. Analytical models are more general, providing insights that might be overlooked by a narrow implementation. General Narrow

12. http://dblab.usc.edu Components of a Simulator 1. Abstraction of a computing environment. DMS’06: Mass storage devices to store clips, admission control, streaming of data as a function of time. DMS’06: Mass storage devices to store clips, admission control, streaming of data as a function of time. Manet: Physical and Data link layers. Manet: Physical and Data link layers. 2. Implementation of alternative strategies. DMS’06: Byte-hit and Frequency-based DMS’06: Byte-hit and Frequency-based Manet: Alternative network routing protocols such as DSR and AODV. Manet: Alternative network routing protocols such as DSR and AODV. 3. Abstractions of an application.  How requests are issued? Uniformly or in a bursty manner?  What processing is required from the system?

13. http://dblab.usc.edu 10 Commandments 1. Thou shall NOT obtain results from a computing environment unless its behavior is validated. 2. Thou shall NOT obtain results from an incorrect implementation of your strategies. 3. Thou shall NOT use unrealistic computing environments. 4. Thou shall NOT use unrealistic workloads or applications. 5. Thou shall NOT use someone else’s computing environment as the basis for your implementation unless you understand its abstractions and behavior for different parameter settings. 6. Thou shall NOT focus on absolute values. (Focus on the observed trends.)

14. http://dblab.usc.edu 10 Commandments (Cont…) 7. Thou shall NOT make a big deal out of differences lower than 15%. 8. Thou shall NOT report on a simulation with a few runs; establish statistical confidence. 9. Thou shall NOT publish results obtained from a simulator without disclosing all your assumptions. (Results must be reproducible.) 10. Thou shall NOT perceive simulation as an end in itself. (Validate your results against a real implementation.)

15. http://dblab.usc.edu 1. Do NOT obtain results from a computing environment unless its behavior is validated Limitation: Trends observed from simple flooding using Opnet, Network Simulator (NS-2), and Global Mobile Information Systems Simulation Library (GloMoSim) do NOT agree! Limitation: Trends observed from simple flooding using Opnet, Network Simulator (NS-2), and Global Mobile Information Systems Simulation Library (GloMoSim) do NOT agree! Claim: Claim:  Do you agree? What to do? What to do?

16. http://dblab.usc.edu 1. Do NOT obtain results from a computing environment unless its behavior is validated Limitation: Trends observed from simple flooding using Opnet, Network Simulator (NS-2), and Global Mobile Information Systems Simulation Library (GloMoSim) do NOT agree! Limitation: Trends observed from simple flooding using Opnet, Network Simulator (NS-2), and Global Mobile Information Systems Simulation Library (GloMoSim) do NOT agree! Claim: Claim:  Do you agree? What to do? What to do?

17. http://dblab.usc.edu 1. Do NOT obtain results from a computing environment unless its behavior is validated With simulation model of Byte-hit & Frequency-based, there are sanity checks to verify correctness of: With simulation model of Byte-hit & Frequency-based, there are sanity checks to verify correctness of:  Simulating time,  Usage of network bandwidth,  Reservation of bandwidth,  Generating requests based on a Zipfian distribution,  Maintaining information about assignment of clips to nodes. A few of these sanity checks are published at: A few of these sanity checks are published at:  http://dblab.usc.edu/Users/Shahram/DownloadPage.htm

18. http://dblab.usc.edu 2. Do NOT obtain results from an incorrect implementation of your strategies. Make sure your simulation implements desired functionality, e.g., Make sure your simulation implements desired functionality, e.g.,

19. http://dblab.usc.edu 2. Do NOT obtain results from an incorrect implementation of your strategies (Cont…) A strategy is realized at three different abstractions: A strategy is realized at three different abstractions:  Conceptual  This is a description of the strategy at a very abstract level. It is written in a paper.  Logical  This is a description of the strategy at a pseudo-code level. It contains data structures and one may perform complexity analysis on the strategy.  Physical  This is implementation of the strategy in a simulator (or a real system). It consists of actual code that one debugs and executes in a computing environment. This Testament applies to all 3 abstractions. This Testament applies to all 3 abstractions.

20. http://dblab.usc.edu 2. Do NOT obtain results from an incorrect implementation of your strategies (Solutions). Conceptual: Conceptual:  Ask colleagues to review your conceptual description of a strategy.  Do a literature search to see if others have considered strategies similar to yours in other disciplines.  Document the limitations of your strategy.  Be aware of your assumptions and document them. Physical: Physical:  Check your implementation at every opportunity by maintaining your sanity tests as an integral component of the simulator.  If the results from a run look strange, be able to run your sanity tests immediately.  Try to implement your strategies on a small scale using a prototype.  Validate simulation models against real-world implementation and environments.  Examine your parameter settings and ask are they real, goes to Testaments 3 and 4.  Be objective and if you have idle computing resource, setup a schedule to run all the possible parameter settings to VERIFY correctness of your strategy. Start with the extreme settings and their interactions. Based on the obtained insights, narrow the possibilities down to those that you speculate to be interesting. Verify the interesting experiments. Avoid surprised by verifying those experiments that you consider not interesting. For each, examine the behavior of your strategy and be able to explain it. If you canNOT explain the behavior of your strategy then something is wrong. One possibility is that your proposed strategy is not implemented correctly.

21. http://dblab.usc.edu 3. Do NOT use unrealistic computing environments.

22. http://dblab.usc.edu 4. Do NOT use unrealistic workloads or applications. Possible workloads: Possible workloads:  Synthetic: Generated using a random number generator.  Trace-driven: More realistic, obtained from an environment.  Hybrid: Derive a synthetic workload starting with a trace. Are synthetic and hybrid workload real? Are synthetic and hybrid workload real?

23. http://dblab.usc.edu 4. Do NOT use unrealistic workloads or applications. Possible workloads: Possible workloads:  Synthetic: Generated using a random number generator.  Trace-driven: More realistic, obtained from an environment.  Hybrid: Derive a synthetic workload starting with a trace. Are synthetic and hybrid workload real? Are synthetic and hybrid workload real? Are trace-driven workloads general purpose enough? Are trace-driven workloads general purpose enough? Solution: Use a trace-driven workload in combination with a synthetic/hybrid workload. Solution: Use a trace-driven workload in combination with a synthetic/hybrid workload.

24. http://dblab.usc.edu 5. Do NOT use someone else’s computing environment as the basis for your implementation unless you understand its abstractions and behavior for different parameter settings. If you decide to use a network simulator such as NS-2, make sure you understand its different parameter settings and their impact on obtained results: If you decide to use a network simulator such as NS-2, make sure you understand its different parameter settings and their impact on obtained results:  Relative ranking between AODV and DSR change depending on the parameter settings of GloMoSim.  Solution: Make sure you understand different parameters of a simulator and how they will impact your observed trends.

25. http://dblab.usc.edu 5. Do NOT use someone else’s computing environment as the basis for your implementation unless you understand its abstractions and behavior for different parameter settings. Make sure you understand a software package such as GloMoSim prior to using it: Make sure you understand a software package such as GloMoSim prior to using it:  Document all its parameter settings and how the simulator behaves.  When reading manuals and other’s technical papers, recreate their experiments to see if you can re-produce their results.  Run simple experiments to see if they are supported correctly.  An experiment is simple when its result is known in advance.

26. http://dblab.usc.edu 6. Do NOT focus on absolute values. (Focus on the observed trends.) Limitation: Limitation:

27. http://dblab.usc.edu 6. Do NOT focus on absolute values. (Focus must be on the observed trends.) Solution: Solution:  Describe behavior of protocol A and B for different parameter settings.  Focus on the observed trends with each protocol.  Identify the strengths and weaknesses of each protocol.  Be aware of technological trends.  Consider scenarios where your anticipated trends are outperformed/underperformed.  Do not make a big deal out of minor differences (less than 15% difference, see Testament 7).

28. http://dblab.usc.edu 7. Do NOT make a big deal out of differences lower than 15%. A simulator is an abstraction: A simulator is an abstraction: Small differences (less than 15%) are typically noise attributed to missing components, lack of detail. Small differences (less than 15%) are typically noise attributed to missing components, lack of detail.

29. http://dblab.usc.edu 8. Do NOT report on a simulation with a few runs; establish statistical confidence.

30. http://dblab.usc.edu Solution: Establish statistical confidence.

31. http://dblab.usc.edu Solution: Establish statistical confidence (Cont…)

32. http://dblab.usc.edu 9. Do NOT publish results obtained from a simulator without disclosing all your assumptions. Study of 114 peer-reviewed manet research papers published between 2000 to 2005. Study of 114 peer-reviewed manet research papers published between 2000 to 2005.

33. http://dblab.usc.edu 9. Do NOT publish results obtained from a simulator without disclosing all your assumptions (Cont…) Typical cause: Lack of space. Conference papers are limited in number of pages. Describing the simulator may leave no room for stating all assumptions. Typical cause: Lack of space. Conference papers are limited in number of pages. Describing the simulator may leave no room for stating all assumptions. Solution: Solution: 1. Write the complete paper as a technical report. 2. Summarize the key parameters in the conference/journal paper and reference the technical paper. 3. Make sure the technical paper is available on the web.

34. http://dblab.usc.edu 10. Do NOT perceive simulation as an end in itself. Simulation is an abstraction. Simulation is an abstraction.  An implementation highlights many important details (the devil is in the detail).  An analytical model provides further verification for the observations made using a simulator. Simulation is to quantify tradeoffs associated with alternative design strategies that should be investigated further. Simulation is to quantify tradeoffs associated with alternative design strategies that should be investigated further.

35. http://dblab.usc.edu Two Popular simulation models Closed simulation models: Closed simulation models:  A system consisting of N Servers,  M Terminals,  A terminal submits one job. It does not submit another job until the pending job completes.  A terminal may think/sleep between its requests. Open simulation models: Open simulation models:  Arrival rate for requests,  A system with a service time,  Queues form when requests arrive and the system is busy.

36. http://dblab.usc.edu Two Popular simulation models Closed simulation models: Closed simulation models:  Service time as a function of the number of: 1. servers, 2. terminals, 3. terminal think time. Open simulation models: Open simulation models:  Service time as a function of request arrival rate.

37. http://dblab.usc.edu Open simulation models Represented as A/S/n where A is the arrival process, S is the service time process and n is the number of servers. A and S may be: Represented as A/S/n where A is the arrival process, S is the service time process and n is the number of servers. A and S may be: 1. M (Markovian), Poisson exponential density 2. D (Deterministic), a fixed value 3. G (General), an arbitrary probability distribution. Popular queuing models: Popular queuing models: 1. M/M/1: A Poisson process generates both the arrival and service times. Key input parameters are arrival rates and service rates. 2. M/D/1: A Poisson process generates arrival times. Service time is deterministic because it is a fixed value. Key input parameters are arrival rates and a fixed service time.

38. http://dblab.usc.edu References T. R. Andel and A. Yasinac. On the Credibility of Manet Simulations. IEEE Computer, July 2006. T. R. Andel and A. Yasinac. On the Credibility of Manet Simulations. IEEE Computer, July 2006.On the Credibility of Manet SimulationsOn the Credibility of Manet Simulations S. Ghandeharizadeh, T. Helmi, T. Jung, S. Kapadia, and S. Shayandeh. An Evaluation of Two Policies for Placement of Continuous Media in Multi-hop Wireless Networks. USC Database Laboratory Technical Report Number 2006-03. Shorter version appeared in the Twelfth International Conference on Distributed Multimedia Systems (DMS 2006), Grand Canyon, Aug 30- Sept 1, 2006. S. Ghandeharizadeh, T. Helmi, T. Jung, S. Kapadia, and S. Shayandeh. An Evaluation of Two Policies for Placement of Continuous Media in Multi-hop Wireless Networks. USC Database Laboratory Technical Report Number 2006-03. Shorter version appeared in the Twelfth International Conference on Distributed Multimedia Systems (DMS 2006), Grand Canyon, Aug 30- Sept 1, 2006. Evaluation of Two Policies for Placement of Continuous Media in Multi-hop Wireless Networks Evaluation of Two Policies for Placement of Continuous Media in Multi-hop Wireless Networks R. Jain. The Art of Computer Systems Performance Analysis, John Wiley and Sons, 1991. R. Jain. The Art of Computer Systems Performance Analysis, John Wiley and Sons, 1991.