We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byCordelia Hamilton
Modified about 1 year ago
Copyright © 2005 Department of Computer Science CPSC 641 Winter 20111 PERFORMANCE EVALUATION Often in Computer Science you need to: – demonstrate that a new concept, technique, or algorithm is feasible –demonstrate that a new method is better than an existing method –understand the impact of various factors and parameters on the performance, scalability, or robustness of a system
Copyright © 2005 Department of Computer Science CPSC 641 Winter 20112 PERFORMANCE EVALUATION There is a whole field of computer science called computer systems performance evaluation that is devoted to exactly this One classic book is Raj Jain’s “The Art of Computer Systems Performance Analysis”, Wiley & Sons, 1991 Much of what is outlined in this presentation is described in more detail in [Jain 1991]
Copyright © 2005 Department of Computer Science CPSC 641 Winter 20113 PERF EVAL: THE BASICS There are three main methods used in the design of performance evaluation studies: Analytic approaches –the use of mathematics, Markov chains, queueing theory, Petri Nets, abstract models… Simulation approaches –design and use of computer simulations and simplified models to assess performance Experimental approaches –measurement and use of a real system
Copyright © 2005 Department of Computer Science CPSC 641 Winter 20114 Analytical Example: Queueing Theory Queueing theory is a mathematical technique that specializes in the analysis of queues (e.g., customer arrivals at a bank, jobs arriving at CPU, I/O requests arriving at a disk subsystem, lineup at Tim Hortons) General diagram: Customer Arrivals Departures Buffer Server
Copyright © 2005 Department of Computer Science CPSC 641 Winter 20115 Queueing Theory (cont’d) The queueing system is characterized by: –Arrival process (M, G) –Service time process (M, D, G) –Number of servers (1 to infinity) –Number of buffers (infinite or finite) Example notation: M/M/1, M/D/1 Example notation: M/M/, M/G/1/k 8
Copyright © 2005 Department of Computer Science CPSC 641 Winter 20116 Queueing Theory (cont’d) There are well-known mathematical results for the mean waiting time and the number of customers in the system for several simple queueing models E.g., M/M/1, M/D/1, M/G/1 Example: M/M/1 –q = rho/ (1 - rho) where rho = lambda/mu < 1
Copyright © 2005 Department of Computer Science CPSC 641 Winter 20117 Queueing Theory (cont’d) These simple models can be cascaded in series and in parallel to create arbitrarily large complicated queueing network models Two main types: –closed queueing network model (finite pop.) –open queueing network model (infinite pop.) Software packages exist for solving these types of models to determine steady-state performance (e.g., delay, throughput, util.)
Copyright © 2005 Department of Computer Science CPSC 641 Winter 20118 Simulation Example: TCP Throughput Can use an existing simulation tool, or design and build your own custom simulator Example: ns-2 network simulator A discrete-event simulator with detailed TCP protocol models Configure network topology and workload Run simulation using pseudo-random numbers and produce statistical output
Copyright © 2005 Department of Computer Science CPSC 641 Winter 20119 OTHER ISSUES Simulation run length –choosing a long enough run time to get statistically meaningful results (equilibrium) Simulation start-up effects and end effects –deciding how much to “chop off” at the start and end of simulations to get proper results Replications –ensure repeatability of results, and gain greater statistical confidence in the results given
Copyright © 2005 Department of Computer Science CPSC 641 Winter 201110 Experimental Example: Benchmarking The design of a performance study requires great care in experimental design and methodology Need to identify –experimental factors to be tested –levels (settings) for these factors –performance metrics to be used –experimental design to be used
Copyright © 2005 Department of Computer Science CPSC 641 Winter 201111 FACTORS Factors are the main “components” that are varied in an experiment, in order to understand their impact on performance Examples: request rate, request size, read/write ratio, num concurrent clients Need to choose factors properly, since the number of factors affects size of study
Copyright © 2005 Department of Computer Science CPSC 641 Winter 201112 LEVELS Levels are the precise settings of the factors that are to be used in an experiment Examples: req size S = 1 KB, 10 KB, 1 MB Example: num clients C = 10, 20, 30, 40, 50 Need to choose levels realistically Need to cover useful portion of the design space
Copyright © 2005 Department of Computer Science CPSC 641 Winter 201113 PERFORMANCE METRICS Performance metrics specify what you want to measure in your performance study Examples: response time, throughput, pkt loss Must choose your metrics properly and instrument your experiment accordingly
Copyright © 2005 Department of Computer Science CPSC 641 Winter 201114 EXPERIMENTAL DESIGN Experimental design refers to the organizational structure of your experiment Need to methodically go through factors and levels to get the full range of experimental results desired There are several “classical” approaches to experimental design
Copyright © 2005 Department of Computer Science CPSC 641 Winter 201115 EXAMPLES One factor at a time –vary only one factor through its levels to see what the impact is on performance Two factors at a time –vary two factors to see not only their individual effects, but also their interaction effects, if any Full factorial –try every possible combination of factors and levels to see full range of performance results
Copyright © 2005 Department of Computer Science CPSC 641 Winter 201116 SUMMARY Computer systems performance evaluation defines standard methods for designing and conducting performance studies Great care must be taken in experimental design and methodology if the experiment is to achieve its goal, and if results are to be fully understood We will see examples of these methodologies and their applications over the next 3 months
1 PERFORMANCE EVALUATION H Often in Computer Science you need to: – demonstrate that a new concept, technique, or algorithm is feasible –demonstrate that.
1 PERFORMANCE EVALUATION H Often one needs to design and conduct an experiment in order to: – demonstrate that a new technique or concept is feasible –demonstrate.
1 Experimental Methodology H Experimental methods can be used to: – demonstrate that a new concept, technique, or algorithm is feasible –demonstrate that.
CPSC 531: Experiment Design1 CPSC 531: Experiment Design and Performance Evaluation Instructor: Anirban Mahanti Office: ICT 745
1 PERF EVAL (CONT’D) H There are many other “tools of the trade” used in performance evaluation H Only a few will be mentioned here: –queueing theory –verification.
OPERATING SYSTEMS CS 3530 Summer 2014 Systems and Models Chapter 03.
OS Fall ’ 02 Performance Evaluation Operating Systems Fall 2002.
Chapter 3 System Performance and Models Introduction A system is the part of the real world under study. Composed of a set of entities interacting.
Queueing Models and Ergodicity. 2 Purpose Simulation is often used in the analysis of queueing models. A simple but typical queueing model: Queueing models.
1 Queueing Theory H Plan: –Introduce basics of Queueing Theory –Define notation and terminology used –Discuss properties of queuing models –Show examples.
Ó 1998 Menascé & Almeida. All Rights Reserved.1 Part VI System-level Performance Models for the Web (Book, Chapter 8)
ICOM 6115: Computer Systems Performance Measurement and Evaluation August 11, 2006.
Chapter 3 System Performance and Models. 2 Systems and Models The concept of modeling in the study of the dynamic behavior of simple system is be able.
Ó 1998 Menascé & Almeida. All Rights Reserved.1 Part V Workload Characterization for the Web.
Chapter 12: Simulation and Modeling Invitation to Computer Science, Java Version, Third Edition.
Computer Simulation of Networks ECE/CSC 777: Telecommunications Network Design Fall, 2013, Rudra Dutta.
Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 1 Introduction to Experiment Design Shiv Kalyanaraman Rensselaer Polytechnic Institute
Queueing Theory: Recap
1 Performance Evaluation of Computer Systems and Networks Introduction, Outlines, Class Policy Instructor: A. Ghasemi Many thanks to Dr. Behzad Akbari.
ISCSI Performance in Integrated LAN/SAN Environment Li Yin U.C. Berkeley.
Modeling and Performance Evaluation of Network and Computer Systems Introduction (Chapters 1 and 2) 10/4/2015H.Malekinezhad1.
Queuing Theory Basic properties, Markovian models, Networks of queues, General service time distributions, Finite source models, Multiserver queues Chapter.
Dynamic and Decentralized Approaches for Optimal Allocation of Multiple Resources in Virtualized Data Centers Wei Chen, Samuel Hargrove, Heh Miao, Liang.
1 OUTPUT ANALYSIS FOR SIMULATIONS. 2 Introduction Analysis of One System Terminating vs. Steady-State Simulations Analysis of Terminating Simulations.
NETE4631: Network Information System Capacity Planning (2) Suronapee Phoomvuthisarn, Ph.D. /
1 Petri Nets H Plan: –Introduce basics of Petri Net models –Define notation and terminology used –Show examples of Petri Net models u Calaway Park model.
Ó 1998 Menascé & Almeida. All Rights Reserved.1 Part VIII Concluding Remarks.
Chapter 12: Simulation and Modeling
1 Performance Evaluation of Computer Networks Objectives Introduction to Queuing Theory Little’s Theorem Standard Notation of Queuing Systems Poisson.
1 Elements of Queuing Theory The queuing model –Core components; –Notation; –Parameters and performance measures –Characteristics; Markov Process –Discrete-time.
Web Server Benchmarking Using the Internet Protocol Traffic and Network Emulator Carey Williamson, Rob Simmonds, Martin Arlitt et al. University of Calgary.
1 PerfCenter and AutoPerf: Tools and Techniques for Modeling and Measurement of the Performance of Distributed Applications Varsha Apte Faculty Member,
Performance Analysis of Real Traffic Carried with Encrypted Cover Flows Nabil Schear David M. Nicol University of Illinois at Urbana-Champaign Department.
1 Wenguang WangRichard B. Bunt Department of Computer Science University of Saskatchewan November 14, 2000 Simulating DB2 Buffer Pool Management.
Simulation enables the study of complex system. Simulation is a good approach when analytic study of a system is not possible or very complex. Informational,
Data Communication and Networks Lecture 13 Performance December 9, 2004 Joseph Conron Computer Science Department New York University
Ó 1998 Menascé & Almeida. All Rights Reserved.1 Part V Workload Characterization for the Web (Book, chap. 6)
Introduction to Discrete Event Simulation Customer population Service system Served customers Waiting line Priority rule Service facilities Figure C.1.
Measuring the Capacity of a Web Server USENIX Sympo. on Internet Tech. and Sys. ‘ Koo-Min Ahn.
ECE 466/658: Performance Evaluation and Simulation Introduction Instructor: Christos Panayiotou.
NETE4631:Capacity Planning (2)- Lecture 10 Suronapee Phoomvuthisarn, Ph.D. /
Copyright © 1998 Wanda Kunkle Computer Organization 1 Chapter 2.1 Introduction.
1 On Dynamic Parallelism Adjustment Mechanism for Data Transfer Protocol GridFTP Takeshi Itou, Hiroyuki Ohsaki Graduate School of Information Sci. & Tech.
1 Part VI System-level Performance Models for the Web © 1998 Menascé & Almeida. All Rights Reserved.
1 Evaluation of Cooperative Web Caching with Web Polygraph Ping Du and Jaspal Subhlok Department of Computer Science University of Houston presented at.
Modeling and Simulation Queuing theory
Building a Parallel File System Simulator E Molina-Estolano, C Maltzahn, etc. UCSC Lab, UC Santa Cruz. Published in Journal of Physics, 2009.
Introduction to Experimental Design
© 2017 SlidePlayer.com Inc. All rights reserved.