We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byCordelia Hamilton
Modified about 1 year ago
Copyright © 2005 Department of Computer Science CPSC 641 Winter PERFORMANCE EVALUATION Often in Computer Science you need to: – demonstrate that a new concept, technique, or algorithm is feasible –demonstrate that a new method is better than an existing method –understand the impact of various factors and parameters on the performance, scalability, or robustness of a system
Copyright © 2005 Department of Computer Science CPSC 641 Winter PERFORMANCE EVALUATION There is a whole field of computer science called computer systems performance evaluation that is devoted to exactly this One classic book is Raj Jain’s “The Art of Computer Systems Performance Analysis”, Wiley & Sons, 1991 Much of what is outlined in this presentation is described in more detail in [Jain 1991]
Copyright © 2005 Department of Computer Science CPSC 641 Winter PERF EVAL: THE BASICS There are three main methods used in the design of performance evaluation studies: Analytic approaches –the use of mathematics, Markov chains, queueing theory, Petri Nets, abstract models… Simulation approaches –design and use of computer simulations and simplified models to assess performance Experimental approaches –measurement and use of a real system
Copyright © 2005 Department of Computer Science CPSC 641 Winter Analytical Example: Queueing Theory Queueing theory is a mathematical technique that specializes in the analysis of queues (e.g., customer arrivals at a bank, jobs arriving at CPU, I/O requests arriving at a disk subsystem, lineup at Tim Hortons) General diagram: Customer Arrivals Departures Buffer Server
Copyright © 2005 Department of Computer Science CPSC 641 Winter Queueing Theory (cont’d) The queueing system is characterized by: –Arrival process (M, G) –Service time process (M, D, G) –Number of servers (1 to infinity) –Number of buffers (infinite or finite) Example notation: M/M/1, M/D/1 Example notation: M/M/, M/G/1/k 8
Copyright © 2005 Department of Computer Science CPSC 641 Winter Queueing Theory (cont’d) There are well-known mathematical results for the mean waiting time and the number of customers in the system for several simple queueing models E.g., M/M/1, M/D/1, M/G/1 Example: M/M/1 –q = rho/ (1 - rho) where rho = lambda/mu < 1
Copyright © 2005 Department of Computer Science CPSC 641 Winter Queueing Theory (cont’d) These simple models can be cascaded in series and in parallel to create arbitrarily large complicated queueing network models Two main types: –closed queueing network model (finite pop.) –open queueing network model (infinite pop.) Software packages exist for solving these types of models to determine steady-state performance (e.g., delay, throughput, util.)
Copyright © 2005 Department of Computer Science CPSC 641 Winter Simulation Example: TCP Throughput Can use an existing simulation tool, or design and build your own custom simulator Example: ns-2 network simulator A discrete-event simulator with detailed TCP protocol models Configure network topology and workload Run simulation using pseudo-random numbers and produce statistical output
Copyright © 2005 Department of Computer Science CPSC 641 Winter OTHER ISSUES Simulation run length –choosing a long enough run time to get statistically meaningful results (equilibrium) Simulation start-up effects and end effects –deciding how much to “chop off” at the start and end of simulations to get proper results Replications –ensure repeatability of results, and gain greater statistical confidence in the results given
Copyright © 2005 Department of Computer Science CPSC 641 Winter Experimental Example: Benchmarking The design of a performance study requires great care in experimental design and methodology Need to identify –experimental factors to be tested –levels (settings) for these factors –performance metrics to be used –experimental design to be used
Copyright © 2005 Department of Computer Science CPSC 641 Winter FACTORS Factors are the main “components” that are varied in an experiment, in order to understand their impact on performance Examples: request rate, request size, read/write ratio, num concurrent clients Need to choose factors properly, since the number of factors affects size of study
Copyright © 2005 Department of Computer Science CPSC 641 Winter LEVELS Levels are the precise settings of the factors that are to be used in an experiment Examples: req size S = 1 KB, 10 KB, 1 MB Example: num clients C = 10, 20, 30, 40, 50 Need to choose levels realistically Need to cover useful portion of the design space
Copyright © 2005 Department of Computer Science CPSC 641 Winter PERFORMANCE METRICS Performance metrics specify what you want to measure in your performance study Examples: response time, throughput, pkt loss Must choose your metrics properly and instrument your experiment accordingly
Copyright © 2005 Department of Computer Science CPSC 641 Winter EXPERIMENTAL DESIGN Experimental design refers to the organizational structure of your experiment Need to methodically go through factors and levels to get the full range of experimental results desired There are several “classical” approaches to experimental design
Copyright © 2005 Department of Computer Science CPSC 641 Winter EXAMPLES One factor at a time –vary only one factor through its levels to see what the impact is on performance Two factors at a time –vary two factors to see not only their individual effects, but also their interaction effects, if any Full factorial –try every possible combination of factors and levels to see full range of performance results
Copyright © 2005 Department of Computer Science CPSC 641 Winter SUMMARY Computer systems performance evaluation defines standard methods for designing and conducting performance studies Great care must be taken in experimental design and methodology if the experiment is to achieve its goal, and if results are to be fully understood We will see examples of these methodologies and their applications over the next 3 months
Methods of Stochastic Performance Analysis Perry Lea Hewlett Packard - Core Technology Laboratory.
1 Chapter 2: Decision Making, Systems, Modeling, and Support Conceptual Foundations of Decision Making The Systems Approach How Support is Provided.
© Copyright 2008, SoftWell Performance AB 1 Performance Measurements of IMS systems, Concepts.
Nica Valentin–Danut SEM 2012 Service system fundamentals: Work system, value chain, and life cycle.
UNIT-II Queuing Theory. Queuing Theory A mathematical method of analyzing the congestions and delays of waiting in line. Queuing theory examines every.
Unit-V -SOFTWARE QUALITY. To develop and deliver robust system, we need a high level of confidence that Each component will behave correctly Collective.
Copyright 2011 John Wiley & Sons, Inc Business Data Communications and Networking 11th Edition Jerry Fitzgerald and Alan Dennis John Wiley & Sons, Inc.
Decision Support Systems and Executive Support Systems.
Common HL7 Interface Implementation Issues Merlin Griscowsky Managing Partner - Tekmerion Consulting.
Software Development QA Best Practices May 20, 2010 Suzette Hackl, CSM Senior Project Manager Skyline Technologies, Inc.
A Primer in Theory Construction By Paul Davidson Reynolds.
Network Design 1 May 2008 Dr. Charles Graff, US Army RDECOM CERDEC-STCD Ft. Monmouth NJ
GROUP 5 PALLAV SUDHIR PRITY BALA ANUP SARANSH RAJAT HIRNI ANTHONY CHETNA ROGER.
McGraw-Hill/Irwin Copyright © 2013 by The McGraw-Hill Companies, Inc. All rights reserved. A PowerPoint Presentation Package to Accompany Applied Statistics.
Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.
Software Quality Management CIS 376 Bruce R. Maxim UM-Dearborn.
1 Advanced Database Application Development Performance Tuning Performance Benchmarks Standardization E-Commerce Legacy Systems.
Implementation Architecture Lecture Implementation View (1) “ how the system is built ” Focuses on “ how the system is built ” technological.
DBMS Storage and Indexing. Disk Storage Disks and Files DBMS stores information on (“hard”) disks. This has major implications for DBMS design! ▫ READ:
Copyright All Rights Reserved 1 Chapter 5 Development and Evolution of User Interface 5.1 User-Centered Design Aim of User center design Human-Computer.
Seven S model Definition Concept used as a measure of the quality of the performance of a firm, as indicated by 7 factors so interrelated that changing.
Public Information Version 3.1: 1/1/2012 Introducing Instant Business Intelligence To IT BI Project Managers What you need, when you need it
Executional Architecture Lecture Conceptual vs execution Conceptual Architecture Execution Architecture Component Connector Domain-level responsibilities.
Distributed Computing Dr. Eng. Ahmed Moustafa Elmahalawy Computer Science and Engineering Department.
What is an Operating System? A program that acts as an intermediary between a user of a computer and the computer hardware. Operating system goals: Execute.
Writing for CS and CE Gabriel Dos Reis Assistant Professor Department of Computer Science and Engineering.
Evaluation of User Interface Design Evaluation is very important in User Interface Design and it is generally considered that there is no way round evaluation.
Introduction to Project Management session 1. Project management Over the course we will look at: Projects and their features. The project Life Cycle,
Course round-up subtitle- Statistical model building Marian Scott University of Glasgow Glasgow, Aug 2012.
Of An Expert System. Introduction What is AI? Intelligent in Human & Machine? What is Expert System? How are Expert System used? Elements of ES Who are.
© 2016 SlidePlayer.com Inc. All rights reserved.