We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byLatrell Capwell
Modified over 2 years ago
3-1 ©2013 Raj Jain Washington University in St. Louis Selection of Techniques and Metrics Raj Jain Washington University in Saint Louis Saint Louis, MO These slides are available on-line at:
3-2 ©2013 Raj Jain Washington University in St. Louis Overview Criteria for Selecting an Evaluation Technique Three Rules of Validation Selecting Performance Metrics Commonly Used Performance Metrics Utility Classification of Metrics Setting Performance Requirements
3-3 ©2013 Raj Jain Washington University in St. Louis Criteria for Selecting an Evaluation Technique TexPoint fonts used in EMF: AAA A AAA
3-4 ©2013 Raj Jain Washington University in St. Louis Three Rules of Validation Do not trust the results of an analytical model until they have been validated by a simulation model or measurements. Do not trust the results of a simulation model until they have been validated by analytical modeling or measurements. Do not trust the results of a measurement until they have been validated by simulation or analytical modeling.
3-5 ©2013 Raj Jain Washington University in St. Louis Selecting Performance Metrics
3-6 ©2013 Raj Jain Washington University in St. Louis Selecting Metrics Include: Performance Time, Rate, Resource Error rate, probability Time to failure and duration Consider including: Mean and variance Individual and Global Selection Criteria: Low-variability Non-redundancy Completeness
3-7 ©2013 Raj Jain Washington University in St. Louis Case Study: Two Congestion Control Algorithms Service: Send packets from specified source to specified destination in order. Possible outcomes: Some packets are delivered in order to the correct destination. Some packets are delivered out-of-order to the destination. Some packets are delivered more than once (duplicates). Some packets are dropped on the way (lost packets).
3-8 ©2013 Raj Jain Washington University in St. Louis Case Study (Cont) Performance: For packets delivered in order, Time-rate-resource Response time to deliver the packets Throughput: the number of packets per unit of time. Processor time per packet on the source end system. Processor time per packet on the destination end systems. Processor time per packet on the intermediate systems. Variability of the response time Retransmissions Response time: the delay inside the network
3-9 ©2013 Raj Jain Washington University in St. Louis Case Study (Cont) Out-of-order packets consume buffers Probability of out-of-order arrivals. Duplicate packets consume the network resources Probability of duplicate packets Lost packets require retransmission Probability of lost packets Too much loss cause disconnection Probability of disconnect
3-10 ©2013 Raj Jain Washington University in St. Louis Case Study (Cont) Shared Resource Fairness Fairness Index Properties: Always lies between 0 and 1. Equal throughput Fairness =1. If k of n receive x and n-k users receive zero throughput: the fairness index is k/n.
3-11 ©2013 Raj Jain Washington University in St. Louis Case Study (Cont) Throughput and delay were found redundant ) Use Power. Variance in response time redundant with the probability of duplication and the probability of disconnection Total nine metrics.
3-12 ©2013 Raj Jain Washington University in St. Louis Commonly Used Performance Metrics Response time and Reaction time
3-13 ©2013 Raj Jain Washington University in St. Louis Response Time (Cont)
3-14 ©2013 Raj Jain Washington University in St. Louis Capacity
3-15 ©2013 Raj Jain Washington University in St. Louis Common Performance Metrics (Cont) Nominal Capacity: Maximum achievable throughput under ideal workload conditions. E.g., bandwidth in bits per second. The response time at maximum throughput is too high. Usable capacity: Maximum throughput achievable without exceeding a pre-specified response-time limit Knee Capacity: Knee = Low response time and High throughput
3-16 ©2013 Raj Jain Washington University in St. Louis Common Performance Metrics (cont) Turnaround time = the time between the submission of a batch job and the completion of its output. Stretch Factor: The ratio of the response time with multiprogramming to that without multiprogramming. Throughput: Rate (requests per unit of time) Examples: Jobs per second Requests per second Millions of Instructions Per Second (MIPS) Millions of Floating Point Operations Per Second (MFLOPS) Packets Per Second (PPS) Bits per second (bps) Transactions Per Second (TPS)
3-17 ©2013 Raj Jain Washington University in St. Louis Common Performance Metrics (Cont) Efficiency: Ratio usable capacity to nominal capacity. Or, the ratio of the performance of an n-processor system to that of a one-processor system is its efficiency. Utilization: The fraction of time the resource is busy servicing requests. Average fraction used for memory.
3-18 ©2013 Raj Jain Washington University in St. Louis Common Performance Metrics (Cont) Reliability: Probability of errors Mean time between errors (error-free seconds). Availability: Mean Time to Failure (MTTF) Mean Time to Repair (MTTR) MTTF/(MTTF+MTTR)
3-19 ©2013 Raj Jain Washington University in St. Louis Utility Classification of Metrics
3-20 ©2013 Raj Jain Washington University in St. Louis Setting Performance Requirements Examples: The system should be both processing and memory efficient. It should not create excessive overhead There should be an extremely low probability that the network will duplicate a packet, deliver a packet to the wrong destination, or change the data in a packet. Problems: Non-Specific Non-Measurable Non-Acceptable Non-Realizable Non-Thorough SMART
3-21 ©2013 Raj Jain Washington University in St. Louis Case Study 3.2: Local Area Networks Service: Send frame to D Outcomes: Frame is correctly delivered to D Incorrectly delivered Not delivered at all Requirements: Speed The access delay at any station should be less than one second. Sustained throughput must be at least 80 Mbits/sec. Reliability: Five different error modes. Different amount of damage Different level of acceptability.
3-22 ©2013 Raj Jain Washington University in St. Louis Case Study (Cont) The probability of any bit being in error must be less than 1E-7. The probability of any frame being in error (with error indication set) must be less than 1%. The probability of a frame in error being delivered without error indication must be less than 1E-15. The probability of a frame being misdelivered due to an undetected error in the destination address must be less than 1E-18. The probability of a frame being delivered more than once (duplicate) must be less than 1E-5. The probability of losing a frame on the LAN (due to all sorts of errors) must be less than 1%.
3-23 ©2013 Raj Jain Washington University in St. Louis Case Study (Cont) Availability: Two fault modes – Network reinitializations and permanent failures The mean time to initialize the LAN must be less than 15 milliseconds. The mean time between LAN initializations must be at least one minute. The mean time to repair a LAN must be less than one hour. (LAN partitions may be operational during this period.) The mean time between LAN partitioning must be at least one-half a week.
3-24 ©2013 Raj Jain Washington University in St. Louis Summary of Part I Systematic Approach: Define the system, list its services, metrics, parameters, decide factors, evaluation technique, workload, experimental design, analyze the data, and present results Selecting Evaluation Technique: The life-cycle stage is the key. Other considerations are: time available, tools available, accuracy required, trade-offs to be evaluated, cost, and saleability of results.
3-25 ©2013 Raj Jain Washington University in St. Louis Summary (Cont) Selecting Metrics: For each service list time, rate, and resource consumption For each undesirable outcome, measure the frequency and duration of the outcome Check for low-variability, non-redundancy, and completeness. Performance requirements: Should be SMART. Specific, measurable, acceptable, realizable, and thorough.
3-26 ©2013 Raj Jain Washington University in St. Louis Homework 3: Exercise 3.1 What methodology would you choose: a. To select a personal computer for yourself? b. To select 1000 workstations for your company? c. To compare two spread sheet packages? d. To compare two data-flow architectures? if the answer was required: i. Yesterday? ii. Next quarter? iii. Next year? Prepare a table of 12 entries. Write 1 line explanation of each of 12 choices. Common Mistake: Not specifying all 12 combinations.
3-27 ©2013 Raj Jain Washington University in St. Louis Thank You!
3-28 ©2013 Raj Jain Washington University in St. Louis Solution to Homework #3
3-29 ©2013 Raj Jain Washington University in St. Louis Exercise 3.2 Make a complete list of metrics to compare: a. Two personal computers b. Two database systems c. Two disk drives d. Two window systems
©Ian Sommerville 2000Dependable systems specification Slide 1 Chapter 17 Critical Systems Specification.
Copyright 2011 John Wiley & Sons, Inc Business Data Communications and Networking 11th Edition Jerry Fitzgerald and Alan Dennis John Wiley & Sons, Inc.
1 Lecture 22: I/O, Disk Systems Todays topics: I/O overview Disk basics RAID Reminder: Assignment 8 due Tue 11/21.
MFG Assessment Application: Assessment Criteria and Metrics 1 Performance assessment criteria and metrics may be used as the basis for determining the.
1 Notes content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved. Critical Systems Specification.
ISO New England Demand Resource Measurement & Verification Standards Manual Overview April 11, 2007 NAESB Development of DSM/EE Business Practices Washington,
CSE 413: Computer Networks Md. Kamrul Hasan Assistant Professor and Chairman Dept. of Computer and Communication Engineering Patuakhali Science and Technology.
1 Advanced Database Application Development Performance Tuning Performance Benchmarks Standardization E-Commerce Legacy Systems.
VMware vCenter Server High Availability Product Support Engineering VMware Confidential.
Silberschatz, Galvin, and Gagne Applied Operating System Concepts Module 14: Network Structures Background Motivation Topology Network Types.
1 Congestion Control Reading: Sections COS 461: Computer Networks Spring 2006 (MW 1:30-2:50 in Friend 109) Jennifer Rexford Teaching Assistant:
19-1 ©2010 Raj Jain 2 k-p Fractional Factorial Designs.
1 GREY BOX TESTING Web Apps & Networking Session 10 Boris Grinberg
1. Parallel Databases Introduction I/O Parallelism Interquery Parallelism Intraquery Parallelism Intraoperation Parallelism Interoperation Parallelism.
1 Resource Limited Project Management Vladimir Liberzon
1 Benefits Transfer and Meta Analysis Professor Anil Markandya Department of Economics and International Development University of Bath
Copyright: Silberschatz, Korth and Sudarshan 1 Chapter 23: Advanced Data Types and New Applications.
1 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved. Critical Systems Development IS301.
Database System Concepts, 6 th Ed. ©Silberschatz, Korth and Sudarshan See for conditions on re-usewww.db-book.com Chapter 26: Advanced.
Lecturer: Sebastian Coope Ashton Building, Room G.18 COMP 201 web-page: Lecture.
Emin Gabrielyan, Three Topics in Parallel Communications 1 Three Topics in Parallel Communications Thesis presentation by Emin Gabrielyan.
1 OSI Transport Layer IT305: Computer Networks – Chapter 4.
© 2007 – 2010, Cisco Systems, Inc. All rights reserved. Cisco Public ROUTE v6 Chapter 5 1 Chapter 5: Implement Path Control CCNP ROUTE: Implementing IP.
The Whirlwind Tour Chapter 1a. 2 Transactions: Where It All Started [Cuneiform] documents now number about half a million, three- quarters of them more.
1 Test documentation and Test case design Iana Mourza QA Lead/Release Lead VMware, Inc
1 3a. Data Link Layer Protocols 1. Introduction 2. DLL Design a.Network Layer Services b.Error Control c.Flow Control 3. Elementary Data Link Protocols.
Adapted from the slides prepared by Alan Feuer Some material from Operating System Concepts by Silberschatz et. al. and Modern Operating Systems by Tanenbaum.
1 Copyright © 2006 M. E. Kabay. All rights reserved. Rational Risk Management: Balancing Costs and Benefits of Security Measures Cybersecurity Conference.
Computing Higher - SD Process – Topic 2 St Andrew’s High School Unit 2 Software Development Process.
Silberschatz, Galvin and Gagne Operating System Concepts Chapter 10: Virtual Memory Background Demand Paging Process Creation Page Replacement.
© 2016 SlidePlayer.com Inc. All rights reserved.