Performance Concepts Mark A. Magumba

Introduction Research done on 1058 correspondents in 2006 found that 75% OF them would not return to a website that took longer than 4 seconds to load, the same research five years earlier showed people were willing to wait 8 seconds for a page to load People tend to value predictability of a system. For instance if a system slows down only once a week to a user this may seem like the system overall is slow Perceived performance is how well a system appears to perform Perceived performance can be improved by communicating with the user by giving estimates of how long a task will take e.g. visually through a progress bar

Determining Performance During Design When designing a system performance must be considered when the system works well but also when it is in a special state for instance what performance is required – When some component failure has occurred – When the system is in maintenance – When patches are being installed – When batch jobs are being run

Ways of determining performance in the design phase Generally it is difficult to determine the performance of a system at the design phase however some methods can be employed like – Using the experience of vendors – Prototyping – Ensuring system scalability

Managing Bottlenecks The performance of the system as a whole is determined by certain components which may be overloaded known as bottlenecks For instance a system may have very fast networks but very slow hard disks, the fast network speeds are immaterial then as performance will be degraded by the hard disks The hard disks in this case would be a performance bottleneck

Benchmarking Benchmarking measures the comparative performance of test component against an industry standard via a test program. The following table shows some common bench mark tests

Common Bench Mark tests

Performing performance Tests There are three general methods for doing performance tests of whole systems – Load testing: it’s a test to see how well a system operates under a particular load – Stress testing: Is a test to see how well a system operates under extreme load, it is a test to see at where (the bottle necks) and when (the load) a particular system breaks – Endurance testing: Is a measure to see how a system behaves when subjected to the expected load for a sustained period of time

Graph of response time against number of users for typical system

Performance testing methodology Performance testing usually employs a group of servers to act as injectors. These simulate users Another server group known as conductors evaluate the test performance It is important that the test environment matches the production environment as much as possible otherwise results will be unreliable Where the cost of setting up a complete test environment is prohibitive it is possible to use a temporary test environment for instance by renting servers from your vendors

Sources of Performance Metrics Kernel Counters – Most operating systems contain counters that store performance information. Tools are provided to retrieve this information for instance on windows you have the perfmon tool and on UNIX systems that sar tool SNMP counters – SNMP is the standard network management protocol and accumulates information about devices in a Management Information Base. This information can be accessed via a Network Management System. SNMP also provides for tarps which is a special message type sent when a device counter exceeds its normal limit

Sources of Performance Metrics Logging. Stores system event information of infrastructure components sometimes on a dedicated log server. Logging usually generates a lot of information but the level of logging is usually configurable Analyzing logs is fundamentally different from monitoring using tools like SNMP as these are able to react in real time, log files are analysed after some time has elapsed

Performance Patterns Performance patterns – There are several ways to improve performance at the infrastructure level but it is important to note that 80% of performance issues do not originate from the infrastructure but from poorly configured applications Increasing performance in upper layers – Database and application tuning can provide higher performance boosts than infrastructure upgrades

Infrastructure Optimizations Caching – Improves infrastructure performance by keeping frequently accessed instructions and data in memory to improve access times examples include Disk caching Web proxies Operational Data Store – Is a smaller database that is a subset of an organization’s larger database whose purpose is to improve access times to a particular subset of frequently queried data In memory databases

Infrastructure Optimizations Scalability – Implies the ease with which configuration changes can be made to a system in response to changes in number of users. A system whose performance improves proportionally to increase in capacity is said to scale well

Vertical VS Horizontal Scaling Vertical scaling (scaling up) involves adding capacity to a single system component for instance increasing the number of CPUs The problem is that this usually has an upper limit also the cost of adding a single component can quickly become very high Horizontal scaling (scaling out) which involves adding more servers to a system. This usually leads to higher management complexity and may increase latency and affect throughput between nodes Horizontal scaling works best when a system is partitioned in layers In general scaling is best achieved by using more components (horizontally) rather than using very fast components (vertically) and this approach also suffers some performance penalty due to increased overhead

Scaling in partitioned system

Load Balancing To take advantage of scaling usually some load balancing is applied Load balancing is the spreading of the application load across multiple identical servers operating in parallel

Load Balancing

High performance clusters – In this arrangement several standard computers are grouped together via high speed connections to emulate an infinitely powerful super computer Grids – Grids are high performance clusters spread geographically. They rely on the idle processor time of the participating computers. The computers communicate over the internet and in this case the network is a possible bottleneck Security is also an important consideration and in some cases computations must be done twice

Design for use In the case of special purpose performance critical systems some guarantees must be made and these will call for specific design decisions Designing for an interactive system for instance will call for a different set of requirements than for a batch system In some cases special products may be required like custom operating systems and hardware Most vendors will provide architects with standard system implementation plans and in most cases it is advisable that these are followed, it is also a good idea to have vendors verify system plans When possible try to spread system load over time to avoid unnecessary spikes Some systems have offline copies of data available to improve performance of I/O expensive operations