ENGS 116 Lecture 181 I/O Interfaces, A Little Queueing Theory RAID Vincent H. Berk November 19, 2007 Reading for Today: Sections 7.1 – 7.4 Reading for Monday: Sections 7.5 – 7.9, 7.14 Homework for Monday: 5.4, 5.17, 5.18, 6.4, 6.10, 6.14

ENGS 116 Lecture 182 Common Bus Standards ISA PCI AGP / PCI Express PCMCIA USB FireWire/IEEE 1394 IDE / ATA / SATA SCSI

3ENGS 116 Lecture 18 Programmed I/O (Polling) CPU IOC Device Memory Is the data ready? read data store data yes no done? no yes Busy wait loop not an efficient way to use the CPU unless the device is very fast! However, checks for I/O completion can be dispersed among computationally intensive code.

4ENGS 116 Lecture 18 User program progress only halted during actual transfer 1000 transfers at 1 ms each: µ sec per interrupt 1000 interrupt 98 µ sec each = 0.1 CPU seconds Device transfer rate = 10 MBytes/sec  0.1 x sec/byte  0.1 µsec/byte  1000 bytes = 100 µsec 1000 transfers  100 µsecs = 100 ms = 0.1 CPU seconds Still far from device transfer rate! 1/2 time in interrupt overhead Interrupt-Driven Data Transfer CPU IOC device Memory add sub and or nop read store... rti memory user program (1) I/O interrupt (2) save PC (3) interrupt service addr interrupt service routine (4)

5ENGS 116 Lecture 18 Direct Memory Access (DMA) Time to do 1000 transfers at 1 msec each: 1 DMA set-up 50 µ sec 1 2 µ sec 1 interrupt service 48 µ sec.0001 second of CPU time CPU sends a starting address, direction, and length count to DMAC. Then issues "start". DMAC provides handshake signals for peripheral controller and memory addresses and handshake signals for memory. DO NOT CACHE I/O ADDRESSES! CPU IOC device Memory DMAC 0 ROM RAM Peripherals DMAC n Memory Mapped I/O

6ENGS 116 Lecture 18 Input/Output Processors CPU IOP Mem D1 D2 Dn... main memory bus I/O bus CPU IOP issues instruction to IOP interrupts when done (1) memory (2) (3) (4) Device to/from memory transfers are controlled by the IOP directly. IOP steals memory cycles. OP Device Address target device where commands are looks in memory for commands OP Addr Cnt Other what to do where to put data how much special requests

ENGS 116 Lecture 187 Summary Disk industry growing rapidly, improves bandwidth and areal density Time = queue + controller + seek + rotate + transfer Advertised average seek time much greater than average seek time in practice Response time vs. bandwidth tradeoffs Processor interface: today peripheral processors, DMA, I/O bus, interrupts

ENGS 116 Lecture 188 A Little Queueing Theory More interested in long-term, steady-state behavior  Arrivals = Departures Little’s Law: mean number of tasks in system = arrival rate  mean response time –Observed by many, Little was first to prove –Applies to any system in equilibrium as long as nothing inside the system (black box) is creating or destroying tasks Queueing models assume state of equilibrium: input rate = output rate Arrivals System Departures

ENGS 116 Lecture 189 A Little Queueing Theory Avg. arrival rate Avg. service rate  Avg. number in system N Avg. system time per customer T = avg. waiting time + avg. service time Little’s Law: N =  T Service utilization  = /  Departures Server Queue Arrivals

ENGS 116 Lecture 1810 A Little Queueing Theory Server spends a variable amount of time with customers –Service distribution characterized by mean, variance, squared coefficient of variance (C) –Squared coefficient of variance: C = variance/mean 2, unitless measure Exponential distribution: C = 1; most short relative to average Hypoexponential distribution: C < 1; most close to average Hyperexponential distribution: C > 1; most further from average Disk response times: C ≈ 1.5, but usually pick C = 1 for simplicity

ENGS 116 Lecture 1811 Average Residual Service Time How long does a new customer wait for the current customer to finish service? Average residual time = If C = 0, average residual service time = 1/2 mean service time

ENGS 116 Lecture 1812 Average Wait Time in Queue If something at server, it takes average residual service time to complete Probability that server is busy is  All customers in line must complete, each averaging T service If exponential distribution, C = 1 and

ENGS 116 Lecture 1813 M/M/1 and M/G/1 Assumptions so far: –System in equilibrium –Time between two successive arrivals in line are random –Server can start on next customer immediately after prior customer finishes –No limit to the queue, FIFO service –All customers in line must complete, each averaging T service Exponential distribution (C = 1) is “memoryless” or Markovian, denoted by M Queueing system with exponential arrivals, exponential service times, and 1 server: M/M/1 General distribution is denoted by G: can have M/G/1 queue

ENGS 116 Lecture 1814 An Example Processor sends 10 8-KB disk I/Os per second, requests and service times exponentially distributed, avg. disk service = 20 ms.

ENGS 116 Lecture 1815 Another Example Processor sends 20 8-KB disk I/Os per second, requests and service times exponentially distributed, avg. disk service = 12 ms.

ENGS 116 Lecture 1816 Yet Another Example Processor sends 10 8-KB disk I/Os per second, C = 1.5, avg. disk service = 20 ms.

ENGS 116 Lecture 1817 Manufacturing Advantages of Disk Arrays 3.5” Disk Array: 1 disk design 14” 10”5.25”3.5” Conventional: 4 disk designs Low End High End Disk Product Families 14”

18ENGS 116 Lecture 18 Replace Small # of Large Disks with Large # of Small Disks! Data Capacity Volume Power Data Rate I/O Rate MTBF Cost IBM 3390 (K) 20 GBytes 97 cu. ft. 3 KW 15 MB/s 600 I/Os/s 250 KHrs $250K IBM 3.5" MBytes 0.1 cu. ft. 11 W 1.5 MB/s 55 I/Os/s 50 KHrs $2K x70 23 GBytes 11 cu. ft. 1 KW 120 MB/s 3900 IOs/s ??? Hrs $150K Disk Arrays have potential for large data and I/O rates high MB per cu. ft., high MB per KW awful reliability

19ENGS 116 Lecture 18 Array Reliability Reliability of N disks = Reliability of 1 Disk ÷ N 50,000 Hours ÷ 70 disks = 700 hours Disk system MTTF: Drops from 6 years to 1 month! Arrays without redundancy too unreliable to be useful! Hot spares support reconstruction in parallel with access: very high media availability can be achieved

20ENGS 116 Lecture 18 Mirroring/Shadowing (high capacity cost) Horizontal Hamming Codes (overkill) Parity & Reed-Solomon Codes Failure Prediction (no capacity overhead!) VaxSimPlus — Technique is controversial Techniques: Redundant Arrays of Disks Files are "striped" across multiple spindles Redundancy yields high data availability Disks will fail Contents reconstructed from data redundantly stored in the array  Capacity penalty to store it  Bandwidth penalty to update

21ENGS 116 Lecture 18 Redundant Arrays of Disks (RAID) Techniques Disk Mirroring, Shadowing Each disk is fully duplicated onto its "shadow" Logical write = two physical writes 100% capacity overhead Parity Data Bandwidth Array Parity computed horizontally Logically a single high data bandwidth disk High I/O Rate Parity Array Interleaved parity blocks Independent reads and writes Logical write = 2 reads + 2 writes Parity + Reed-Solomon codes Parity disk

22ENGS 116 Lecture 18 RAID 0: Disk Striping Data is distributed over disks Improved bandwidth and seek time on read and write Larger virtual disk No redundancy

23ENGS 116 Lecture 18 RAID 1: Disk Mirroring/Shadowing Each disk is fully duplicated onto its "shadow" Very high availability can be achieved Bandwidth sacrifice on write: Logical write = two physical writes Half seek time on reads Reads may be optimized Most expensive solution: 100% capacity overhead Targeted for high I/O rate, high availability environments recovery group

24ENGS 116 Lecture 18 RAID 3: Parity Disk P logical record Striped physical records Parity computed across recovery group to protect against hard disk failures 33% capacity cost for parity in this configuration wider arrays reduce capacity costs, decrease expected availability, increase reconstruction time Arms logically synchronized, spindles rotationally synchronized logically a single high capacity, high transfer rate disk Targeted for high bandwidth applications

25ENGS 116 Lecture 18 RAID 4 & 5: Block-Interleaved Parity and Distributed Block-Interleaved Parity Similar to RAID 3, requiring same number of disks. Parity is computed over blocks and stored in blocks. RAID 4 places parity on last disk RAID 5 places parity blocks distributed over all disks: Advantage: parity block is always accessed on read/writes Parity is updated by reading the old-block and parity-block, and writing the new-block and the new parity-block. (2 reads, 2 writes)

26ENGS 116 Lecture 18 Disk access advantage RAID 4/5 over RAID 3

27ENGS 116 Lecture 18 Problems of Disk Arrays: Small Writes D0D1D2D3PD0' + + D1D2D3P' new data old data old parity XOR (1. Read) (2. Read) (3. Write) (4. Write) RAID-5: Small Write Algorithm 1 Logical Write = 2 Physical Reads + 2 Physical Writes

ENGS 116 Lecture 1828 I/O Benchmarks: Transaction Processing Transaction Processing (TP) (or On-line TP = OLTP) –Changes to a large body of shared information from many terminals, with the TP system guaranteeing proper behavior on a failure –If a bank’s computer fails when a customer withdraws money, the TP system would guarantee that the account is debited if the customer received the money and that the account is unchanged if the money was not received –Airline reservation systems & banks use TP Atomic transactions makes this work Each transaction => 2 to 10 disk I/Os and 5,000 to 20,000 CPU instructions per disk I/O –Efficient TP software, avoid disks accesses by keeping information in main memory Classic metric is Transactions Per Second (TPS) –Under what workload? How is machine configured?

ENGS 116 Lecture 1829 I/O Benchmarks: Transaction Processing Early 1980s great interest in OLTP –Expecting demand for high TPS (e.g., ATM machines, credit cards) –Tandem’s success implied medium range OLTP expands –Each vendor picked own conditions for TPS claims, reported only CPU times with widely different I/O –Conflicting claims led to disbelief of all benchmarks => chaos 1984 Jim Gray of Tandem distributed paper to Tandem employees and 19 in other industries to propose standard benchmark Published “A measure of transaction processing power,” Datamation, 1985 by Anonymous et. al –To indicate that this was effort of large group –To avoid delays of legal department of each author’s firm –Author still gets mail at Tandem

ENGS 116 Lecture 1830 I/O Benchmarks: TP by Anon et. al Proposed 3 standard tests to characterize commercial OLTP –TP1: OLTP test, DebitCredit, simulates ATMs (TP1) –Batch sort –Batch scan Debit/Credit: –One type of transaction: 100 bytes each –Recorded 3 places: account file, branch file, teller file; events recorded in history file (90 days) 15% requests for different branches –Under what conditions, how to report results?

ENGS 116 Lecture 1831 I/O Benchmarks: TP1 by Anon et. al DebitCredit Scalability: size of account, branch, teller, history function of throughput TPSNumber of ATMsAccount-file size 101, GB 10010, GB 1,000100, GB 10,0001,000, GB –Each input TPS => 100,000 account records, 10 branches, 100 ATMs –Accounts must grow since a person is not likely to use the bank more frequently just because the bank has a faster computer! Response time: 95% transactions take ≤ 1 second Configuration control: just report price (initial purchase price + 5 year maintenance = cost of ownership) By publishing, in public domain