1. 1  2004 Morgan Kaufmann Publishers Chapters 8 Storage, Networks, and Other Peripherals

2. 2  2004 Morgan Kaufmann Publishers

3. 3 Outline 8.1Introduction 8.2Disk Storage and Dependability 8.3Networks 8.4Buses and Other Connections between Processors, Memory, and I/O Devices 8.5Interfacing I/O Devices to the Processor, Memory, and Operating System 8.6I/O Performance Measures: Examples from Disk and File Systems 8.7Designing an I/O System 8.8Real Stuff: A Digital Camera 8.9Fallacies and Pitfalls 8.10Concluding Remarks 8.11Historical Perspective and Further Reading

4. 4  2004 Morgan Kaufmann Publishers 8.1Introduction

5. 5  2004 Morgan Kaufmann Publishers Keywords I/O requests Reads or writes to I/O devices

6. 6  2004 Morgan Kaufmann Publishers Interfacing Processors and Peripherals I/O Design affected by many factors (expandability, resilience) Performance: — access latency — throughput — connection between devices and the system — the memory hierarchy — the operating system A variety of different users (e.g., banks, supercomputers, engineers)

7. 7  2004 Morgan Kaufmann Publishers I/O Important but neglected “The difficulties in assessing and designing I/O systems have often relegated I/O to second class status” “courses in every aspect of computing, from programming to computer architecture often ignore I/O or give it scanty coverage” “textbooks leave the subject to near the end, making it easier for students and instructors to skip it!” GUILTY! — we won’t be looking at I/O in much detail — be sure and read Chapter 8 in its entirety. — you should probably take a networking class!

8. 8  2004 Morgan Kaufmann Publishers I/O Devices Very diverse devices — behavior (i.e., input vs. output) — partner (who is at the other end?) — data rate

9. 9  2004 Morgan Kaufmann Publishers Figure 8.2 The diversity of I/O devices. DeviceBehaviorPartnerData rate (Mbit/sec) KeyboardInputHuman0.0001 MouseInputHuman0.0038 Voice inputInputHuman0.2640 Sound inputInputmachine3.0000 ScannerInputHuman3.2000 Voice outputOutputHuman0.2640 Sound outputOutputHuman8.0000 Laser printerOutputHuman3.2000 Graphics displayOutputHuman800.0000-8000.0000 ModemInput or outputMachine0.0160-0.0640 Network/LANInput or outputMachine100.0000-1000.0000 Network/wireless LANInput or outputMachine11.0000-54.0000 Optical diskStorageMachine80.0000 Magnetic tapeStorageMachine32.0000 Magnetic diskstoragemachine240.0000-2560.0000

10. 10  2004 Morgan Kaufmann Publishers 8.2Disk Storage and Dependability

11. 11  2004 Morgan Kaufmann Publishers Keywords Nonvolatile Storage device where data retains its value even when power is removed. Track One of thousands of concentric circles that makes up the surface of a magnetic disk. Sector One of the segments that make up a track on a magnetic disk; a sector is the smallest amount of information that is read or written on a disk. Seek The process of positioning a read/write head over the proper track on a disk. Rotation latency Also called delay. The time required for the desired sector of a disk to rotate under the read/write head; usually assumed to be half the rotation time. Small computer systems interface (SCSI) A bus used as a standard for I/O devices.

12. 12  2004 Morgan Kaufmann Publishers Keywords Redundant arrays of inexpensive disks (RAID) An organization of disks that uses an array of small and inexpensive disks so as to increase both performance and reliability. Striping Allocation of logically sequential blocks to separate disk to allow higher performance than a single disk can deliver. Mirroring Writing the identical data to multiple disks to increase data availability. Protection group The group of data disks or blocks that share a common check disk or block. Hot swapping Replacing a hardware component while the system is running. Standby spares Reserve hardware resources that can immediately take the place of a failed component.

13. 13  2004 Morgan Kaufmann Publishers I/O Example: Disk Drives To access data: — seek: position head over the proper track (3 to 14 ms. avg.) — rotational latency: wait for desired sector (.5 / RPM) — transfer: grab the data (one or more sectors) 30 to 80 MB/sec

14. 14  2004 Morgan Kaufmann Publishers Disk Read Time Q ： What is the average time to read or write a 512-byte sector for a typical disk rotating at 10,000 RPM? The advertised average seek time is 6 ms, the transfer rate is 50 MB/sec, and the controller overhead is 0.2 ms. Assume that the disk is idle so that there is no waiting time. A ： If the measured average seek time is 25% of the advertised average time. The answer is 1.5ms + 3.0ms + 0.01ms + 0.2ms = 4.7ms

15. 15  2004 Morgan Kaufmann Publishers Figure 8.3 Six magnetic disks, varying in diameter from 14 inches down to 1.8 inches.

16. 16  2004 Morgan Kaufmann Publishers Figure 8.4 Characteristics of three magnetic disks by a single manufacturer in 2004. CharacteristicsSeagate ST373453Seagate ST3200822Seagate ST94811A Disk diameter (inches)3.50 2.50 Formatted data capacity (GB)73.4200.040.0 Number of disk surfaces (heads)842 Rotation speed (PRM)15,00072005400 Internal disk cache size (MB)888 External interface, bandwidth (MB/sec)Ultra320 SCSI, 320Serial ATA, 150ATA, 100 Sustained transfer rate (MB/sec)57-8632-5834 Minimum seek (read/write) (ms)0.2/0.41.0/1.21.5/2.0 Average seek read/write (ms)3.6/3.98.5/9.512.0/14.0

17. 17  2004 Morgan Kaufmann Publishers Mean time to failure (MTTF) (hours) 1,200,000@25 ℃ 600,000@25 ℃ 330,000@25 ℃ Warranty (years)53 — Nonrecoverable read errors per bits read<1 per Temperature, vibration limits (operating) 5-55 ℃, 400Hz@0.5G 0-60 ℃, 350Hz@0.5G 5-55 ℃, 400Hz@1G Size: dimensions (in.), weight (pounds)1.0”X4.0”X5.8”, 1.9 lbs 1.0”X4.0”X5.8”, 1.4 lbs 0.4”X2.7”X3.9”, 0.2 lbs Power: operating/idle/standby (watts) 20?/12/ — 12/8/12.4/1.0/0.4 GB/cu. In., GB/watt3GB/cu.in., 4GB/W9GB/cu.in., 16GB/W10GB/cu.in., 17GB/W Price in 2004, $/GB $400, $5/GB $100, $0.5/GB $100, $2.50/GB

18. 18  2004 Morgan Kaufmann Publishers Figure 8.5 Summary of studies of reasons for failures. OperatorSoftwareHardwareSystemYear data collected 42%25%18%Data center (Tandem)1985 15%55%14%Data center (Tandem)1989 18%44%39%Data center (DEC VAX)1985 50%20%30%Data center (DEC VAX)1993 50%14%19%U.S. public telephone network1996 54% 7%30%U.S. public telephone network2000 60%25%15%Internet services2002

19. 19  2004 Morgan Kaufmann Publishers Figure 8.6 RAID for an example of four data disks showing extra check disks per RAID level and companies that use each level.

20. 20  2004 Morgan Kaufmann Publishers Figure 8.7 Small write update on RAID 3 versus RAID4

21. 21  2004 Morgan Kaufmann Publishers Figure 8.8 Block-interleaved parity (RAID 4) versus distributed block-interleaved parity (RAID 5)

22. 22  2004 Morgan Kaufmann Publishers 8.3Networks

23. 23  2004 Morgan Kaufmann Publishers Networks are growing in popularity over time, and unlike other I/O devices, there are many books and courses on them. For readers who have not taken courses or read books on networking, Section 8.3 on the CD gives a quick overview of the topics and terminology, including internetworking, the OSI model, protocol families such as TCP/IP, long-haul networks such as ATM, local area networks such as Ethernet, and wireless networks such as IEEE 802.11.

24. 24  2004 Morgan Kaufmann Publishers 8.4Buses and Other Connections between Processors, Memory, and I/O Devices

25. 25  2004 Morgan Kaufmann Publishers Keywords Bus transaction A sequence of bus operations that includes a request and may includes a request and may include a response, either of which may carry data. A transaction is initiated by a single request and may take many individual bus operations. Processor-memory bus A bus that connects processor and memory and that is short, generally high speed, and matched to the memory system so as to maximize memory-processor bandwidth. Backplane bus A bus that is designed to allow processors, memory. And I/O devices to coexist on a single bus. Synchronous bus A bus that includes a clock in the control lines and a fixed protocol for communicating that is relative to the clock. Asynchronous bus A bus that uses a handshaking protocol for coordinating usage rather than a clock; can accommodate a wide variety of devices of differing speeds.

26. 26  2004 Morgan Kaufmann Publishers Keywords Handshaking protocol A series of steps used to coordinate asynchronous bus transfers in which the sender and receiver proceed to the next step only when both parties agree that the current step has been completed. Split transaction protocol A protocol in which the bus is released during a bus transaction while the requester is waiting for the data to be transmitted, which frees the bus for access by another requester.

27. 27  2004 Morgan Kaufmann Publishers I/O Example: Buses Shared communication link (one or more wires) Difficult design: — may be bottleneck — length of the bus — number of devices — tradeoffs (buffers for higher bandwidth increases latency) — support for many different devices — cost Types of buses: — processor-memory (short high speed, custom design) — backplane (high speed, often standardized, e.g., PCI) — I/O (lengthy, different devices, e.g., USB, Firewire) Synchronous vs. Asynchronous — use a clock and a synchronous protocol, fast and small but every device must operate at same rate and clock skew requires the bus to be short — don’t use a clock and instead use handshaking

28. 28  2004 Morgan Kaufmann Publishers I/O Bus Standards Today we have two dominant bus standards:

29. 29  2004 Morgan Kaufmann Publishers Figure 8.10 The asynchronous handshaking protocol consists of seven steps to read a word from memory and receive it in an I/O device.

30. 30  2004 Morgan Kaufmann Publishers

31. 31  2004 Morgan Kaufmann Publishers Figure 8.11 Organization of the I/O system on a Pentium 4 PC using the intel 875 chip

32. 32  2004 Morgan Kaufmann Publishers Figure 8.12 Two Pentium 4 I/O chip sets from Intel.

33. 33  2004 Morgan Kaufmann Publishers 8.5Interfacing I/O Devices to the Processor, Memory, and Operating System

34. 34  2004 Morgan Kaufmann Publishers Keywords Memory-mapped I/O An I/O scheme in which portions of address space are assigned to I/O devices and reads and writes to those addresses are interpreted as commands to the I/O device. I/O instructions A dedicated instruction that is used to give a command to an I/O device and that specifies both the device number and the command word (or the location of the command word in memory). Polling The process of periodically checking the status of an I/O device to determine the need to service the device. Interrupted-driven I/O An I/O scheme that employs interrupts to indicate to the processor that an I/O device needs attention. Direct memory access (DMA) A mechanism that provides a device controller the ability to transfer data directly to or from the memory without involving the processor. Bus master A unit on the bus that can initiate bus requests.

35. 35  2004 Morgan Kaufmann Publishers Other important issues Bus Arbitration: — daisy chain arbitration (not very fair) — centralized arbitration (requires an arbiter), e.g., PCI — collision detection, e.g., Ethernet Operating system: — polling — interrupts — direct memory access (DMA) Performance Analysis techniques: — queuing theory — simulation — analysis, i.e., find the weakest link (see “I/O System Design”) Many new developments

36. 36  2004 Morgan Kaufmann Publishers Figure 8.13 The Cause and Status registers.

37. 37  2004 Morgan Kaufmann Publishers 8.6I/O Performance Measures ： Examples from Disk and File Systems

38. 38  2004 Morgan Kaufmann Publishers Keywords Transaction processing A type of application that involves handling small short operations (called transactions) that typically require both I/O and computation. Transaction processing applications typically have both response time requirements and a performance measurement based on the throughput of transactions. I/O rate Performance measure of I/Os per unit time. Such as reads per second. Data rate Performance measure of bytes per unit time, such as GB/second.

39. 39  2004 Morgan Kaufmann Publishers Impact of I/O on system performance Q ： Suppose we have a benchmark that executes in 100 seconds of elapsed time, where 90 seconds is CPU time and the rest is I/O time. If CPU time improves by 50% per year for the next five years but I/O time doesn’t improve, how much faster will our program run at the end of five years? A ： We know that Elapsed time = CPU time + I/O time 100 = 90 + I/O time I/O time = 10 seconds The new CPU times and the resulting elapsed times are computed in the following table ：

40. 40  2004 Morgan Kaufmann Publishers After n yearsCPU timeI/O timeElapsed time% I/O time 090 seconds10 seconds100 seconds10 % 160 seconds10 seconds70 seconds14 % 240 seconds10 seconds50 seconds20 % 327 seconds10 seconds37 seconds27 % 418 seconds10 seconds28 seconds36 % 512 seconds10 seconds22 seconds45 % The improvement in CPU performance over five years is However, the improvement in elapsed time is only and the I/O time has increased from 10% to 45% of the elapsed time.

41. 41  2004 Morgan Kaufmann Publishers 8.7Designing an I/O System

42. 42  2004 Morgan Kaufmann Publishers I/O System Design Q ： Consider the following computer system ： –A CPU that sustain 3 billion instructions per second and averages 100,000 instruction in the operating system per I/O operation –A memory backplane bus capable of sustaining a transfer rate of 1000 MB/sec –SCSI Ultra320 controllers with a transfer rate of 320 MB/sec and accommodating up to 7 disks –Disk drives with a read/write bandwidth of 75 MB/sec and an average seek plus rotational latency of 6 ms. If the workload consists of 64 KB reads (where the block is sequential on a track) and the user program needs 200,000 instructions per I/O operation, find the maximum sustainable I/O rate and the number of disks and SCSI controllers required. Assume that the reads can always be done on an idle disk if one exists (i.e., ignore disk conflicts).

43. 43  2004 Morgan Kaufmann Publishers A ： The maximum number of disks per SCSI bus is 7, which won’t saturate this bus. This means we will need 69/7, or 10 SCSI buses and controllers.

44. 44  2004 Morgan Kaufmann Publishers 8.8Real Stuff ： A Digital Camera

45. 45  2004 Morgan Kaufmann Publishers Figure 8.14 The Sanyo VPC-SX500 with Flash memory card and IBM Microdrive.

46. 46  2004 Morgan Kaufmann Publishers Figure 8.15 Characteristics of three storage alternatives for digital cameras. Characteristics Sandisk Type I compactFlash SDCFB-128-768 Sandisk Type II compactFlash SDCFB-1000-768 Hitachi 4 GB Microdrive DSCM-10340 Formatted data capacity (MB)12810004000 Bytes per sector512 Data transfer rate (MB/sec)4 (burst) 4 – 7 Link speed to buffer (MB/sec)6633 Power standby/operating (W)0.15/0.66 0.07/0.83 Size: height X width X depth (inches)1.43 X 1.68 X 0.13 1.43 X 1.68 X 0.16 Weight in grams (454 grams/pound)11.413.516 Write cycles before sector wear-out300,000 Not applicable Mean time between failures (hours)> 1,000,000 (see caption) Best price (2004)$40$200$480

47. 47  2004 Morgan Kaufmann Publishers Figure 8.16 The system on a chip (SOC) found in Sanyo digital cameras.

48. 48  2004 Morgan Kaufmann Publishers 8.9Fallacies and Pitfalls

49. 49  2004 Morgan Kaufmann Publishers Fallacies and Pitfalls Fallacy: the rated mean time to failure of disks is 1,200,000 hours, so disks practically never fail. Fallacy: magnetic disk storage is on its last legs, will be replaced. Fallacy: A 100 MB/sec bus can transfer 100 MB/sec. Pitfall: Moving functions from the CPU to the I/O processor, expecting to improve performance without analysis.

50. 50  2004 Morgan Kaufmann Publishers Multiprocessors Idea: create powerful computers by connecting many smaller ones good news: works for timesharing (better than supercomputer) bad news: its really hard to write good concurrent programs many commercial failures

51. 51  2004 Morgan Kaufmann Publishers Questions How do parallel processors share data? — single address space (SMP vs. NUMA) — message passing How do parallel processors coordinate? — synchronization (locks, semaphores) — built into send / receive primitives — operating system protocols How are they implemented? — connected by a single bus — connected by a network

52. 52  2004 Morgan Kaufmann Publishers Supercomputers Plot of top 500 supercomputer sites over a decade:

53. 53  2004 Morgan Kaufmann Publishers Using multiple processors an old idea Some SIMD designs: Costs for the the Illiac IV escalated from $8 million in 1966 to $32 million in 1972 despite completion of only ¼ of the machine. It took three more years before it was operational! “For better or worse, computer architects are not easily discouraged” Lots of interesting designs and ideas, lots of failures, few successes

54. 54  2004 Morgan Kaufmann Publishers Topologies

55. 55  2004 Morgan Kaufmann Publishers Clusters Constructed from whole computers Independent, scalable networks Strengths: –Many applications amenable to loosely coupled machines –Exploit local area networks –Cost effective / Easy to expand Weaknesses: –Administration costs not necessarily lower –Connected using I/O bus Highly available due to separation of memories In theory, we should be able to do better

56. 56  2004 Morgan Kaufmann Publishers Google Serve an average of 1000 queries per second Google uses 6,000 processors and 12,000 disks Two sites in silicon valley, two in Virginia Each site connected to internet using OC48 (2488 Mbit/sec) Reliability: –On an average day, 20 machines need rebooted (software error) –2% of the machines replaced each year In some sense, simple ideas well executed. Better (and cheaper) than other approaches involving increased complexity

57. 57  2004 Morgan Kaufmann Publishers Concluding Remarks Evolution vs. Revolution “More often the expense of innovation comes from being too disruptive to computer users” “Acceptance of hardware ideas requires acceptance by software people; therefore hardware people should learn about software. And if software people want good machines, they must learn more about hardware to be able to communicate with and thereby influence hardware engineers.”