1. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 1 Lecture 17 I/O Interfaces and I/O Busses

2. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 2 Storage System Issues Historical Context of Storage I/O Storage I/O Performance Measures Secondary and Tertiary Storage Devices Processor Interface Issues I/O Buses Redundant Arrays of Inexpensive Disks (RAID)

3. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 3 Disk Time Example Disk Parameters: –Transfer size is 8K bytes –Advertised average seek is 12 ms –Disk spins at 7200 RPM –Transfer rate is 4 MB/sec –Controller overhead is 2 ms Assume that disk is idle so no queuing delay What is Average Disk Access Time for a Sector? –Ave seek + ave rot delay + transfer time + controller overhead –12 ms + 0.5/(7200 RPM/60) + 8 KB/4 MB/s + 2 ms –12 + 4.15 + 2 + 2 = 20 ms Advertised seek time assumes no locality: typically 1/4 to 1/3 advertised seek time –Average Disk Access Time for a Sector: 20 ms => 12 ms

4. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 4 Processor Interface Issues Interconnections –Busses Processor interface –Interrupts –Memory mapped I/O I/O Control Structures –Polling –Interrupts –DMA –I/O Controllers –I/O Processors Capacity, Access Time, Bandwidth

5. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 5 I/O Interface Seperate I/O instructions (in,out) Lines distinguish between I/O and memory transfers VME bus Multibus-II Nubus 40 Mbytes/sec optimistically 10 MIPS processor completely saturates the bus! CPU Interface Peripheral Memory memory bus Independent I/O Bus Common Memory & I/O bus CPU Interface Peripheral Memory

6. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 6 Memory Mapped I/O Single Memory & I/O Bus No Separate I/O Instructions CPU Interface Peripheral Memory L2 $ Memory Bus MemoryBus Adapter I/O bus $ CPU ROM RAM I/O

7. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 7 Programmed I/O (Polling) CPU IOC device Memory not an efficient way to use the CPU, unless the device is very fast! but checks for I/O completion can be dispersed among computationally intensive code busy on wait loop Is data ready? read data store data yes no done? no yes

8. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 8 Interrupt Driven Data Transfer (2) save PC (3) interrupt service addr (4) 1000 transfers/second: 1000 interrupts @ 2 sec per interrupt=> 2 msec 1000 interrupt service @ 98 sec each=> 98 msec 100 msec = 0.1 CPU seconds CPU IOC device Memory User program halts only during actual transfer memory user program interrupt service routine read store... rti add sub and or nop (1) I/O interrupt

9. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 9 Direct Memory Access 0 ROM RAM Peripherals DMAC n Memory Mapped I/O Time to do 1000 xfers in 1 msec: 1 DMA set-up sequence: @ 50 sec 1 interrupt: @ 2 sec 1 interrupt service sequence: @ 48 sec 100 sec.0001 second of CPU time CPU sends a starting address, direction(R/W), and word count to DMAC. Then issues "start". DMAC provides; Peripheral controller Handshake signals Memory Addresses Handshake signals CPU IOC device Memory DMAC I/O

10. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 10 Input/Output Processors (2) (3) Device to/from memory transfers are controlled by the IOP directly. IOP steals memory cycles from CPU OP Device Address target device where cmnds are looks in memory for commands CPU IOP Mem D1 D2 Dn... main memory bus I/O bus OP Addr Cnt Other what to do where to put data how much special requests Command memory CPU IOP (1) issues instruction to IOP interrupts when done (4)

11. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 11 Relationship to Processor Architecture I/O instructions and busses have largely disappeared Interrupt vectors have been replaced by jump tables PC <- M [ IVA + interrupt number ] Interrupts: –Stack replaced by shadow registers –Handler saves registers and re-enables higher priority interrupts

12. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 12 Relationship to Processor Architecture Caches required for processor performance cause problems for I/O –Flushing is expensive, I/O pollutes cache –Solution is borrowed from shared memory multiprocessors "snooping" Virtual memory frustrates DMA Stateful processors are hard to switch context

13. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 13 Interconnect Trends >1000 m10 - 100 m 1 mDistance 10 - 100 Mb/s40 - 1000 Mb/s 320 - 1000+ Mb/s Bandwidth high (>ms)medium low (<µs)Latency low Extensive CRC medium Byte Parity high Byte Parity Reliability Interconnect = glue that interfaces computer system components High speed hardware interfaces + logical protocols Networks, channels, backplanes Memory-mapped Wide pathways Centralized arbit. Message-based Narrow pathways Distributed arbitration NetworkChannel Backplane

14. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 14 Backplane Architectures Distinctions begin to blur: SCSI channel is like a bus FutureBus is like a channel (disconnect/reconnect) HIPPI forms links in high speed switching fabrics 128 No 16 - 32 Single/Multiple Multiple No Async 25 12.9 27.9 13.6 21.5 m IEEE 1014 96 Yes 32 Single/Multiple Multiple Optional Async 37 15.5 95.2 20.8 20.5 m IEEE 896 MetricVMEFutureBus 96 Yes 32 Single/Multiple Multiple Optional Sync 20 10 40 13.3 21.5 m ANSI/IEEE 1296 MultiBus II Bus Width (signals) Address data multiplexed? Data Width Xfer Size # of Bus Masters Split Transactions Clocking Bandwidth, Single W ord (0 ns mem) Bandwidth, Single Word (150ns mem) Bandwidth Multiple W ord (0 ns mem) Bandwidth Multiple W ord (150 ns mem) Max # of devices Max Bus Length Standard 25 na 8 Single/Multiple Multiple Optional Either 5, 1.5 7 25 m ANSI X3.131 SCSI-I

15. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 15 Bus-Based Interconnect Bus: a shared communication link between subsystems –Low cost: a single set of wires is shared multiple ways –Versatility: Easy to add new devices & peripherals may even be ported between computers using common bus Disadvantage –A communication bottleneck, possibly limiting the maximum I/O throughput Bus speed is limited by physical factors –the bus length –the number of devices (and, hence, bus loading). –these physical limits prevent arbitrary bus speedup.

16. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 16 Bus-Based Interconnect Two generic types of busses: –I/O busses (sometimes called a channel) lengthy many types of devices connected wide range in the data bandwidth follow a bus standard –CPU-Memory buses (sometimes called a backplane) short high speed, matched to the memory system to maximize memory-CPU bandwidth –To lower costs, low cost (older) systems combine together Bus transaction –Sending address & receiving or sending data

17. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 17 Bus Protocols Multibus: 20 address, 16 data, 5 control, 50ns Pause Bus Master: has ability to control the bus, initiates a transaction Bus Slave: module activated by the transaction Bus Communication Protocol: specification of sequence of events and timing requirements in transferring information. –Asynchronous Bus Transfers: control lines (req., ack.) serve to orchestrate sequencing –Synchronous Bus Transfers: sequence relative to the common clock ° ° ° MasterSlave Control Lines Address Lines Data Lines

18. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 18 Synchronous Bus Protocols Pipelined/Split transaction Bus Protocol Address Data Wait addr 1 data 0 addr 2 wait 1 data 1 addr 3 OK 1 begin read Read complete Address Data Read Wait Clock Read transaction

19. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 19 Asynchronous Handshake Write Transaction t3: Master releases req Address Data Read Req. Ack. 4 Cycle Handshake t0 t1 t2 t3 t4 t5 Master Asserts Address Next Address Master Asserts Data t0: Master has obtained control and asserts address, direction, data; Waits a specified amount of time for slaves to decode target t2: Slave asserts ack, indicating data received t1: Master asserts request line t4: Slave releases ack

20. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 20 Asynchronous Handshake Time Multiplexed Bus: address and data share lines Read Transaction Address Data Read Req Ack Master Asserts Address Next Address t0 t1 t2 t3 t4 t5 4 Cycle Handshake t0: Master has obtained control and asserts address, direction; Waits a specified amount of time for slaves to decode t2: Slave asserts ack, indicating ready to transmit data; Slave asserts data t3: Master releases req, data received t1: Master asserts request line t4: Slave releases ack

21. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 21 Bus Arbitration Parallel (Centralized) Arbitration BR BG M M M Bus Arbiter M BGi BGo BR M BGi BGo BR M BGi BGo BR BG BR A.U. BR A M M M BR A A.U. Serial Arbitration (daisy chaining) Polling BR: Bus Request BG: Bus Grant Bus sequence: BR BG Busy data

22. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 22 Bus Options Bus widthSeparate addressMultiplex address & data lines& data lines Data widthWider is fasterNarrower is cheaper (e.g., 32 bits)(e.g., 8 bits) Transfer sizeMultiple words hasSingle-word transfer less bus overheadis simpler Bus mastersMultipleSingle master (requires arbitration)(no arbitration) Split Yes - separate No - continuous transaction? Request and Replyconnection is cheaper packets gets higher and has lower latency bandwidth (needs multiple masters) ClockingSynchronousAsynchronous OptionHigh performanceLow cost

23. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 23 1990 Bus Survey Signals1289696168 Addr/Data muxnoyesyesn/an/a Data width16 - 323232168 Mastersmultimultimultisinglemulti ClockingAsyncAsyncSyncAsynceither MB/s 253720251.5 (asyn) (0ns, word) 5 (sync) 150ns word12.915.510== 0ns block27.995.240== 150ns block13.620.813.3== Max devices21202187 Max meters0.50.50.55025 Standard IEEE 1014 IEEE 896.1 ANSI/IEEE ANSI X3.129 ANSI X3.131 1296 VME FutureBus Multibus IIIPISCSI

24. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 24 VMEVME 3 96-pin connectors 128 defined as standard, rest customer defined –32 address –32 data –64 command & power/ground lines

25. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 25 SCSI: Small Computer System Interface Up to 8 devices to communicate on a bus or string at sustained speeds of 4-5 MBytes/sec SCSI-2 up to 20 MB/sec Devices can be slave (target) or master(initiator) SCSI protocol: a series of phases, during which specific actions are taken by the controller and the SCSI disks –Bus Free: No device is currently accessing the bus –Arbitration: When the SCSI bus goes free, multiple devices may request (arbitrate for) the bus; fixed priority by address –Selection: informs the target that it will participate (Reselection if disconnected) –Command: the initiator reads the SCSI command bytes from host memory and sends them to the target –Data Transfer: data in or out, initiator: target –Message Phase: message in or out, initiator: target (identify, save/restore data pointer, disconnect, command complete) –Status Phase: target, just before command complete

26. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 26 SCSI Bus: Channel Architecture Command Setup Arbitration Selection Message Out (Identify) Command Disconnect to seek/¼ll buffer Message In (Disconnect) - - Bus Free - - Arbitration Reselection Message In (Identify) DataTransfer Data In Disconnect to ¼ ll buffer Message In (Save Data Ptr) Message In (Disconnect) - - Bus Free - - Arbitration Reselection Message In (Identify) Command Completion Status Message In (Command Complete) If no disconnect is needed Completion Message In (Restore Data Ptr) peer-to-peer protocols initiator/target linear byte streams disconnect/reconnect

27. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 27 1993 I/O Bus Survey OriginatorSunDECIBMIntel Clock Rate (MHz)16-2512.5-25async33 AddressingVirtualPhysicalPhysicalPhysical Data Sizes (bits)8,16,328,16,24,328,16,24,32,648,16,24,32,64 MasterMultiSingleMultiMulti ArbitrationCentralCentralCentralCentral 32 bit read (MB/s)33252033 Peak (MB/s)898475111 (222) Max Power (W)16261325 BusSBus TurboChannel MicroChannelPCI

28. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 28 1993 MP Server Memory Bus Survey OriginatorHPSGISun Clock Rate (MHz)604866 Split transaction?YesYes Yes? Address lines4840?? Data lines128256144 (parity) Data Sizes (bits)5121024512 Clocks/transfer454? Peak (MB/s)96012001056 MasterMultiMultiMulti ArbitrationCentralCentralCentral AddressingPhysicalPhysicalPhysical Slots16910 Busses/system112 Length13 inches12? inches17 inches Bus Summit Challenge XDBus

29. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 29 Communications Networks Performance limiter is memory system, OS overhead, not protocols Send/receive queues in processor memories Network controller copies back and forth via DMA No host intervention needed Interrupt host when message sent or received

30. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 30 I/O Controller Architecture Request/Response block interface Backdoor access to host memory Peripheral Bus (VME, FutureBus, etc.) Host Memory Processor Cache Host Processor Peripheral Bus Interface/DMA I/O Channel Interface Buffer Memory ROM µProc I/O Controller

31. I/O InterfaceCS510 Computer ArchitecturesLecture 17 - 31 I/O Data Flow I/O DeviceHead/Disk Assembly DMA over Peripheral Bus OS Buf fers (>10 MByte) Xfer over Serial Interface Track Buffers (32K - 256KBytes) Embedded Controller Memory-to-Memory Copy ApplicationAddress Space Host Processor Xfer over Disk Channel HBA Buffers (1 M - 4 MBytes)I/O Controller Impediment to high performance: multiple copies, complex hierarchy