1. Storage Protocols Storage Protocols Introduction
In order for a file system to access the physical device, a command protocol is needed. This lesson provides an introduction to the basic protocols that can be used in storage systems. It provides a high-level overview of SCSI, FCP, Fibre Channel, and iSCSI.
Importance
Having a basic knowledge of storage protocols enables the Systems Engineer (SE) to understand how storage is used by host systems and the key characteristics of different types of storage protocols. Additionally, understanding storage protocols will help the SE to assess possible performance bottlenecks that exist independent of Cisco's products.
© 2003, Cisco Systems, Inc. All rights reserved. 1

2. Lesson Objective
Upon completion of this module, you will explain the basic storage protocols of SCSI, FCP, iSCSI, and FCIP.
Performance Objective
Upon completion of this module, you will explain the basic storage protocols of SCSI,FCP, iSCSI, and FCIP.
Enabling Objectives
Explain the role of scsi in a storage environment, including references to SCSI-1, -2, and -3
Identify the performance metrics of various SCSI standards
Explain SCSI characteristics including the I/O channel, network view of SCSI, and the SCSI bus
Identify the three components of SCSI addresses
Explain the nature of half duplex communications in SCSI operation
Explain the function of SCSI encapsulation in relation to the distance limitations of SCSI
Explain the function of the various layers in FCP
Identify the various components of a simple Fibre channel SAN
Explain the role of iSCSI, including a definition, the importance of iSCSI,and a description of a basic iSCSI topology
Explain how the FCIP functions to facilitate data transfer over long distances
Explain how the FC data is loaded into TCP sessions
Give a basic explanation of an FCIP application topology
Explain the similarities and difference between three potential FCIP environments
Explain a diagram featuring FCP, FCIP, and iSCSI

3. Lesson Outline SCSI standards SCSI characteristics SCSI addressing
What is Parallel SCSI?
SCSI standards
SCSI characteristics
SCSI addressing
SCSI operation
SCSI encapsulation
Fibre Channel Protocol (FCP): Serial SCSI for FC SANs
A simple Fibre channel SAN
What is iSCSI?
The FCIP protocol
FCIP packet structure
An FCIP application topology
Potential FCIP environments
FCP, FCIP, and iSCSI together
Prerequisite
The qualifications to attend the program.

4. Two Parts What is Parallel SCSI?
In the beginning there was (Parallel) SCSI
Two Parts
What is Parallel SCSI?
Objective
Explain the role of scsi in a storage environment, including references to SCSI-1, -2, and -3.
Introduction
This section defines the acronym "scsi" and explains scsi's function in a storage environment.
Facts
SCSI is an acronym for "small computer system interface." It is pronounced "scuzzy". SCSI is a parallel interface standard. In this context, "parallel" means that multiple bits are being sent simultaneously over a given distance.
SCSI defines a set of commands used to access a wide range of devices. This command protocol is the most widely used protocol in the SAN (Storage Area Network) environment. The command protocol can be transported across several different interfaces. The original interface used was the SCSI bus. More recently, SCSI commands have been mapped to the Fibre Channel and IP/Ethernet interfaces.
SCSI is the most important storage protocol—the serial form of SCSI is the basis for modern SANs. SCSI began as a complete integrated environment mixing communications and commands. It was not referred to as parallel SCSI, but just SCSI, because there was no other type.
SCSI was developed by Shugart Associates and NCR Corporation in 1981 as part of a disk drive interface called SASI. Its name was eventually changed to SCSI. Attempts to make SASI an ANSI standard failed. In 1982 the ANSI committee X3T9.2 was formed. The first true SCSI standard was published in Since that time, the ANSI committee has continued to refine the SCSI specification. SCSI has been a successful technology, and will probably be around for some time to come.
Continued …
Physical Parallel Bus
Logic Commands and Control

5. What is Parallel SCSI? (cont.)
Developed for speed and reliability
Parallel bus
Control Information
What is Parallel SCSI? (cont.)
Facts
Parallel SCSI was originally developed for internal disk and tape connections using ribbon cables. Over the years, there have been many varieties of parallel SCSI, with a broad range of connectors and cables.
It is fast and reliable, but distances are short. This physical manifestation gave rise to a master/slave relationship between host system host adapters and their connected storage.
Data Information

6. SCSI Standards Name Bus Width Bits Bus Speed MHz
Transfer Speed Mbytes/Sec
Maximum Cable Length meters
SCSI-1 8 5 25
Fast SCSI 10
Wide SCSI 16
Fast Wide SCSI 20
Ultra SCSI
Wide Ultra SCSI 40
Ultra 2 SCSI
Wide Ultra 2 SCSI 80
Ultra 160 SCSI
160 12
Ultra 320 SCSI
320
SCSI Standards
Objective
Identify the performance metrics of various SCSI standards.
Introduction
This section delineates important performance metrics for the various SCSI standards.
Facts
The table documents the SCSI standards with respect to bus width, speed and transfer rates.
Practice
Answer the following questions.:
Which is faster, Ultra SCSI or Wide Ultra 2 SCSI?
Which is faster, Wide Ultra SCSI or Fast SCSI?
Which is faster, Wide Ultra 2 SCSI or Ultra 320 SCSI?
Which has the greater maximum cable length, Ultra 320 SCSI or Wide SCSI?

7. SCSI Characteristics • Parallel implementation
• SCSI uses half-duplex communications
• Parallel implementation
Bus width: 8, 16 bits
Bus speed: 5–80 Mhz
Throughput: 5–320 MBps
Devices/Bus: 2–16 Devices
Cable length: 1.5m–25m
LUNs/Target: 64
SCSI Characteristics
Objective
Explain SCSI characteristics, including the I/O channel, network view of SCSI, and the SCSI bus.
Introduction
This section introduces the basic SCSI characteristics, including the I/O channel, network view of SCSI, and the SCSI bus.
Facts
The I/O channel is the transport technology that resides between a computer and its storage devices. It can be internal to the computer enclosure or can extend to external storage devices.
The storage device is accessible by host bus adapters, called HBAs.
The parallel SCSI protocol stack resides alongside the network protocol stack.
NB: While I ist theoretically possible to have 64 LUNs per target, in practice, parallel SCSI implementations have at most 8 LUNs per target.

8. SCSI Characteristics (cont.)
Facts
The parallel SCSI bus was built by daisy chaining devices together on the bus.
There are no hubs or switches with parallel SCSI—it is a shared bus architecture.
Access to the bus is determined by SCSI arbitration, which uses the address to determine priority
It is possible to have multi-initiator SCSI (used in clustering scenarios), but this is not very common. Historically, it was assumed that a single host adapter would control the SCSI devices connected to the SCSI bus.
SCSI Bus

9. SCSI Characteristics (cont.)
Facts
The relationship between host bus adapters and targets is a MASTER/SLAVE relationship. SCSI is a command/response protocol.
SCSI Architecture Model

10. SCSI Addressing SCSI Addressing Objective
Identify the three components of SCSI addresses.
Introduction
This section introduces basic SCSI addressing.
Facts
The diagram illustrates the addressing hierarchy of parallel SCSI and illustrates multi-initiator mode. SCSI addresses have 3 components:
Host (controller in the host): Identifies which HBA is used
Target (device controller on the SCSI bus): Identifies the SCSI bus address (there are a max of 16)
LUN (storage device at the target): Identifies a storage device or resource that communicates through a target controller
This same hierarchy is still used in modern SANs.
Practice
List the 3 components of SCSI addresses.

11. SCSI Operation SCSI Operation Objective
Explain the nature of half duplex communications in a SCSI operation.
Introduction
This section introduces a basic SCSI operation.
Facts
The graphic illustrates the nature of the half duplex communications in SCSI.
Host/target pairs in SANs are also half duplex communications, but hosts and targets can have multiple sessions simultaneously with different node partners.

12. SCSI: A Network Application
SCSI Encapsulation
SCSI: A Network Application
(iSCSI)
(Fibre Channel)
Transport
SCSI Encapsulation
Objective
Explain the function of SCSI encapsulation in relation to the distance limitations of SCSI.
Introduction
This section introduces the importance of SCSI encapsulation.
Facts
In addition to the original parallel SCSI standard, SCSI-3 allows for the transmission of SCSI commands over networked serial transports, including Ethernet and Fibre Channel.
Serial SCSI allows for much greater transmission distances than parallel SCSI, which is limited to 25 meters. Serial SCSI transports such as Fibre Channel and Ethernet can extend distances up to 10 kilometers and farther.
From the network’s perspective, serial SCSI is an APPLICATION. This is not necessarily intuitive, as parallel SCSI provided the whole integrated stack down to the physical layer, but in SANs, SCSI is an application that uses an underlying network for transport.
Routing IP
MAC
Physical

13. Fibre Channel Protocol (FCP): Serial SCSI for FC SANs
FCP is implemented to the specifications of Fibre Channel’s FC-4 application abstraction layer
FCP (serial scsi)
FC-4: Protocol Mapping
Fibre Channel Protocol (FCP): Serial SCSI for FC SANs
Objective
Explain the function of the various layers in FCP.
Introduction
This section introduces the functions of the various layers in FCP.
Facts
The implementation of serial SCSI over Fibre channel (FC) is called FCP. There are many other protocols that have been ported to Fibre Channel’s abstraction layer, but FCP is by far the most heavily used.
FC-0 defines the physical interface, including the types of cabling, which may be optical or electrical, and connectors
FC-1 provides low level link controls and data encoding for gigabit transport
FC-2 defines segmentation and reassembly of data via frames, flow control and classes of service
FC-3 is for common services, such as data encryption and compression
FC-4 is the upper layer protocol interface between FC and IP, SCSI-3, and other protocols
FC-3: Common Services
FC-2: Framing
FC-1: Link
FC-0: Physical

14. A Simple Fibre Channel SANFC FC FC
FC storage
subsystems FC FC
Servers with FC HBAs
FC switch FC
A Simple Fibre Channel SAN
Objective
Identify the various components of a simple FC SAN.
Introduction
This section introduces the components of a simple FC SAN.
Facts
The diagram illustrates a simple FC SAN. It is displaying the storage network back-end, that is, not the company LAN on the left.
Each host is equipped with an FC HBA and device driver, and is connected directly to an FC switch. The FC disk subsystems and tape storage subsystem are also directly attached to the FC switch. FC FC
Tape storage
subsystem

15. Servers with iSCSI drivers or iSCSI NICs with TOE
What is iSCSI?
iSCSI
iSCSI FC
iSCSI
Ethernet FC
Servers with iSCSI drivers or iSCSI NICs with TOE
Fibre Channel storage FC
iSCSI
Ethernet Switch
iSCSI to FC Gateway
What is iSCSI?
Objective
Explain the role of iSCSI, including a definition, the importance of iSCSI,and a description of a basic iSCSI topology.
Introduction
This section gives a basic overview of iSCSI, including what iSCSI is, and iSCSI topology.
Facts
In 1998 IBM completed a proof of concept for transporting SCSI commands over IP. Two years later, in March of 2000, with original contributions from IBM and Cisco, the first draft of the Internet SCSI (iSCSI) standard was presented to the Internet Engineering Task Force (IETF).
Developed by the IETF, iSCSI is a new IP-based storage networking standard for linking data storage facilities. By carrying SCSI commands over IP networks, iSCSI facilitates data transfers over intranets and can access storage over long distances. The iSCSI protocol is among the key technologies expected to facilitate rapid development of the SAN market, by increasing the capabilities and performance of storage data transmission. Because of the ubiquity of IP networks, iSCSI can be used to transmit data over LANs, wide area networks (WANs), or the Internet, and can enable location-independent data storage and retrieval (see
The diagram illustrates a simple SAN topology using a combination of iSCSI and FC. The servers are using either standard Ethernet NICs with iSCSI drivers installed, or special iSCSI NICs that include a TCP/IP Offload Engine (TOE) for increased performance. It may use Gigabit Ethernet network technology, or in some cases it may use 100 Mb Ethernet.
Continued …
iSCSI
iSCSI

16. Direct Attached SCSI Disk
What is iSCSI? (cont.)
Applications
iSCSI
Server
Data Base
SCSI Driver
TCP/IP
File System
Fibre Channel
Gigabit Ethernet
SCSI Tape
SCSI Disk
iSCSI Gateway
SCSI Generic
Fibre Channel Hub or Switch
iSCSI Driver
SCSI Adapter Drivers
5420
What is iSCSI? (cont.)
Facts
The graphic illustrates a more complex diagram of a network using iSCSI.
Note that the host computer on the right, and its applications, have an existing TCP/IP connection to other computers on the network. In addition, the host computer has a direct connection to its local SCSI disk (direct attached storage).
On the left side of the diagram a fibre channel attached storage array with dual controllers is attached to dual fibre channel switches. The fibre channel switches in turn are attached to dual iSCSI gateway devices. This highly redundant design provides multiple paths from the host to the disk array, reducing single points of failure.
The data path from host to disk array begins on the right, where SCSI commands are encapsulated in iSCSI packets for transmission across the TCP/IP network. These packets pass through the TCP/IP network to one of the iSCSI gateways where the TCP/IP and iSCSI headers are removed and replaced with FC headers. The FC frames are then placed on the FC storage network and routed to the destination disk array. The return data path reverses this process.
Host TCP/IP
Network Drivers
Controller
Controller
NIC
HBA
UNIX or NT Host
Direct Attached SCSI Disk
Disk Array

17. End-to-End Native iSCSI Environment
What is iSCSI? (cont.)
End-to-End Native iSCSI Environment
Initiators—iSCSI HBAs (or NICs) function as SCSI ‘masters’
Targets and LUNs—iSCSI devices and subsystems take the role of SCSI ‘slaves’
SCSI Command Descriptor Blocks (CDB)—Storage operations are conveyed in standard CDBs
iSCSI
iSCSI
What is iSCSI? (cont.)
Facts
SCSI is a master/slave (command/response) protocol.
iSCSI initiators are typically Ethernet network interface cards (NICs) with integrated SCSI functions that originate SCSI commands
iSCSI targets are typically subsystems with Ethernet network ports that are capable of executing SCSI commands
iSCSI LUNs are SCSI standard sub-addresses that represent virtual (or real) storage resources in the SAN
iSCSI uses standard SCSI CDBs (command descriptor blocks - discussed in Module 1, Evolution of Storage Solutions, Lesson 4, File Systems lesson), and are the same as with parallel SCSI and Fibre Channel FCP
NB - the configuration in this diagram is not common at present, but will come into increasing use.
Ethernet switch
SCSI ‘Master’ Initiator
SCSI ‘Slave’ Target w/optional LUNs
iSCSI: Standard Serial SCSI

18. The FCIP Protocol FCIP—Fibre Channel over Internet Protocol
The Encapsulation of Fibre Channel frames into IP packets and tunneled through an existing IP network infrastructure
FC Server
FC Tape Library
FC Server
FC Tape Library
FC Switch
FC Switch
FC Switch
FC Switch
Fibre Channel SAN
IP Network
Fibre Channel SAN
The FCIP Protocol
Objective
Explain how the FCIP functions to facilitate data transfer over long distances.
Introduction
This section gives a basic overview of the FCIP protocol.
Facts
Another SAN protocol is the Fibre Channel over IP (FCIP or FC/IP, also known as Fibre Channel tunneling or storage tunneling). FCIP is used to extend FC SANs over MAN (metropolitan area networks) or WAN (wide area network) links. This creates a single fabric out of two remote SANs. Servers and storage in both SANs see the same resources and services.
FCIP is an Internet Protocol-based storage networking technology. FCIP mechanisms enable the transmission of FC information by tunneling data between SAN facilities over IP networks; this capacity facilitates data sharing over a geographically distributed enterprise. FCIP is among the key technologies expected to help facilitate rapid development of the SAN market by increasing the capabilities and performance of storage data transmission. (see
The pricing of iSCSI NICs with TOE is not necessarily less than FC HBAs. However, a cost benefit of iSCSI is using it to bring low and mid-range servers with existing NICs into an existing FC SAN. The addition of more expensive TOE NICs will bring improved performance not only for the storage network, but for other network I/O as well.
Tunnel Session
FC Switch
FC Switch
FC Switch
FC Switch
FC Server
FC disk subsystem
FC disk subsystem
FC Server

19. FCIP Packet Structure TCP/IP is the underlying transport protocol
Flow Control/Retransmission during Network Congestion
In-order packet delivery of error-free data
FC frames are encapsulated as Datagrams
Fragmentation and re-assembly of FCP data
IP is unaware of the Fibre Channel Payload and the Fibre channel fabric is unaware of the IP network
FCIP forms a virtual FC Inter Switch Link (ISL)
FCIP Packet Structure
Objective
Explain how the FC data is loaded into TCP sessions.
Introduction
This section gives a basic overview of the FCIP packet structure.
Facts
Like other tunnelling protocols, FCIP loads the entire FC protocol data unit (PDU) into TCP sessions
The FC data is fragmented and needs to be re-assembled at the other end of the FCIP connection. This is necessary because the MTU (max transmission unit) of FC, is larger than the MTU of ethernet.
FCIP and the underlying hardware and network(s) form a virtual inter switch link between a pair of Fibre channel switches
Practice Items
Why does the FC data need to be fragmented and reassembled?
Data Link
Header
(Ethernet) IP
Checksum
TCP
Hdr.
FCIP
FC Encoded Data
(SCSI Data + CRC) FC S O F E

20. An FCIP Application Topology
FCIP Gateways perform Fibre Channel encapsulation process into IP Packets and reverse that process at the other end
FC Switches connect to the FCIP gateways through an E_Port for SAN fabric extension to remote location
A tunnel connection is set up through the existing IP network routers and switches across LAN/WAN/MAN
Servers
Servers FC
SAN FC
SAN
An FCIP Application Topology
Objective
Provide a basic explanation of an FCIP application topology
Introduction
This section provides a basic overview of an FCIP application topology.
Facts
The diagram illustrates how the storage subsystem on the left side has its data copied to the remote storage site on the right side.
All the servers and storage and servers in this diagram appear to be local to each other.
Practice Items
What is FCIP used for?
How will this impact the adoption of SAN technology?
Storage
Existing IP
Network
LAN/WAN/MAN
FCIP
Gateway
EMC SRDF
FCIP
Gateway
FC Switch
FC Switch
Storage
Backup, R&D, Shared Storage,
Data Warehousing, etc.
Production Site
Standby
Production

21. Potential FCIP Environments
Local Datacenter
Remote Datacenter
1Gb->OC48 or Higher
Relatively low latency
Synch/Asynch Applications
Gateway
Gateway
Metro Ethernet
FCIP
FCIP
SAN
SAN
Short distance ~ <= 60km
Local Datacenter
Remote Datacenter
Typical OC3 / OC12
Relatively low latency
Mainly asynchronous
Suitable for some synchronous apps
Gateway
Gateway
SONET
FCIP
FCIP
SAN
SAN
Potential FCIP Environments
Objective
Explain the similarities and difference between three potential FCIP environments.
Introduction
This section provides a basic overview of an FCIP application topology.
Facts
One advantage of TCP/IP based network traffic is that it can pass over many types of transports and cover wide distances. FCIP enables FC based storage traffic to traverse local, metropolitan and wide area network topologies.
A metropolitan area network (MAN) is a network that interconnects users and computer resources in a geographic area or in a region typically less than 100 km in diameter. Metropolitan Ethernet providers use a combination of fibre, dense wavelength division multiplexing (DWDM) and Ethernet to build less expensive networks than those based on Synchronous Optical Network (SONET) equipment.
SONET is the ANSI standard for synchronous data transmission of digital signals over optical media. The international equivalent of SONET is synchronous digital hierarchy (SDH). Together, they insure standards so that digital networks can interconnect internationally, and that existing conventional transmission systems can take advantage of optical media through tributary attachments.
Signal rates for transmitting digital signals on optical fiber are referred to as optical carrier rates, or OC-X, and range in multiples of Mbps. For example, OC-3 is Mbps, OC-12 is Mbps and OC-48 is Gbps.
Continued …
Medium distance ~ <= 160km
Local Datacenter
Remote Datacenter
Low speed (T1–DS3)
Higher latency
Longer distance
Mainly asynchronous
Gateway
Gateway
IP Routed WAN
FCIP
FCIP
SAN
SAN
Long distance > 160km

22. FCIP: SAN-to-SAN over IP iSCSI: Host to FC-SAN over IP
FCP, FCIP and iSCSI
FCP: Local SANs
FCIP: SAN-to-SAN over IP
iSCSI: Host to FC-SAN over IP
iSCSI
iSCSI
iSCSI gateway
IP Network
iSCSI gateway
FCP, FCIP, and iSCSI on the same network
Objective
Explain a diagram featuring FCP, FCIP, and iSCSI.
Introduction
This section gives a basic overview of a network featuring FCP, FCIP, and iSCSI.
Facts
The diagram illustrates a network with all three protocols:
FCP
FCIP
iSCSI
The host systems at the top are all using iSCSI HBAs, and communicate to FC storage subsystems in the two SANs through a pair of iSCSI gateways. A pair of FCIP gateways connect the two SANs together for the purpose of business continuity (for example, remote mirroring).
FCP
FCP
FCIP
FCIP gateway
FCIP gateway

23. SCSI Characteristics Practice
Client/Initiator
Application Client
Practice: SCSI Characteristics
Instructions
Label the SCSI architecture diagram and explain the components' functions.
SCSI Architecture Model

24. The FCIP Protocol Practice
“Business continuance" describes the processes and procedures an organization puts in place to ensure that essential functions can continue during and after a disaster. Business continuance planning seeks to prevent interruption of mission-critical services, and to reestablish full functioning as swiftly and smoothly as possible.
Based on this definition, explain the importance of FCIP in facilitating business continuance.
Practice: The FCIP Protocol

25. Parallel SCSI Practice
Answer the following questions. Be prepared to share your answers with the class.
What is SCSI used for?
How is SCSI-3 different from SCSI-1 and SCSI-2?
What is one of the major limitations of parallel SCSI for use in SANs?
Practice: Parallel SCSI

26. Fibre Channel Protocol (FCP): Serial SCSI for FC SANs Practice
Identify and explain the function of each layer
FCP
FC-4
FC-3
FC-2
Practice: Fibre Channel Protocol (FCP): Serial SCSI for FC SANs
Instructions
1. Complete the labeling of the boxes
2. Explain the function of each layer
FC-1
FC-0

27. SCSI Encapsulation Practice
Answer the following question and be prepared to share your answers with the class.
Why is SCSI encapsulation crucial for overcoming the limitations of SCSI?
Practice: SCSI Encapsulation

28. What is iSCSI? Practice
Answer the following questions. Be prepared to share your answers with the class.
1. What is iSCSI used for?
2. List the advantages of iSCSI.
Practice: What is iSCSI?

29. Potential FCIP Environments Practice
Explain the similarities and differences between the following potential FCIP environments:
1. Metro ethernet
2. SONET
3. IP Routed WAN
Practice: Potential FCIP Environments
Instructions
Explain the similarities and differences between the three potential FCIP environments, including distances.

30. SCSI, iSCSI and FCIP Lesson Practice
Answer the following questions (or as many as your instructor assigns). Be prepared to share your answers with the class.
Complete the chart below:
2. Describe each of the following protocols:
Parallel SCSI iSCSI FCP FCIP
Optional: Create a diagram for each showing how it might be used in a real world situation.
Key Benefits/Features
SCSI-1
SCSI-2
SCSI-3
Practice: SCSI, iSCSI and FCIP

31. Summary This lesson presented these key points:
SCSI - the most important command protocol
iSCSI (SCSI over IP)
FCIP (Fibre channel over IP)
Summary: Protocols
SCSI is the most important storage protocol—the serial form of SCSI is the basis for modern SANs. There have been a number of standards over the years. SCSI uses half-duplex communications and is characterized by parallel implementation. SCSI is a command/response protocol. There are three components to SCSI addressing. Serial SCSI allows for greater transmission distances than parallel SCSI. Serial SCSI can be implemented over Fibre Channel. SCSI commands can also be transported over IP (iSCSI). Another SAN protocol is the Fibre Channel over IP (FCIP or FC/IP, also known as Fibre Channel tunneling or storage tunneling). FCIP is used to extend FC SANs over MAN (metropolitan area networks) or WAN (wide area network) links.
Continued…