1. 1 © 2004 Cisco Systems, Inc. All rights reserved. iSCSI Overview IP Storage Networking FCIP/iSCSI Steve Tegeler Storage Networking Team Northwest Territory 425/468-0836 stegeler@cisco.com

2. 222 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Agenda Storage Networking Technology Review IP Storage Networking FCIP Write Acceleration, Compression, IPSec, SAN Extension Tuner FCIP Wizard iSCSI What, Why, How Performance iSCSI Terminology and Topology Design considerations when deploying iSCSI Summary

3. 333 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Agenda Storage Networking Technology Review iSCSI and IP Storage Networking What, Why, How Performance iSCSI Terminology and Topology Design considerations when deploying iSCSI Summary

4. 444 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 The Old Storage Environment Direct Attached Storage (DAS) Storage is captive ‘behind’ the server Server CPU must handle user I/O requests, but also: User-database inquiries User file/print serving Data-integrity checking Communication with other devices Data access is file system and platform dependant Costly to scale; complex to manage FC Clients SCSI FC Direct-Attached Storage (DAS) Servers Win2kLinuxWin2kLinuxUnix IP Network

5. 555 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 The SCSI I/O Channel SCSI is the dominant protocol used to communicate between servers and storage devices in open system SCSI I/O channel is a half-duplex pipe for SCSI CDBs and data Parallel bus evolution Bus width: 8, 16 bits Bus speed: 5–80 Mhz Throughput: 5–320 MBps Devices/bus: 2–16 devices Cable length: 1.5m–25m A network approach can scale the I/O channel in many areas (length, devices, speed) SCSI CDB: SCSI Command Descriptor Block Used to Relay SCSI Commands, Parameters, and Status between SCSI Initiators and SCSI Targets; Typically 6, 10, or 12 Byte Block SCSI Adapter Applications File System Block Device SCSI Generic TCP/IP Stack NIC Driver Adapter Driver Half-Duplex SCSI I/O Channel SCSI Initiator SCSI Target SCSI Raw Ethernet NIC Ethernet

6. 666 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Networking the I/O Channel Same SCSI protocol (SCSI-3) carried over a network transport layer via serial implementation Transport must not jeopardize SCSI payload (security, integrity, latency) Two primary transports to choose from today: Fibre Channel and IP A networked I/O channel allows for multiple improvements: Distance limitations greatly increased High number of addressable devices Initiator Target and LUNs Networked I/O Channel Channel Controller SCSI Host System Network

7. 777 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Fibre Channel Networking Very common method for networking SCSI Fibre Channel provides high-speed transport for SCSI payload Fibre Channel SAN overcomes many shortcomings of DAS including: Addressing for up to 16-million nodes (24 bits) Loop (shared) and Fabric (switched) transport Speeds of 100 or 200 MBps (1 or 2 Gbps) Distance of up to 10km (without extenders) Can utilize CWDM or DWDM for over 10km Support for multiple protocols Combines best attributes of a channel and a network Fibre Channel HBA Fibre Channel Fabric Initiator SCSI Host System Target

8. 888 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 IP: An Alternate I/O Transport Viable transport for I/O traffic Not necessarily for long-haul I/O only Similar characteristics to Fibre Channel: Addressing for close to 4 billion nodes (IPv4) Primarily a switched transport (with routing) Ethernet speeds of 10/100 Mbps or 1/10 Gbps or various WAN speeds Support for multiple high-level protocols Cost and manageability advantages with IP IP knowledge base widespread in industry IP “Channel Adapter” Target SCSI Host System Initiator IP Network

9. 999 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 IP Storage Networking IP storage networking provides solution to carry storage traffic within IP Uses TCP: a reliable transport for delivery Applicable to local data center and long-haul applications Two primary protocols: iSCSI—Internet-SCSI—used to transport SCSI CDBs and data within TCP/IP connections FCIP—Fibre-Channel-over-IP—used to transport Fibre Channel frames within TCP/IP connections—any FC frame—not just SCSI IP TCP FCIP FC SCSIData IP TCP iSCSI SCSIData

10. 10 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 FCIP – Extending your FC SAN 10 © 2004, Cisco Systems, Inc. All rights reserved.

11. 11 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Fibre Channel over IP (FCIP) Point to Point FCIP – Fibre Channel over Internet Protocol The encapsulation of Fibre Channel frames into IP packets and tunneling through an existing TCP/IP network infrastructure, in order to connect geographically distant islands LAN/MAN/WAN FCIP Tunnel Session FC Disk SAN Ethernet Catalyst Switches & Routers Optical Extension Metro DWDM & CWDM IPS Sync or Async Replication E-port FCIP tunnels can be thought of as ISL’s with Latency

12. 12 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 iSCSI 12 © 2004, Cisco Systems, Inc. All rights reserved.

13. 13 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Audience Poll Who has a FC Network today? Who has deployed iSCSI, Array based, or gateway based?

14. 14 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 What is iSCSI? A SCSI transport protocol that operates over TCP/IP Encapsulates SCSI CDBs (operational commands: e.g. read or write) and data into TCP/IP byte streams Allows IP hosts to access IP-based SCSI targets (either natively or via iSCSI to FC Gateways) Standards status RFC 3720 on iSCSI Collection of RFCs describing iSCSI RFC 3347—iSCSI Requirements RFC 3721—iSCSI Naming and Discover RFC 3723—iSCSI Security Broad industry support Server vendors now publishing own supported iSCSI drivers Native iSCSI storage arrays now appearing

15. 15 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 SCSI Block Commands SCSI Stream Commands Parallel SCSI Transport SCSI Applications (File Systems, Databases) Parallel SCSI Interfaces SCSI Device-Type Commands SCSI Generic Commands SCSI Transport Protocols Layer 3 Network Transport Layer 2 Network Fibre Channel Ethernet, PPP, HDLC… Other SCSI Commands IP TCP SCSI Commands, Data, and Status Recap SCSI Architectural Model Transports FCP SCSI over FC iSCSI SCSI over TCP/IP

16. 16 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Why - iSCSI vs. FCIt’s all about the $$$ Leverage IP infrastructure for storage connectivity Low-cost complement to FC SAN, provides additional resource consolidation 1000BaseT NIC + GigE Port: Roughly $100 + $300 = $400 FC HBA+ FC Port: Roughly $1000 + $1000 = $2000 Secure connectivity via CHAP-based authentication Transparent iSCSI routing gives iSCSI hosts a pWWN Uses controller-based LUN masking or MDS-based virtual targets for resource provisioning Uses zoning for device connectivity iSCSI driver (free) works with any Ethernet NIC TOE only necessary with processor-bound servers iSCSI is an industry-supported IETF standard Many O/S vendors providing iSCSI initiator (MS, HP, Novell, Linux), others provided by Cisco iSCSI-enabled Hosts FC Disk Array Catalyst Ethernet Switch IP Network Cisco MDS 9000 with IP Services Module iSCSI FC Servers FC Tape Library

17. 17 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 iSCSI for Storage Consolidation IP access to open systems iSCSI and Fibre Channel storage iSCSI driver is loaded onto hosts on Ethernet network Able to consolidate servers via iSCSI onto existing storage arrays Able to build Ethernet-based SANs using iSCSI arrays Storage assigned on a LUN-by- LUN basis at iSCSI router iSCSI-Enabled Hosts (Initiators) iSCSI Array (Target) Storage Pool (Target) iSCSI Gateway IP Network FC Fabric FC HBA- Attached Host (Initiator) iSCSI Logical Unit Number (LUN): A Field within SCSI Containing up to 64 Bits that Identifies the Logically Addressable Unit within a Target SCSI Device iSCSI

18. 18 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 iSCSI for Remote Block Access Block access to remote storage over IP Application must tolerate latency for long distances Metro Ethernet services offer lower-latency transport alternative Remote backup over IP WAN Centralized management from centralized storage iSCSI-Enabled Host Remote Mirrors IP WAN Storage Pool FC Fabric iSCSI Devic e Site A Site B iSCSI

19. 19 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 How - iSCSI Architecture: Software Driver iSCSI GW Module SCSI Driver TCP/IP Driver FC HBA GigE NIC NIC SCSI Adapter File System Block Device SCSI Generic TCP/IP Stack NIC Driver Adapter Driver iSCSI iSCSI GW Device Host iSCSI Host Driver Conventional SCSI Path iSCSI Path IP Network Fibre Channel Applications

20. 20 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 OS Support Many operating systems supported via Cisco drivers and/or from OS vendor Cisco provides full-driver suite Solaris 2.6 (EOL),7,8,9 Linux-based on 2.6 kernel Win 2000 with SP2 or later Windows XP Pro WinNT 4.0 with SP6A HP/UX 10.2, 11.0 AIX 4.3.3, 5.1, 5.2 OS vendors support native iSCSI drivers Windows *native* Win 2000, XP, 2003 support HP *native* HP/UX 11i support IBM *native* AIX 5.x support Novell Netware *native* support Solaris 10 (March 2005) Linux (RedHat Suse) NIC Adapter SCSI Adapter Adapter Driver iSCSI SCSI Generic Applications File System Block Device TCP/IP Stack NIC Driver iSCSI Software Driver

21. 21 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 iSCSI HBAs and TCP Offload Engines (TOEs) Offloads TCP and, optionally, iSCSI processing into hardware Relieves host CPU from: TCP processing—16-bit checksum per packet iSCSI—optional 32-bit header and data digests (CRC32C) TCP Offload iSCSI and TCP Offload File System Block Device SCSI Generic HBA Driver TCP/IP Stack NIC Driver iSCSI Driver TOE Driver SCSI Adapter Adapter Driver TCP/IP Stack iSCSI TCP/IP Stack Applications

22. 22 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Agenda Storage Networking Technology Review iSCSI and IP Storage Networking What, Why, How Performance iSCSI Terminology and Topology Design considerations when deploying iSCSI Summary

23. 23 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Example performance impact on CPU util % FC vs. iSCSI TOE vs. iSCSI SW Driver CPU % Throughput MB/s FC HBA iSCSI TOE iSCSI SW Driver 15-35MB/s Inflection point determined by system resources (CPU/Memory) ?

24. 24 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Agenda Storage Networking Technology Review iSCSI and IP Storage Networking What, Why, How Performance iSCSI Terminology and Topology Design considerations when deploying iSCSI Summary

25. 25 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 iSCSI Naming Initiator and target require iSCSI names Name is location independent iSCSI node name = SCSI device name of iSCSI device Associated with iSCSI nodes, NOT adapters Up to 255-byte displayable/human readable string (UTF-8 encoding) Use SLP (Service Location Protocol) V2, iSNS, or query target for names (SendTargets) Two iSCSI name types: iqn—iSCSI qualified name eui—Extended Unique Identifier (IEEE EUI-64— also used for FC WWNs)

26. 26 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 – – – iSCSI Name Structure – Unique String iqn.1987-05.com.cisco.1234abcdef987601267da232.betty iqn.2001-04.com.acme.storage.tape.sys1.xyz Type Date Organization Naming Authority Subgroup Naming Authority or String Defined by Organization Naming Authority iqn eui Date = yyyy-mm When Domain Acquired Reversed Domain Name – Type EUI-64 Identifier (ASCII Encoded Hexadecimal) eui.02004567a425678d Type

27. 27 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 iSCSI Connectivity iSCSI Initiator knows IP and IQN FC Target knows WWN and FCID iSCSI HBA HBA iSCSI HBA HBA FC Fibre Channel Fabric iqn.host-3 IP-10.1.1.4 iqn.host-1 IP-10.1.1.2 pWWN – P6 nWWN –N6 FCID – XXXX06 pWWN – P5 nWWN – N5 FCID –XXXX05 IP-10.1.1.1 IP Network iSCSI HBA HBA iqn.host-2 IP-10.1.1.3

28. 28 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 iSCSI Mapping to a WWN Each iSCSI Initiator gets a unique WWN and FCID iSCSI HBA HBA iSCSI HBA HBA FC Fibre Channel Fabric iqn.host-3 IP-10.1.1.4 iqn.host-1 IP-10.1.1.2 pWWN – P6 nWWN –N6 FCID – XXXX06 pWWN – P5 nWWN – N5 FCID –XXXX05 IP-10.1.1.1 iSCSI HBA HBA iqn.host-2 IP-10.1.1.3 pWWN – P2 nWWN- N2 FCID XXXX02 pWWN – P4 nWWN- N4 FCID XXXX04 pWWN – P3 nWWN – N3 FCID XXXX03

29. 29 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Agenda Storage Networking Technology Review iSCSI and IP Storage Networking What, Why, How Performance iSCSI Terminology and Topology Design considerations when deploying iSCSI Summary

30. 30 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Considerations when building an iSCSI Fabric iSCSI fabric topology Ethernet fabric topology iSCSI fabric scalability Trunking Port channeling iSCSI fabric availability VRRP iSCSI fabric security Authentication and binding iSCSI fabric manageability iSCSI identity and management Scalability Availability Security Manageability iSCSI Clients ENDTO ENDENDTO END ENDTO ENDENDTO END Shared Storage Pool iSCSI IPSIPS

31. 31 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Dedicated IP Storage Network Separate logical IP network but not necessarily separate physical network Can use a VLAN of existing Ethernet network Recommend use of dedicated NIC on host for iSCSI Minimized potential for bandwidth contention iSCSI-Enabled Hosts Storage Pool iSCSI Routers Catalyst Switches Dedicated IP Storage Network FC Fabric Clients Front-Side IP Network FC-Attached Hosts with HBAs iSCSI

32. 32 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Performance Objectives and Determining Factors Understand performance objectives Number of users Number of I/O requests Acceptable response time Desired throughput Factors impact performance System resources (CPU, memory, bus architecture) Storage resources (RPM, cache, RAID level) Network topology/equipment/gateway Available IP network bandwidth (especially in WAN) Distance between iSCSI initiators and targets TCP implementation and configuration I/O block size

33. 33 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 IP Network Security Techniques Firewall Standalone or intelligent firewall service module Allow well-known TCP port 3260 for iSCSI IPSec VPN VPN tunnel for iSCSI remote access Access Control List (ACL) VLAN and PVLAN Subinterface implementation on iSCSI Separated VLAN for iSCSI Port security Allow, block, or restrain access to Ethernet based on MAC address

34. 34 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 What is iSNS? Internet Storage Name Service (iSNS) is a name registration service for IP storage devices: Analogous to FCNS and DNS Provides centralized management capabilities iSNS supports: Target device discovery Discovery Domains (similar to zones) Authentication State change notification Supports iSCSI and iFCP

35. 35 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 What is iSNS? (cont.) IP FC iFCP Gateway IP iSNS server FC iSCSI IP iSCSI FC iSNS

36. 36 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Agenda Storage Networking Technology Review iSCSI and IP Storage Networking What, Why, How Performance iSCSI Terminology and Topology Design considerations when deploying iSCSI Summary

37. 37 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Summary Leverages the existing IP infrastructure Hence the intelligence, capacity, and best practice design can be leveraged in the iscsi infrastructure Complementary to FC yet represents a low-cost transport choice Midrange applications connectivity Midrange server connectivity with blade server integration as new system candidate Potential long-distance SAN transport

38. 38 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Reference Materials http://www.t10.org/ http://www.t11.org/index.htm http://www.ietf.org/rfc.html on RFC 3720http://www.ietf.org/rfc.html http://www.cisco.com/en/US/partner/products/hw/p s4159/index.htmlhttp://www.cisco.com/en/US/partner/products/hw/p s4159/index.html http://www.lightreading.com/webinar_archive_hom e.asp?webinar_id=27003http://www.lightreading.com/webinar_archive_hom e.asp?webinar_id=27003

39. 39 © 2004 Cisco Systems, Inc. All rights reserved. OPT-2053 9761_05_2004_c2 Appendix SCSI—Small Computer System Interface iSCSI—internet SCSI DAS—Direct Attached Storage FC—Fibre Channel CDB—Command Descriptor Block R2T—Ready To Transfer LUN—Logical Unit Number SLP—Service Location Protocol IQN— iSCSI Qualified Name EUI—Extended Unique Identifier iSNS—Internet Storage Name Service TOE—TCP Offload Engine NFS—Network File System GPFS—General Parallel File System DMZ—Demilitarized Zone MZ—Militarized Zone IVR—Inter-VSAN Routing GTS—Generic Traffic Shaping FRTS—Frame-Relay Traffic Shaping CAR—Committed Access Rate PQ—Priority Queuing