1. 11/05/07 1TDC564-08 TDC 564 Local Area Networks Lecture 8 IP-based Storage Area Network

2. 11/05/07 2TDC564-08 Course Outline IP over FC (RFC 2625) IP over FC (RFC 2625) IP-SAN IP-SAN iSCCI (RFC 3720) iSCCI (RFC 3720) IP and FC-SAN Interworking IP and FC-SAN Interworking FC Encapsulation (RFC 3643) FC Encapsulation (RFC 3643) FCIP (RFC 3821) – FC over IP FCIP (RFC 3821) – FC over IP iFCP (RFC 4172) iFCP (RFC 4172)

3. 11/05/07 3TDC564-08 RFC 2625 – IP and ARP over FC FC supports multiple higher layer protocols, and SCSI is the most widely used one. FC supports multiple higher layer protocols, and SCSI is the most widely used one. Needs/Applications of IP over FC: Needs/Applications of IP over FC: Access data in SAN from IP-based servers Access data in SAN from IP-based servers interworking between NAS and SAN interworking between NAS and SAN RFC 2625 addresses two issues. RFC 2625 addresses two issues. A scheme to encapsulate IP and ARP packets inside the FC frame (as the FC payload) A scheme to encapsulate IP and ARP packets inside the FC frame (as the FC payload) A procedure to resolve the address mapping A procedure to resolve the address mapping

4. 11/05/07 4TDC564-08 IP over FC (RFC 2625) App-1: accessing SAN from IP-based servers SAN Data IP L2 PHY IP L2 PHY IP RFC 2625 FC IP RFC 2625 FC Data SAN FC/IP Gateway FC-based Storage Device

5. 11/05/07 5TDC564-08 IP over FC (RFC 2625) (App-2: interworking between SAN and NAS) SAN Data IP L2 PHY IP L2 PHY IP RFC 2625 FC IP RFC 2625 FC L2 PHY IP Data IP L2 PHY SAN FC/IP Gateway NAS-based Storage Device

6. 11/05/07 6TDC564-08 FC Address Resolution Three steps schemes Three steps schemes ARP - IP to MAC ARP - IP to MAC MAC => World-Wide Port Name (WW_PN) MAC => World-Wide Port Name (WW_PN) FC-ARP - WW_PN => Port_ID FC-ARP - WW_PN => Port_ID Optional Header is used for Network Header by RFC 2625 Optional Header is used for Network Header by RFC 2625 D_NAA (4)Dst. Network Address (60) S_NAA (4)Src. Network Address (60) 16 bytes NAA: Network Address Authority unused (12 bits)IEEE 802.3 MAC Address (48 bits) Format of Network Header Format of Network Address (WW_PN)

7. 11/05/07 7TDC564-08 RFC 2625: IP and ARP over FC (1) ARP Request (broadcast) - Sender HW (MAC) Address = HW(A) - Sender IP Address = IP(A) - Target HW (MAC) Address = ?? - Target IP Address IP(B) A B C D FC switch FC SAN

8. 11/05/07 8TDC564-08 RFC 2625: IP and ARP over FC (2) ARP Reply (unicast): - Sender HW (MAC) Address = HW(B) - Sender IP Address = IP(B) - Target HW (MAC) Address = HW(A) - Target IP Address IP(A) A B C D

9. 11/05/07 9TDC564-08 RFC 2625: IP and ARP over FC (3) FC-ARP Request: - Requester N_Port_Name (WW_PN) - Requester N_Port_ID - Responder N_Port_Name (WW_PN) - Responder N_Port_ID = ?? A B C D unused (12 bits)IEEE 802.3 MAC Address (48 bits) WW_PN

10. 11/05/07 10TDC564-08 RFC 2625: IP and ARP over FC (4) FC-ARP Reply: - Requester N_Port_Name - Requester N_Port_ID - Responder N_Port_Name - Responder N_Port_ID A B C D

11. Advantages of IP for SAN 11/05/07 11TDC564-08

12. IP Network Capabilities 11/05/07 12TDC564-08

13. IP-SAN Protocols 11/05/07 13TDC564-08

14. 11/05/07 14TDC564-08 IP - SAN FC -SANIP Storage Devices

15. 11/05/07 15TDC564-08 IP storage networking – carrying storage traffic over IP IP storage networking – carrying storage traffic over IP Uses TCP, a reliable transport for delivery Uses TCP, a reliable transport for delivery Can be used for local data center and long haul applications Can be used for local data center and long haul applications Two primary IETF protocols/standards: Two primary IETF protocols/standards: iSCSI – Internet SCSI – allows block storage to be accessed over a TCP/IP network as though it were locally attached iSCSI – Internet SCSI – allows block storage to be accessed over a TCP/IP network as though it were locally attached FCIP – Fibre-Channel-over-IP – used to tunnel Fibre Channel frames over TCP/IP connections FCIP – Fibre-Channel-over-IP – used to tunnel Fibre Channel frames over TCP/IP connections IP - Storage Area Network (SAN) IP IP TCP TCP iSCSI iSCSI SCSI SCSI Data Data IP IP TCP TCP FCIP FCIP FC FC SCSI SCSI Data Data

16. 11/05/07 16TDC564-08 Internet SCSI (iSCSI) iSCSI is a proposed industry standard that allows SCSI block I/O protocols (commands, sequences, and attributes) to be sent over a network using the popular TCP/IP protocol. iSCSI is a proposed industry standard that allows SCSI block I/O protocols (commands, sequences, and attributes) to be sent over a network using the popular TCP/IP protocol. A way to access storage across an IP network as though it was locally attached. A way to access storage across an IP network as though it was locally attached. Transports SCSI protocol commands and data across an IP network Transports SCSI protocol commands and data across an IP network Cisco and IBM co-authored original iSCSI protocol draft Cisco and IBM co-authored original iSCSI protocol draft iSCSI Protocol is a standard maintained by the IETF iSCSI Protocol is a standard maintained by the IETF IP Storage (IPS) Working Group IP Storage (IPS) Working Group RFC 3720 RFC 3720

17. iSCSI 11/05/07 17TDC564-08

18. iSCSI over TCP/IP 11/05/07 18TDC564-08 Command SCSI Command DataCommandData TCP HeaderTCP Payload

19. 11/05/07 19TDC564-08 iSCSI Applications Connectivity : iSCSI can be used for DAS or SAN connections. Connectivity : iSCSI can be used for DAS or SAN connections. iSCSI capable devices could be placed on an existing LAN (shared with other applications) in a similar way to NAS devices. iSCSI capable devices could be placed on an existing LAN (shared with other applications) in a similar way to NAS devices. iSCSI capable devices could be attached to a LAN which is dedicated to storage I/O (in other words an IP SAN), or even on a LAN connected to only one processor (like a DAS). iSCSI capable devices could be attached to a LAN which is dedicated to storage I/O (in other words an IP SAN), or even on a LAN connected to only one processor (like a DAS).

20. 11/05/07 20TDC564-08 iSCSI Protocol Stack Data SCSI iSCSI TCP IP Ethernet PHY Driver TCP IP Ethernet PHY Driver Data SCSI iSCSI TCP IP Ethernet PHY Driver Data SCSI iSCSI Adapter Card Host Processing Host Bus Adapter (HBA) Software iSCSI iSCSI TCP Offload Engine (TOE) iSCSI Host Bus Adapter (HBA)

21. iSCSI Performance Consideration 11/05/07 21TDC564-08

22. iSCSI Benefit 11/05/07 22TDC564-08

23. Host Bus Adapter (Fibre Channel) 11/05/07 23TDC564-08 QLogic PCI-EXPRESS 4Gb 1PT Fibre Host Bus Adapter – STD FW PCI-Express 1 Port (Fibre Channel) Data Rate: 4Gb Standard Firmware Price Quote: $951 Source: http://www.cdw.com 11/05/07 23TDC564-08

24. Host Bus Adapter (iSCSI) 11/05/07 24TDC564-08 The QLE4060C iSCSI Host Bus Adapter (HBA) provides connectivity to SANs over Ethernet and TCP/IP network infrastructures for PCI Express bus-based servers. Using a widely deployed and familiar networking technology, iSCSI can lower the total cost of ownership to better manage and support storage networking. Source: http://www.cdw.orghttp://www.cdw.org Price quote: $815 10/100/1000BaseT PCI

25. iSCSI Storage 11/05/07 25TDC564-08 Promise Technology VTrak M-Class iSCSI RAID Price quote: $4,929 Source: //www.cdw.com Network: Gigabit Ethernet (2) Protocol: TCP/IP, iSCSI Capacity: 16-drive bay (500G – 1TB per drive)

26. 11/05/07 26TDC564-08 iSCSI Application Data SCSI iSCSI TCP IP Layer2 PHY iSCSI TCP IP Layer2 PHY IP SCSI FCP FC 0-2 SCSI FCP FC 0-2 Data Cisco iSCSI Configuration Example iSCSI Gateway Server Storage Device

27. Multi-TCP Sessions 11/05/07 27TDC564-08

28. HBA: Multi-Path I/O 11/05/07 28TDC564-08 LUN: Logical [Storage] Units

29. HBA iSCSI vs. Software iSCSI No performance advantage of using iSCSI HBA Throughput Round trip time Significantly higher cost of iSCSI HBA $800 vs. $100 (Server NIC) What are the advantages of iSCSI HBA? Answer: CPU Utilization (off-load) Reference: http://download.intel.com/network/connectivity/products/iSCSIcomp0407.pdf 11/05/07 29TDC564-08

30. Is IP-SAN similar to NAS? What are the advantages, if any, of IP-SAN vs. NAS? 11/05/07 TDC564-08 30

31. Performance Analysis: iSCSI vs. NAS (software based – no HBA) http://technomagesinc.com/pdf/ip_paper.pdf iSCSI NFS 11/05/07 31TDC564-08

32. Sequential Read/Write Tests Conclusion: 1.Comparable performance in character read/write and block write 2.Significant advantage of iSCSI in block read (20-25%) 11/05/07 32TDC564-08

33. Small Files Read/Write 11/05/07 33TDC564-08

34. IOGen Test (Emulation of Database) 11/05/07 34TDC564-08

35. 11/05/07 35TDC564-08 FC-SAN vs. iSCSI Since the iSCSI appliance attaches to the existing Ethernet network, NAS and iSCSI are very similar in network architecture Since the iSCSI appliance attaches to the existing Ethernet network, NAS and iSCSI are very similar in network architecture However, the performance would be significantly different. However, the performance would be significantly different. Both iSCSI and SAN use Block I/O to transport data, whereas NAS uses File I/O. Both iSCSI and SAN use Block I/O to transport data, whereas NAS uses File I/O. SAN offers better performance (c.f. NAS), but is more expensive and requires a higher skill set to implement. iSCSI and NAS offer better pricing and skills may already be in place to implement them. SAN offers better performance (c.f. NAS), but is more expensive and requires a higher skill set to implement. iSCSI and NAS offer better pricing and skills may already be in place to implement them. Both SAN and iSCSI offer the performance benefit of Block I/O. Both SAN and iSCSI offer the performance benefit of Block I/O.

36. Case Study: FC-SAN vs. iSCSI http://www.promise.com/marketing/whitepaper/file/2_MClass- iSCSI%20SCSI%20and%20Fibre-10-07-05.pdf 11/05/07 36TDC564-08

37.  4G bps 11/05/07 37TDC564-08

38. 11/05/07 38TDC564-08 FC over IP The encapsulation of Fibre Channel frames into IP packets and tunneled through an existing TCP/IP network infrastructure, in order to connect remote islands of FC-based SANs FC SANIPFC SAN IP IP TCP TCP FCIP FCIP FC FC SCSI SCSI Data Data

39. FC Frame Encapsulation (RFC 3643) Describes the common Fibre Channel (FC) frame encapsulation format and a procedure for the measurement and calculation of frame transit time through the IP network Encapsulation Header SOFFC Frame BodyEOF 11/05/07 39TDC564-08

40. FC Encapsulation Header Q: Why is time stamp needed in the FC encapsulation header? 11/05/07 40TDC564-08

41. FC Encapsulation 11/05/07 41TDC564-08

42. FCIP (RFC 3821) Fibre Channel Over TCP/IP (FCIP) describes mechanisms that allow the interconnection of islands of Fibre Channel storage area networks over IP-based networks to form a unified storage area network in a single Fibre Channel fabric 11/05/07 42TDC564-08

43. FCIP Protocol Stacks 11/05/07 43TDC564-08

44. 11/05/07 44TDC564-08 FC over IP IPSAN Data SCSI FC 0-2 FCIP TCP IP L2 PHY IP L2 PHY TCP IP L2 PHY FC 0-2 Data SCSI FC 0-2 FCP SAN an IP tunnel for FC-based SAN Application: interconnect SAN over IP-WAN. FCIPFCP

45. FCIP Configuration Example 11/05/07 45TDC564-08

46. 11/05/07 46TDC564-08 Internet Fiber Channel Protocol (iFCP) iFCP – RFC 4172 (Sept. 2005) iFCP – RFC 4172 (Sept. 2005)RFC 4172 RFC 4172 iFCP is a protocol that uses the FCP over IP networks to interconnect FC-based storage devices. iFCP is a protocol that uses the FCP over IP networks to interconnect FC-based storage devices. It is considered a gateway protocol, while FCIP is considered a tunnel protocol. It is considered a gateway protocol, while FCIP is considered a tunnel protocol.

47. iFCP (RFC 4172) iFCP specifies an architecture and a gateway-to-gateway protocol for the implementation of fibre channel fabric functionality over an IP network. This functionality is provided through TCP protocols for fibre channel frame transport and the distributed fabric services specified by the fibre channel standards. The architecture enables internetworking of fibre channel devices through gateway-accessed regions with the fault isolation properties of autonomous systems and the scalability of the IP network 11/05/07 47TDC564-08

48. iFCP Network Example 11/05/07 48TDC564-08

49. 11/05/07 49TDC564-08 iFCP (RFC 4172) IP Data SCSI FC 0-2 FCP TCP IP L2 PHY IP L2 PHY TCP IP L2 PHY FC 0-2 Data SCSI FC 0-2 FCP Gateway approach Application: interconnect SAN over IP-WAN. FCP iFCP iFCP GW F-port 11/05/07 49TDC564-08

50. iFCP 11/05/07 50TDC564-08

51. FCIP vs. iFCP 11/05/07 51TDC564-08 http://whatis.techtarget.com/magPrintFriendly/0,293813,sid35_gci1096482,00.html

52. FCIP vs. iFCP McDATA 11/05/07 52TDC564-08

53. 11/05/07 53TDC564-08 SAN Performance Comparison http://www.netapp.com/library/tr/3423.pdf

54. 11/05/07 54TDC564-08 Summary Needs for large storage – continual growth Needs for large storage – continual growth 10 9 (G) => 10 12 (T) => 10 15 (P) => 10 (E) ……. 10 9 (G) => 10 12 (T) => 10 15 (P) => 10 18 (E) ……. From dedicated solution to network-based solution From dedicated solution to network-based solution DAS => NAS => SAN => IP-SAN DAS => NAS => SAN => IP-SAN Convergence of SAN and IP-LAN/WAN Convergence of SAN and IP-LAN/WAN It is an IP world! It is an IP world! SCSI is the protocol for block data transmission SCSI is the protocol for block data transmission SCSI over FC SCSI over FC SCSI over IP (iSCSI) SCSI over IP (iSCSI) FC and IP interworking protocols FC and IP interworking protocols IP over FC IP over FC FC over IP (FCIP) and iFCP FC over IP (FCIP) and iFCP