1. FCoE Overview IEEE CommSoc/SP Chapter Austin, Texas, May 21 2009 Tony Hurson tony.hurson@ieee.org

2. Networked Storage History

3. SANs Latency, Throughput Requirement

4. SCSI Read, Write over FC

5. FC Fabric Port Terminology

6. FC Initialization Flow Host (Physical Machine) initiates Fabric Login with nearest switch – FLOGIN – receives a VN_Port ID in return (Additional entities on same Host – e.g. Virtual Machines may also log into fabric using NPIV FDISC, each getting a unique VN_Port ID). (Host may use well-known FC servers to discover the VN_Port IDs of its Targets and their LUNs). Host initiates Port Login with chosen Targets

7. FC Routing

8. Ethernet Routing Dynamic Scheme: Source Learning If unicast DstMAC is not in lookup table, flood frame to all ports except its source port. Note source port of SrcMAC in lookup table, if not already present Age/invalidate lookup entries Similar flooding behavior for multicast Precludes loops in fabric

9. FC Frame Format

10. FC Classes of Service Class 1 – dedicated connection – obselete; unsupported in FCoE Class 2 – acknowledged, with ACKs from remote peer. Supported in FCoE, though some link-layer features (F_RJT, F_BSY) irrelevant. Useful for large Sequences and sequential devices. Class 3 – unacknowledged, FCoE-supported. Most prevalent for disk applications. Error recovery punted to higher (FCP+) levels. Class F – inter-switch link only. FCoE-supported

11. Protocol Stack History and Comparison

12. iSCSI SAN Pros and Cons For: Runs over existing, ubiquitous Ethernet, TCP/IP fabrics. Internetworking built in. Lots of fabric management tools (ready for enterprise storage?). Against TCP Slow Start impacts I/O latency, throughput (but newer TCPs are tunable) Lossy fabrics impact I/O latency through re- transmission and complicate receive endpoint data placement. Bridging to legacy FC SANs slow/expensive because of TCP termination overhead.

13. Lossless Ethernet – via PAUSE

14. FCoE Early Deployment Example

15. FCoE Frame Format

16. FCoE Endpoint Model

17. FCoE Switch Functional Model

18. Converged Ethernet AKA Data Center Bridging (DCB). Run up to four major traffic classes on single 10 GbE fabric. In order of market prevalence: Networking (TCP/IP, lossy). Block Storage (lossless FCoE, or lossless/lossy iSCSI). Management (heartbeat traffic, low bandwidth, but must get through). Inter-Process Communication (clustered computing: high bandwidth, low latency, lossless preferred).

19. Groundwork for DCB IEEE 802.1Qaz – ETS & DCBX – bandwidth allocation to major traffic classes (Priority Groups); plus DCB management protocol. IEEE 802.1Qbb – Priority PAUSE. Selectively PAUSE traffic on link by Priority Group. IEEE 802.1Qau – Dynamic Congestion Notification.

20. IEEE 802.1Qaz Enhanced Transmission Selection Support at least 3 Priority Groups/traffic classes PGs identified by Priority field of existing 802.1Q VLAN Tag Configured Bandwidth per PG has 1% resolution PG15 has limitless bandwidth (use sparingly!, for Management) Work Conservation – if the wires free, use it.

21. ETS Configuration Example PG0 (Storage): 40% of port b/w PG1 (Networking): 20% of port b/w PG2 (IPC): 40% of port b/w PG15 (mgmt): limitless If a PG underutilizes, others can fill the space. Typical implementation: DWRR.

22. IEEE 802.1Qbb Priority PAUSE

23. IEEE 802.1Qau Dynamic Congestion Control Background Lossless fabrics are prone to congestion spreading (congestion trees). Ethernet-FC gateways with their different port speeds (10 GbE; 8 Gbps) are natural bottlenecks. ETS Work Conservation model adds fuel to fire. Solution: switches/endpoints notify traffic sources of incipient congestion, via feedback messages; sources reduce rates accordingly.

24. Congestion Notification in Action

25. Congestion Control at Endpoint Transmit

26. FCoE Summary Presents new, but very familiar, PHY and Link Layers for FC. Core switching discipline remains FC-SW-5. Higher FC layers almost completely unchanged (thats the legacy value!) Biggest Ethernet-level requirement: lossless fabric. Part of Converged Ethernet initiative – lots of ancillary activity at IEEE.

27. Further Reading FCoE: www.t11.orgwww.t11.org IEEE 802.1Q(az|au|bb): www.ieee.org www.ieee.org Thank you! Questions?