Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 © 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Flow Control in Fibre Channel (BB_Credits) Lincoln Dale

Similar presentations


Presentation on theme: "1 © 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Flow Control in Fibre Channel (BB_Credits) Lincoln Dale"— Presentation transcript:

1 1 © 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Flow Control in Fibre Channel (BB_Credits) Lincoln Dale

2 222 © 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential Topology Hosts FC StorageSAN Switches Hosts connected to a single switch Storage connected to a second switch Switches connected to each other

3 333 © 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential Buffer Credits (BB_Credits) at FC Class 3 FC BB_Credits are the admission control mechanism in FC to ensure that FC switches dont run out of buffers (FC Switches cannot drop frames) For Devices operating at FC Class 3 (most devices), Buffer Credits are negotiated on a per-hop basis at device FLOGI. BB_Credits are the only flow-control mechanism for FC Class 3. available BB_Credits available BB_Credits 12 available BB_Credits Frame towards Disk shelf Return BB_Credit token available BB_Credits available BB_Credits

4 444 © 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential Buffer Credits: The Good and The Bad FC Since BB_Credits are per-hop for FC Class 3: Any speed mismatch (1G/2G) between senders & receivers and/or Insufficient numbers of BB_Credits in a device can result in Congestion due to Head of Line (HoL) Blocking This Congestion can impact other ports & devices in a SAN Disk shelf capable of sustaining 200MB/sec Tape capable of sustaining 15MB/sec

5 555 © 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential Buffer Credits (BB_Credits): Utopia FC Disk shelf capable of sustaining 200MB/sec Tape capable of sustaining 15MB/sec Frame towards Tape Return BB_Credit token Frame towards Disk shelf / Available BB_Credits Frames to switchA Frames across switchA Frames to device /16 Available BB_Credits 16/1613/1612/1614/1615/1610/169/16 11/16 13/1612/1614/1615/16 11/16 8/167/1616/1613/1612/1614/1615/16 11/16 In an ideal FC network, there is no blocking in any device connected to the fabric. (all devices can process frames at the same rate and negotiate equal levels of BB_Credits) DO NO DELETE THIS STUFF OFF THE LEFT OF THE SLIDE. IT IS USED FOR THE ANIMATIONS!!

6 666 © 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential Buffer Credits (BB_Credits): Real world impact on performance FC Disk shelf capable of sustaining 200MB/sec Tape capable of sustaining 15MB/sec Frame towards Tape Return BB_Credit token Frame towards Disk shelf / Available BB_Credits In reality, not all devices can process frames at the same rate and different devices negotiate different BB_Credits This means that a single device is capable of causing HOL blocking across a FC switch or even across a SAN! Frames to switchA Frames across switchA Frames to device /16 Available BB_Credits 16/1613/1612/1614/1615/1610/169/16 11/16 13/1614/1615/168/167/16 DO NO DELETE THIS STUFF OFF THE LEFT OF THE SLIDE. IT IS USED FOR THE ANIMATIONS!! 1/22/2 0/2 FC Frame blocks at head of Ingress port due to 0 available BB_Credits on egress port Frames to other devices/ports backlog behind blocked Frame at Head-of-Queue 1 Frame Blocked2 Frames Blocked3 Frames Blocked4 Frames Blocked5 Frames Blocked6 Frames Blocked7 Frames Blocked 6/165/16 8 Frames Blocked Uncongested Fabric While this example isnt perhaps the most realistic case (flooding FC frames at a device with only 2 BB_Credits), it does show the ramifications of Fibre Channel standards mandating frames must not be dropped.

7 777 © 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential Buffer Credits (BB_Credit) impact on performance While Head-of-Line Blocking is a transitory event (until some BB_Credits are returned on the blocked port), performance can be adversely affected across an entire multi-switch FC Fabric by a single blocking port The Cisco MDS 9000 Series has multiple features to help alleviate the problem: Virtual Output Queueing (VoQ) on all ports Deep Buffers – 255 BB_Credits per port Fibre Channel Congestion-Control (FCC) – Detects congested ports and throttles the port causing the congestion at its origin

8 888 © 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential Virtual Output Queueing (VoQ) In a typical FC switch, each port has an input queue for frames arriving on a port All frames, regardless of destination, are queued in the order theyre received This can potentially cause Head-of- Line blocking should one of the destination ports become congested Frames will begin to backup in the queue and the sourcing devices will eventually have to stop transmitting data. Switch without VoQ Frame to Port 5 Frame to Port 6 Frame to Port 4 Frame to Port 6 Frame to Port 4 Input Queue at Port 1 Top of Queue

9 999 © 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential Virtual Output Queueing (VoQ) Virtual Output Queuing Model Frame to Port 5 Frame to Port 6 Frame to Port 4 Frame to Port 6 Frame to Port 4 Input Queue at Port 1 Top of Virtual Queue Input Queue at Port 1 Top of Virtual Queue The MDS 9000 series utilizes Virtual Output Queues (VoQ) to eliminate Head-of-Line blocking. Each Destination Port is given its own Virtual Queue on the Input Port. Congestion on any Destination Port has no effect on traffic destined for other ports. Every Source Port on the MDS 9000 series has four Virtual Output Queues for every Destination Port – to cover 4 QoS levels per-port across a chassis

10 10 © 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential Deep Buffers – 255 Buffer-to-Buffer Credits (BB_Credits) per port Typical rule-of-thumb is: 2 BB_Credits are required for every 1gbit/s 4 BB_Credits are required for every 2gbit/s Other Fibre Channel switches in the marketplace support up to 60 Buffer-to-Buffer Credits (BB_Credits) Performance is limited to 2gbit/s maximum over <15km Performance is limited to 1gbit/s maximum over <30km The MDS 9000 Series has up to 255 Buffer-to- Buffer Credits (BB_Credits) per port Wire-rate (2gbit/s) achievable for 64+km 1gbit/s achievable for 127+km

11 11 © 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential 3. Switch A rate-limits incoming traffic Fibre Channel Congestion Control (FCC) FC Disk shelf capable of sustaining 200MB/sec Tape capable of sustaining 15MB/sec Consider the following topology: Host A is issuing write operations to 100MB/sec Host B is issuing writes operations to 50MB/sec A B Congestion on Switch Bs port connecting the tape can cause a performance degredation for Host A writing to Disk Congestion FCC mitigates the congestion by throttling the traffic at the originating port 1. Switch B detects congestion on port connecting to Tape 2. Switch B signals Switch A to quench the initiator Limit HostB Tape to 15MB/sec Switch A Switch B Port Rate-Limited on VoQ

12 12 © 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential 12 © 2001, Cisco Systems, Inc. All rights reserved.


Download ppt "1 © 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Flow Control in Fibre Channel (BB_Credits) Lincoln Dale"

Similar presentations


Ads by Google