1. Flow Control in Fibre Channel (BB_Credits)
All presenters should use this section (Notes Pages) to provide additional information and details about the bullet points above. These notes will help remind students of key points when reviewing materials in their office. It will also help students conduct more standardized and effective train-the-trainer sessions when they return to their offices.
Lincoln Dale

2. Topology Hosts connected to a single switch
Storage connected to a second switch
Switches connected to each other
Hosts
SAN Switches
Storage FC

3. Buffer Credits (BB_Credits) at FC Class 3
BB_Credits are the “admission control” mechanism in FC to ensure that FC switches don’t run out of buffers (FC Switches cannot drop frames)
For Devices operating at FC Class 3 (most devices), Buffer Credits are negotiated on a per-hop basis at device FLOGI.
BB_Credits are the only flow-control mechanism for FC Class 3. 11 12
available
BB_Credits 15 16 15 16
available BB_Credits
available BB_Credits FC 12
available BB_Credits 16
Frame towards Disk shelf
available BB_Credits
Return BB_Credit token

4. Buffer Credits: The Good and The Bad
Since BB_Credits are per-hop for FC Class 3:
Any speed mismatch (1G/2G) between senders & receivers
and/or
Insufficient numbers of BB_Credits in a device
can result in Congestion due to Head of Line (HoL) Blocking
This Congestion can impact other ports & devices in a SAN FC
Disk shelf capable of sustaining 200MB/sec
Tape capable of sustaining 15MB/sec

5. Buffer Credits (BB_Credits): Utopia
DO NO DELETE THIS STUFF OFF THE LEFT OF THE SLIDE. IT IS USED FOR THE ANIMATIONS!!
Buffer Credits (BB_Credits): Utopia
Frames to switchA 1 1 1 1 2 2 2 2 3 3 3 3
In an ideal FC network, there is no blocking in any device connected to the fabric. (all devices can process frames at the same rate and negotiate equal levels of BB_Credits) 4 4 4 4 5 5 5 5 6 6 6 6 FC 7 7 7 7 8 8 8 8
Disk shelf capable of sustaining 200MB/sec
Frames to device
Frames across switchA
12/16
16/16
13/16
14/16
15/16
11/16 1 1 1 1 1 1
Available BB_Credits 2 2 2 2 2 2
13/16
16/16
10/16
11/16
8/16
7/16
15/16
12/16
14/16
9/16
Available BB_Credits 3 3 3 3 3 3 4 4 4 4 4 4 5 5 5 5 5 5
11/16
14/16
15/16
13/16
12/16
16/16 6 6 6 6 6 6
Available BB_Credits 7 7 7 7 7 7
Tape capable of sustaining 15MB/sec 8 8 8 8 8 8
Frame towards Disk shelf
Frame towards Tape /
Return BB_Credit token

6. Buffer Credits (BB_Credits): Real world impact on performance
DO NO DELETE THIS STUFF OFF THE LEFT OF THE SLIDE. IT IS USED FOR THE ANIMATIONS!!
Buffer Credits (BB_Credits): Real world impact on performance
Frames to switchA 1 1 1 1 2 2 2 2
In reality, not all devices can process frames at the same rate and different devices negotiate different BB_Credits
This means that a single device is capable of causing HOL blocking across a FC switch or even across a SAN! 3 3 3 3
Frames to other devices/ports backlog behind blocked Frame at Head-of-Queue 4 4 4 4 5 5 5 5 6 6 6 6 FC 7 7 7 7
7 Frames Blocked
2 Frames Blocked
Uncongested Fabric
8 Frames Blocked
1 Frame Blocked
3 Frames Blocked
6 Frames Blocked
4 Frames Blocked
5 Frames Blocked 8 8 8 8
Disk shelf capable of sustaining 200MB/sec
Frames to device
Frames across switchA
15/16
14/16
13/16
16/16 1 1 1 1 1 1
Available BB_Credits 2 2 2 2 2 2
11/16
12/16
16/16
14/16
13/16
7/16
15/16
5/16
8/16
9/16
6/16
10/16
While this example isn’t perhaps the most realistic case (flooding FC frames at a device with only 2 BB_Credits), it does show the ramifications of Fibre Channel standards mandating “frames must not be dropped”.
Available BB_Credits 3 3 3 3 3 3
FC Frame blocks at head of Ingress port due
to 0 available BB_Credits on egress port 4 4 4 4 4 4 5 5 5 5 5 5
2/2
1/2
0/2 6 6 6 6 6 6
Available BB_Credits 7 7 7 7 7 7
Tape capable of sustaining 15MB/sec 8 8 8 8 8 8
Frame towards Disk shelf
Frame towards Tape /
Return BB_Credit token

7. Buffer Credits (BB_Credit) impact on performance
While Head-of-Line Blocking is a transitory event (until some BB_Credits are returned on the blocked port), performance can be adversely affected across an entire multi-switch FC Fabric by a single blocking port
The Cisco MDS 9000 Series has multiple features to help alleviate the problem:
Virtual Output Queueing (VoQ) on all ports
Deep Buffers – 255 BB_Credits per port
Fibre Channel Congestion-Control (FCC) – Detects congested ports and throttles the port causing the congestion at its origin

8. Virtual Output Queueing (VoQ)
In a typical FC switch, each port has an input queue for frames arriving on a port
All frames, regardless of destination, are queued in the order they’re received
This can potentially cause Head-of-Line blocking should one of the destination ports become congested
Frames will begin to backup in the queue and the sourcing devices will eventually have to stop transmitting data.
Switch without VoQ
Frame to Port 5
Frame to Port 6
Frame to Port 4
Input Queue at Port 1
Top of Queue

9. Virtual Output Queueing (VoQ)
Virtual Output Queuing Model
Frame to Port 5
Frame to Port 6
Frame to Port 4
Input Queue at Port 1
Top of Virtual Queue
The MDS 9000 series utilizes Virtual Output Queues (VoQ) to eliminate Head-of-Line blocking.
Each Destination Port is given its own Virtual Queue on the Input Port.
Congestion on any Destination Port has no effect on traffic destined for other ports.
Every Source Port on the MDS 9000 series has four Virtual Output Queues for every Destination Port – to cover 4 QoS levels per-port across a chassis

10. Deep Buffers – 255 Buffer-to-Buffer Credits (BB_Credits) per port
Typical rule-of-thumb is:
2 BB_Credits are required for every 1gbit/s
4 BB_Credits are required for every 2gbit/s
Other Fibre Channel switches in the marketplace support up to 60 Buffer-to-Buffer Credits (BB_Credits)
Performance is limited to 2gbit/s maximum over <15km
Performance is limited to 1gbit/s maximum over <30km
The MDS 9000 Series has up to 255 Buffer-to-Buffer Credits (BB_Credits) per port
Wire-rate (2gbit/s) achievable for 64+km
1gbit/s achievable for 127+km

11. Fibre Channel Congestion Control (FCC)
Consider the following topology:
Host A is issuing write operations to 100MB/sec
Host B is issuing writes operations to 50MB/sec
Congestion on Switch B’s port connecting the tape can cause a performance degredation for Host A writing to Disk
FCC mitigates the congestion by throttling the traffic at the originating port
1. Switch B detects congestion on port connecting to Tape FC
Disk shelf capable of sustaining 200MB/sec
2. Switch B signals Switch A to quench the initiator A
3. Switch A rate-limits incoming traffic
Switch A
Switch B
Congestion
Congestion
Tape capable of sustaining 15MB/sec
Port Rate-Limited on VoQ
Limit HostBTape to 15MB/sec B

12. 12
© 2001, Cisco Systems, Inc. All rights reserved.