1. Design of a Diversified Router: TCAM Usage
John DeHart

2. TCAM Issues Number of Databases to use Key size Result Type
16 available
Key size
Can be different for each Database.
Result Type
Absolute Index
Database Relative Index
Memory Pointer
TCAM Associated Data of width 32, 64 or 128 bits
Memory Constraints:
TCAM Associated Data
512K x 36 bit ZBT SRAM
Supports 128K 128-bit Results
If used for Rx and Tx then 64K in each direction
IXP QDR SRAM Channel
2 x 2Mx18 (effective 4M x 18b)
4 times as much as the TCAM ZBT SRAM.
Supports 512K 128-bit Results
If used for Rx and Tx then 256K in each direction

3. TCAM Issues Types of Lookups supported: Lookup (Direct)
Multi-Hit Lookup (Direct)
Simultaneous Multi-Database Lookup (Direct)
Multi-Database Lookup (Indirect)
Simultaneous Multi-Database Lookup (Indirect)
Re-Issue Multi-Database Lookup (Indirect)

4. TCAM Issues Performance Impacts IXP/TCAM Interface
16 bits wide
200 MHz QDR
Effective 32bits per clock tick
Example: 128b results from ZBT SRAM via TCAM: 4 ticks  max of 50M results/sec
Key Size (200MHz, 16 bit Interface for commands)
Rates are max PER LA-1 Interface, up to CAM Core max rate
32b (1W): 50M Lookups/sec (CAM Core constraint)
36b (2W): 50M Lookups/sec (CAM Core constraint)
64b (2W):100M Lookups/sec (Interface constraint)
72b (3W): 67M Lookups/sec (Interface constraint)
128b (4W): 50M Lookups/sec (Interface constraint)

5. TCAM Issues Performance Impacts (continued) Result Type
Index or Pointer Types:
Key Size: Max CAM Core lookup rate
36b: 50M/sec
72b: 100M/sec
144b: 100M/sec
288b: 50M/sec
576b: 25M/sec
Associated Data: CAM Core rates:
These rates are Total across both LA-1 Interfaces
Key Size: (32b Result Rate, 64b Result Rate, 128b Result Rate)
36b: , 50, 25
72b: 100, 50, 25
144b: 100, 50, 25

6. TCAM Performace Tables

7. TCAM Performace Graphs

8. TCAM Performace Graphs

9. LC: Notes on TCAM Lookups
The following slides show a way to use the TCAM for the lookups.
Slight adjustments might be desirable depending on:
Ease of doing operations on non-byte length bit fields
What we learn about methods for using the TCAM.
Field and Identifier sizes:
MN id: 32 bits
MI id: 16 bits (64K Meta Interfaces per Meta Router)
MLI: bits (64K Meta Links per Substrate Link)
Port: bits (256 Physical Interfaces per Line Card)
QID: bits (128K Queues per Queue Manager)
QM ID: 3 bits (8 Queue Managers per LC or PE.)
We probably can only support 4 QMs (2 bits)
(64 Q-Array Entries) / (16 CAM entries)  4 QMs per SRAM Controller.

10. LC: Notes on TCAM Lookups
Lookup Key size options:
32/36, 64/72, 128/144, 256/288, 512/576 (all in bits)
Lookup Result options:
Absolute Index: relative to beginning of TCAM array
Database Relative Index: relative to beginning of selected database
Memory Pointer Index: points to SRAM location of result data
Associated Data: 32, 64 or 128 bits of data associated with the lookup result.
Associated Data is stored in ZBT SRAM attached to TCAM.
TCAM Databases
How many to use?
1 for TX and 1 for RX?
1 for TX and 1 for each of the SL Types on Rx (5 types)?
Other…

11. LC: TCAM Lookup Keys on RX
Each SL Type will use a different Database on RX
Thus we do not need an SL Type field in the Key
SL Type ID will be used to identify which database to use
P2P-DC(24b):
SL Type ID: 0x0
Key Fields: Port(8)/MLI(16)
P2P-Tunnel(IPv4)(72b):
SL Type ID: 0x1
Key Fields: Port(8)/EtherType(16)/IPSrc(32)/MLI(16)
P2P-VLAN0(72b):
SL Type ID: 0x2
Key Fields: Port(8)/EthSAddr(48)/MLI(16)
MA(24b):
SL Type ID: 0x3
Legacy(24b):
SL Type ID: 0x4
Key Fields: Port(8)/EtherType(16)
Key Fields:
Port(8b): Physical Interface number
MLI(16b): Meta Link Identifier
Ethertype(16b): Ethernet Header Type field
IPSrc(32b): IP Source Address
EthSAddr(48b): Ethernet Header Source Address

12. LC: TCAM Lookup Keys on RX
P2P-DC
Port(8b)
MLI(16b)
24 bits
IPv4 Tunnel
Port (8b)
EtherType (16b)
0x0800
IP SAddr (32b)
MLI (16b)
72 bits MA
Port (8b)
Ethernet SAddr (48b)
MLI (16b)
72 bits
P2P-VLAN0
Port(8b)
MLI(16b)
24 bits
Legacy
Port (8b)
EtherType (16b)
0x0800
24 bits
DstAddr (6B)
Blue Shading: Determine SL Type
Black Outline: Key Fields from pkt
DstAddr (6B)
SrcAddr (6B)
SrcAddr (6B)
Type=802.1Q (2B)
Type=802.1Q (2B)
TCI ≠ VLAN0 (2B)
TCI ≠ VLAN0 (2B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
Type=IP (2B)
Multi-Access
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI≠VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI=VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
P2P-VLAN0
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
ID/Flags/FragOff (4B)
TTL (1B)
TTL (1B)
Protocol=Substrate (1B)
Protocol (1B)
Hdr Cksum (2B)
Hdr Cksum (2B)
Src Addr (4B)
Src Addr (4B)
Dst Addr (4B)
Dst Addr (4B)
MLI (2B)
IP Payload
LEN (2B)
Meta Frame
PAD (nB)
PAD (nB)
CRC (4B)
CRC (4B)
P2P-DC
Configured
P2P-Tunnel
Legacy

13. LC Packet RX: All SL Types
Type=IP (2B)
MLI (2B)
LEN (2B)
PAD (nB)
CRC (4B)
Meta Frame
Dst Addr (4B)
Src Addr (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol=Substrate (1B)
Hdr Cksum (2B)
Type=802.1Q (2B)
TCI ≠ VLAN0 (2B)
DstAddr (6B)
SrcAddr (6B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI=VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI≠VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
P2P-DC
P2P-Tunnel
P2P-VLAN0
Multi-Access

14. LC: TCAM Lookup Results on RX
Fields (68b/128b):
VLAN (16b)
We could probably drop this to 12b since our switch is supposed to only support 4K VLANs but we might want to leave this open to switches supporting larger numbers of VLANs.
MI (16b)
Blade Eth Hdr (8b)
Only needs to have the DAddr.
We can control the MAC Addresses of the Blades, so lets say that 40 of the 48 bits are fixed and 8 bits are assigned and stored in the Lookup Result. Even 8 bits is overkill but it leaves us some flexibility.
SAddr can be configured and constant per LC
First EtherType can be constant: 802.1Q
Second EtherType can be constant: Substrate
QID (20b)
MnFlags(8b):
NhAddr(60b): If needed

15. LC: TCAM Lookup Results on RX
Can we say there will be no Pass Through Meta Links where one side will be on a Multi Access and hence might need a NhAddr field?
If so then we can drop the MnFlags and NhAddr fields from result
Result size then becomes: 60b
Pass Through Meta Link Fields:
MnFlags(8b)
NhAddr(nB)
We have 60 bits left over that could be used for NhAddr
If we need more:
Do a second lookup with the following fields to retrieve the Next Hop bits from another Database for NH Bits?
Port (8b)
VLAN (16b)
MI (16b)

16. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
The Lookup Result for TX will consist of several parts:
Lookup Result
Constant fields
Calculated fields
Fields that can be stored in Local Memory
Some of these are common across all SL Types
Other fields are specific to each SL Type
Common across all SL Types (96b):
From Result (48b)
SL Type(4b)
Port(8b)
MLI(16b)
QID (20b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
P2P-DC Hdr:
P2P-MA Hdr:
P2P-VLAN0 Hdr
P2P-Tunnel Hdr for IPv4 Tunnel

17. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
Common across all SL Types (96b):
From Result (48b)
SL Type(4b)
Port(8b)
MLI(16b)
QID (20b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
P2P-DC Hdr (64b)
Constant (16b)
EtherType (16b) = Substrate
Calculated (0b)
Eth DA (48b)
Lookup Result Total (Common From Result + Specific From Result): 96 bits
Total (Common + Specific) : 160 bits

18. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
Common across all SL Types (96b):
From Result (48b)
SL Type(4b)
Port(8b)
MLI(16b)
QID (20b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
P2P-MA Hdr (64b) :
Constant (16b)
EtherType (16b) = Substrate
Calculated (0b)
Eth DA (48b)
Lookup Result Total (Common From Result + Specific From Result): 96 bits
Total (Common + Specific) : 164 bits

19. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
Common across all SL Types (96b):
From Result (48b)
SL Type(4b)
Port(8b)
MLI(16b)
QID (20b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
P2P-VLAN0 Hdr (80b):
Constant (16b)
EtherType1 (16b) = 802.1Q
EtherType2 (16b) = Substrate
Calculated (0b)
From Result (64b)
Eth DA (48b)
TCI (16b)
Lookup Result Total (Common From Result + Specific From Result): 112 bits
Total (Common + Specific) : 176 bits

20. LC: TCAM Lookups on TX Result (continued)
Common across all SL Types (96b):
From Result (48b)
SL Type(4b)
Port(8b)
MLI(16b)
QID (20b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
P2P-Tunnel Hdr for IPv4 Tunnel (224b):
Constant (64b)
Eth Hdr EtherType (16b) = 0x0800
IPHdr Version(4b)/HLen(4b)/Tos(8b) (16b): All can be constant?
IP Hdr Flags(3b)/FragOff(13b) (16b) : what is our stance on Fragments? If never used, these are constants, if it is possible we will have to use them, then this has to be calculated. Either way, shouldn’t be in Result
IP Hdr TTL (8b): Constant
IP Hdr Proto (8b) = Substrate
Calculated (48b)
IP Hdr Len(16b) : needs to be calculated for each packet sent, so shouldn’t be in Result.
IP Hdr ID(16b): should be unique for each packet sent, so shouldn’t be in Result.
IP Hdr Checksum (16b): Needs to be calculated, so shouldn’t be in Result.
Local Memory (32b)
IP Hdr Src Addr (32b) : tied to port
From Result (80b)
Eth Hdr DA (48b)
IP Hdr Dst Addr (32b)
Lookup Result Total (Common From Result + Specific From Result): 128 bits
Total (Common + Specific) : 320 bits

21. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
Common across all SL Types (96b):
From Result (48b)
SL Type(4b)
Port(8b)
MLI(16b)
Ignored for Legacy Traffic
QID (20b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
Legacy (IPv4) with VLAN Hdr (96b):
Constant (16b)
EtherType1 (16b) = 802.1Q
Calculated (0b)
ARP Lookup on NhAddr (48b)
Eth DA (48b)
From Result (32b)
EtherType2 (16b) = IPv4
TCI (16b)
Lookup Result Total (Common From Result + Specific From Result): 80 bits
Total (Common + Specific) : 192 bits

22. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
Common across all SL Types (96b):
From Result (48b)
SL Type(4b)
Port(8b)
MLI(16b)
Ignored for Legacy Traffic
QID (20b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
Legacy (IPv4) without VLAN Hdr (64b):
Constant (0b)
Calculated (0b)
ARP Lookup on NhAddr (Is ARP cache another database in TCAM?) (48b)
Eth DA (48b)
From Result (16b)
EtherType (16b) = IPv4
Lookup Result Total (Common From Result + Specific From Result): 64 bits
Total (Common + Specific) : 160 bits

23. SUMMARY: LC: TCAM LookupsDC
Tunnel
VLAN0 MA
Legacy w/ vlan
Legacy w/o vlan RX
Key 24 72
same as
Result
68/128 TX 32 96
128
112 80 64
Rx Key: 72 bits
Rx Result Size: 68/128 bits
Depends on whether we need to support MA on other end of pass through MetaLink and on what size the NhAddr needs to be.
If we can limit NhAddr to 60 bits, then it fits in 128 bits.
Otherwise we will have to do something else…
Tx Key Size: 32 bits
Tx Result Size: 128 bits
We need to watch out for the Tx Result for Tunnels. If we introduce anything else we want in there then we go beyond the 128 bits supportable through the TCAM’s Associated memory.

24. Databases on LC Rx: TX: 0000: DC 0001: IPv4 Tunnel 0010: VLAN0
0011: MA
0100: Legacy (non-substrate) with or without VLAN
TX:
1000: all TX lookups use one database

25. Extra
The next set of slides are for templates or extra information if needed

26. Text Slide Template

27. Image Slide Template

28. OLD
The rest of these are old slides that should be deleted at some point.

29. LC: TCAM Lookup Keys on RX
P2P-DC(28b): SL_Type(4)/Port(8)/MLI(16)
P2P-Tunnel(IPv4)(76b): SL_Type(4)/Port(8)/EtherType(16)/IPSrc(32)/MLI(16)
P2P-VLAN0(76b): SL_Type(4)/Port(8)/EthSAddr(48)/MLI(16)
MA(28b): SL_Type(4)/Port(8)/MLI(16)
Legacy(28b): SL_Type(4)/Port(8)/EType(16)
Fields:
SL_Type (4b): Substrate Link Type
0000: DC
0001: IPv4 Tunnel
0010: VLAN0
0011: MA
0100: Legacy (non-substrate) without VLAN
0101: Legacy (non-substrate) with VLAN
Port(8b): Physical Interface number
MLI(16b): Meta Link Identifier
Ethertype(16b): Ethernet Header Type field
IPSrc(32b): IP Source Address
EthSAddr(48b): Ethernet Header Source Address

30. Databases on LC
Rx:
0000: DC
0001: IPv4 Tunnel
0010: VLAN0
0011: MA
0100: Legacy (non-substrate) without VLAN
0101: Legacy (non-substrate) with VLAN
TX:
1000: all TX lookups use one database
Do we really need to use the TCAM for TX Lookups?
VLAN(16b)/TxMI(16b) is the lookup key.
Really only 4K VLANs support: 12 bits
How many VLANs (i.e. MRs) should be supported per LC?
How many MIs should be supported per MR?
8MB per SRAM/(16B per Lookup)  512K Entries

31. LC: TCAM Lookup Results on RX
Standard Fields (64b/128b):
Type (4b): Probably don’t need this since there is a size in MnFlags.
0000: Default, not Pass Through, ignore MnFlags
1000: Pass Through with no extra lookup needed for NhAddr(nB), MnFlags(8b) should be 0x00
1001: Pass Through with extra lookup needed for NhAddr(nB)
VLAN (16b)
MI (16b)
Blade Eth Hdr (8b)
Only needs to have the DAddr.
We can control the MAC Addresses of the Blades, so lets say that 40 of the 48 bits are fixed and 8 bits are assigned and stored in the Lookup Result. Even 8 bits is overkill but it leaves us some flexibility.
SAddr can be configured and constant per LC
First EtherType can be constant: 802.1Q
Second EtherType can be constant: Substrate
QID (20b)
MnFlags(8b): If needed
NhAddr(56b): If needed

32. SUMMARY: LC: TCAM LookupsDC
Tunnel
VLAN0 MA
Legacy w/ vlan
Legacy w/o vlan RX
Key 28 76
Result
64/128 TX 32 96
128
112 80 64
Result in Loc. Mem 48
Combined
148
212
164
132
116
Rx Key Size: 128 bits
If We dropped the SL Type field and just made each SL Type a separate Database then we could get the RX Key down to 72 bits
Rx Result Size: 64/128 bits
Depends on whether we need to support MA on other end of pass through MetaLink and on what size the NhAddr needs to be.
Tx Key Size: 32 bits
Tx Result Size: 128 bits
We need to watch out for the Tx Result for Tunnels. If we introduce anything else we want in there then we go beyond the 128 bits supportable through the TCAM’s Associated memory.
If we find we can’t use the 128 bit Result size for Tx, we might as well include the Local Memory result in the TCAM lookup.
The Local Memory result was going to be the data that was only dependent on the Physical Interface id and was going to go into Local memory to try to keep the TCAM result below 128 bits.

33. LC: TCAM Lookup Keys on RX
SL(4b)
0000
Port(8b)
MLI(16b)
PAD(100b)
SL (4b)
0001
Port (8b)
MLI (16b)
PAD (52b)
IP SAddr (32b)
EtherType (16b)
0x0800
SL (4b)
0010
Port (8b)
MLI (16b)
PAD (52b)
Ethernet SAddr (48b)
SL(4b)
0011
Port(8b)
MLI(16b)
PAD(100b)
SL (4b)
0100
Port (8b)
PAD (100b)
EtherType (16b)
0x0800
DstAddr (6B)
Blue Shading: Determine SL Type
Black Outline: Key Fields from pkt
DstAddr (6B)
SrcAddr (6B)
SrcAddr (6B)
Type=802.1Q (2B)
Type=802.1Q (2B)
TCI ≠ VLAN0 (2B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
DstAddr (6B)
DstAddr (6B)
TCI ≠ VLAN0 (2B)
Type=IP (2B)
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
SrcAddr (6B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
SrcAddr (6B)
ID/Flags/FragOff (4B)
TTL (1B)
Type=802.1Q (2B)
Type=802.1Q (2B)
TTL (1B)
Protocol=Substrate (1B)
TCI≠VLAN0 (2B)
Protocol (1B)
Hdr Cksum (2B)
TCI=VLAN0 (2B)
Type=Substrate (2B)
Type=Substrate (2B)
Hdr Cksum (2B)
Src Addr (4B)
MLI (2B)
MLI (2B)
Src Addr (4B)
Dst Addr (4B)
LEN (2B)
LEN (2B)
Dst Addr (4B)
MLI (2B)
Meta Frame
Meta Frame
IP Payload
LEN (2B)
Meta Frame
PAD (nB)
PAD (nB)
PAD (nB)
PAD (nB)
CRC (4B)
CRC (4B)
CRC (4B)
CRC (4B)
P2P-DC
Configured
P2P-Tunnel
P2P-VLAN0
Multi-Access
Legacy