Design of a Diversified Router: Dedicated CRF plus IPv4 Metarouter John DeHart jdd@arl.wustl.edu http://www.arl.wustl.edu/arl

Revision History 5/22/06 (JDD): 6/1/06 (JDD): 6/2/06 (JDD): Created Buffer descriptor stuff probably needs updating. 6/1/06 (JDD): Updating data going between blocks, still in progress. 6/2/06 (JDD): More cleanup of data going between blocks. Buffer descriptor details still need updating. 6/5/06 (JDD): Slight change to format for Lookup Key and defining what goes in each word in the NN ring. Add IP Pkt Length to data Demux passes to Parse 6/6/06 (JDD): Reorganized the Lookup Result given to Hdr Format to distinguish between MR portion and Substrate portion. Clean up labeling of data to Parse (MN vs. IP Pkt) Output from Parse is still IP Pkt Offset and Length. Data from Parse to Lookup needs update to reflect case where lookup is just for Substrate mapping of MI to LC. 6/7/06 (JDD): Updated notes about Parse block’s input/output and functionality 6/15/06 (JDD): Removed CRC from Rx to Demux data. MSF does not pass us a CRC like we thought so we will skip the CRC checking. Updated data going from Demux to Parse, Parse to Lookup and Lookup to Hdr Format

Revision History 6/19/06 (BDH): Split Header Format into MR Header Format and Substrate Encap Demux is now Substrate Decap Reorganization of all slides into logical and physical formats, coloring scheme IPv4 MR now has own section, integrated JL’s internal format slides

Dedicated CRF Slide Organization Lookup Rx Tx QM MR Parse MR Hdr Format Substr Decap Encap L1 L2 L3 Metarouter Block Input Data Output Data Substrate In the “at-a-glance” format, all blocks are logical Logical inputs and outputs High-level overview of processing Each logical block is like an Intel microblock not necessarily an ME. In the detailed format, all blocks are physical Physical inputs and outputs Specific functionality and implementation notes Color scheme Blue = Substrate, should not change! Green = Metarouter, different for each MR

Logical Formats

Receive Rx Coordinate transfer of packets from RBUF to DRAM Lookup Rx Tx QM MR Parse MR Hdr Format Substr Decap Encap L1 L2 L3 Rx Buffer Handle RBUF Ethernet Frame Len Port Coordinate transfer of packets from RBUF to DRAM

Substrate Decapsulate Lookup Rx Tx QM MR Parse MR Hdr Format Substr Decap Encap L1 L2 L3 Buffer Handle Block Buffer Handle Substrate Type Ethernet Frame Len Src ID Port MN Frame Length MN Frame Offset Read and validate Ethernet header from DRAM Read and validate substrate header from DRAM Extract Rx MI or Source MPE Calculate MN frame length and offset

Metarouter Parse MR Parse Lookup Rx Tx QM MR Parse MR Hdr Format Substr Decap Encap L1 L2 L3 Buffer Handle Buffer Handle MR Parse Lookup Flags Parse Flags to Lookup Lookup Key Src ID (RxMI/SrcMPE) Src ID MN Frame Length MN Pkt Length MN Frame Offset to MR Hdr Format MR Data Read and align MN header (includes IPv4 Hdr) from DRAM MR-specific Consume internal header (if packet from other MPE of MR) Header validation Header modification Exception checks Extract lookup key and set lookup flags Write aligned modified IPv4 header back to DRAM

Lookup Lookup Perform lookup in TCAM Rx Tx QM MR Parse MR Hdr Format Substr Decap Encap L1 L2 L3 Buffer Handle Buffer Handle Lookup Lookup Result Flags Lookup Input Flags Dest Addr Lookup Key Output Port Src ID QID MN Pkt Length MR Lookup Result Perform lookup in TCAM Increment counters based on Stats Index Priority resolution of results from multiple databases, if needed

Metarouter Header Format Lookup Rx Tx QM MR Parse MR Hdr Format Substr Decap Encap L1 L2 L3 Buffer Handle Buffer Handle Lookup Result Flags MR Hdr Format MN Frame Length from Lookup Dest Addr MN Frame Offset Output Port Dest Addr QID Output Port MR Lookup Result QID Substrate Type from MR Parse MR Data Substr. Type-dep. Data Process Lookup result For exceptions, generate internal header Decide substrate type

Substrate Encapsulation Lookup Rx Tx QM MR Parse MR Hdr Format Substr Decap Encap L1 L2 L3 Buffer Handle MN Frame Length Substr Encap Buffer Handle MN Frame Offset Output Port Dest Addr QID Output Port MN Frame Length QID Substrate Type Substr. Type-dep. Data Write substrate and ethernet headers

Queue Manager CRF queue management for Meta Interface queues WRR? Lookup Rx Tx QM MR Parse MR Hdr Format Substr Decap Encap L1 L2 L3 Buffer Handle QM Output Port Buffer Handle QID Output Port MN Frame Length CRF queue management for Meta Interface queues WRR? Details

Transmit Tx Coordinate transfer of packets from DRAM to TBUFs Lookup Rx Tx QM MR Parse MR Hdr Format Substr Decap Encap L1 L2 L3 Tx Buffer Handle TBUF Output Port Coordinate transfer of packets from DRAM to TBUFs Recycle buffer handle

Physical Formats

Receive RBUF format details here Buf Handle details here Notes: VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size RBUF Buf Handle(32b) Port (8b) Reserved Eth. Frame Len (16b) RBUF format details here Buf Handle details here Notes: We’ll pass the Buffer Handle which contains the SRAM address of the buffer descriptor. From the SRAM address of the descriptor we can calculate the DRAM address of the buffer data.

Substrate Decapsulate Buf Handle(32b) Port (8b) Reserved Eth. Frame Len (16b) Buf Handle(32b) VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size P Flags (4b) MR ID (VLAN) (12b) Rx MI(16b) MN Frm Length(16b) MN Frm Offset (16b) P Flags: indicate RxMI vs. SrcMPE bit0: 0: RxMI , 1: ScrMPE

Lookup Key[143-112] MR/MI (32b) Metarouter Parse Buf Handle(32b) Buf Handle(32b) VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size L Flags (4b) MR Passthrough (28b) P Flags (4b) MR ID (VLAN) (12b) RxMI/SrcMPE (16b) MR Passthrough (32b) MN Pkt Length (16b) MN Pkt Offset (16b) Lookup Key[143-112] MR/MI (32b) Lookup Key[111-80] (32b) Lookup Key[ 79-48] (32b) Lookup Key[ 47-16] (32b) Lookup Key [15- 0] (16b) Reserved (16b) L Flags: bit 0: 0: Normal, 1: Substrate Lookup bit 1: 0: Normal, 1: NH MN Address present in Key Word[1] Key Word[0] = MR/MI Bit 1 should never be set without bit 0 also being set. Hdr Format needs to start at the beginning of the IP Header and re-write headers upward in the Buffer. Can Parse adjust the buffer/packet size and offset? Can Parse do something like, terminate a tunnel and strip off an outer header? Rx MI needs to be passed to Header Format (through Lookup) so that Header Format can include it in the shim of packets that end up on the slow path. This will allow the Control Processor know what interface the exception packets arrived on.

Lookup Key[143-112] MR/MI (32b) Buf Handle(32b) Buf Handle(32b) VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size L Flags (4b) MR Passthrough (28b) H Flags (4b) MR Passthrough (28b) MR Passthrough (32b) MR Passthrough (32b) Lookup Key[143-112] MR/MI (32b) MR Lookup Result (32b) Lookup Key[111-80] (32b) Lookup Key[ 79-48] (32b) MR Lookup Result (32b) Lookup Key[ 47-16] (32b) DA(8b) Port (4b) QID(20b) Lookup Key [15- 0] (16b) Reserved (16b) L Flags: bit 0: 0: Normal, 1: Substrate Lookup bit 1: 0: Normal, 1: NH MN Address present in Key Word[1] Key Word[0] = MR/MI Bit 1 should never be set without bit 0 also being set.

Metarouter Header Format Buf Handle(32b) Buffer Handle(32b) VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size H Flags (4b) MR Passthrough (28b) Port(8b) Rsv (4b) QID(20b) MR Passthrough (32b) MN Pkt Offset (8b) MN Pkt Length (16b) MR Lookup Result (32b) SH Type (8b) SH Len (8b) Rsv(16b) MR Lookup Result (32b) DA(8b) Port (4b) QID(20b) Substrate Header Data (LWO) Substrate Header Data (LW1) Substrate Header Data (LW2) Text Substrate Header Data (LW3) Substrate Header Data (LW4)

Substrate Encapsulation Buffer Handle(32b) QID(20b) MN Pkt Length (16b) Buffer Handle(32b) Rsv (4b) Reserved (16b) Port(8b) VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size Port(8b) Rsv (4b) QID(20b) MN Pkt Offset (8b) MN Pkt Length (16b) SH Type (8b) SH Len (8b) Rsv(16b) Substrate Header Data (LWO) Substrate Header Data (LW1) Substrate Header Data (LW2) Substrate Header Data (LW3) Substrate Header Data (LW4) Substrate header types/formats here?

Queue Manager Text QID(20b) Buffer Handle(32b) Buffer Handle(32b) VLAN MN Pkt Length (16b) Buffer Handle(32b) Rsv (4b) Reserved (16b) Port(8b) Buffer Handle(32b) Port(8b) Reserved (24b) VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size Text

Transmit Text VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size Buffer Handle(32b) Port(8b) Reserved (24b) TBUF Text

MR Parse and MR Header Format Logical and Physical Data Formats IPv4 Metarouter MR Parse and MR Header Format Logical and Physical Data Formats

IPv4 Logical Formats MR Parse Lookup MR Hdr Format Exception Bits Rx MI / Source MPE MN Frame Length MN Frame Offset MR Parse MR Hdr Format Lookup Buffer Handle Buffer Handle Buffer Handle Buffer Handle Parse Flags Lookup Flags Lookup Result Flags MN Frame Length Rx MI / Source MPE Lookup Key Dest Addr MN Frame Offset MN Frame Length Rx MI / Tx MI Output Port Dest Addr MN Frame Offset QID Output Port IPv4 Lookup Flags QID TxMI Substrate Type IPv4 Lookup Result Substr. Type-dep. Data

Lookup Key[ 47-16] Ports (32b) Lookup Key[143-112] MR/MI (32b) IPv4 Physical Formats RxMI/SrcMPE (16b) Buf Handle(32b) MR ID (VLAN) (12b) MN Pkt Offset (16b) MN Pkt Length (16b) P Flags (4b) IP Pkt Length (16b) Buffer Handle(32b) Reserved (16b) QID(20b) Rsv (4b) Port(8b) MR Parse MR Hdr Format Lookup Lookup Key[111-80] DA (32b) Rx MI/SrcMPE (16b) Buf Handle(32b) IP Pkt Length (16b) IP Pkt Offset (16b) Lookup Key[ 79-48] SA (32b) Lookup Key[ 47-16] Ports (32b) Lookup Key Proto/TCP_Flags [15- 0] (16b) Exception Bits (12b) Reserved (16b) Lookup Key[143-112] MR/MI (32b) L Flags (4b) Port (4b) TxMI(16b) QID(20b) DA(8b) MrBits[31:0](32b) Buf Handle(32b) IP Pkt Length (16b) IP Pkt Offset (16b) Rx MI/SrcMPE (16b) Reserved (11b) H (1b) D Exception Bits (12b) Flags N M A C L H: Hit D: Drop

IPv4 Parse and Header Format Consume internal header Verify IP header (per RFC1812 5.2.2) Decrement TTL Recalc IP checksum Write updated checksum to DRAM Store drop/exception/pass statistics Extract lookup key, set lookup flags Header Format

IPv4 Flag Formats IPv4 Lookup Flags IPv4 Exception Bits H: Hit D: Drop NH: NH MN Address present MAC: MAC Address needed LD: Local Delivery IPv4 Exception Bits Bit 0: TTL = 0 or 1 Bit 1: Options Bit 2?: RxMI or SrcMPE

IPv4 Internal Header Formats For packets going from IPv4 MPE to IPv4 MPE Packets from Ingress LC or to Egress LC don’t have Metanet Internal Header Type (1B) Length (1B) Type Dependent Data (2-6B) RxMI (2B)

IPv4 Internal Header Types Packets entering routing MPE Routing MPE: MPE that does routing lookup FwdKey = [TxMI + MnNhAddr if multi-access link] (**) Source Category Type Value Reason Incoming MR Internal Hdr RMPE Action Ingress LC Normal Fwd No MR Int Hdr Classify and fwd MPE No Classify (w/ FwdKey**) 0x00 If from Control MPE, it is an original pkt that is reinjected back to data path RxMI + FwdKey Perform substrate lookup to resolve LCAddr, port and QID Classify (w/o FwdKey) 0x01 New pkt generated by MPE. If from Control MPE, it is ICMP or local traffic RxMI

IPv4 Internal Header Types Packets exiting Routing MPE If FwdKey in MR Internal hdr is invalid, lookup raises error flag and hdr_fmt sends pkt to CMPE for debug (***) Destination Category Type Value Reason Outgoing MR Internal Hdr Egress LC Fast path No MR Int Hdr Regular MPE 0x02 RxMI + FwdKey Control MPE Exception 0x03 No route RxMI 0x04 TTL expire 0x05 IP w/ options 0x06 Redirect due to RxMI = TxMI Control 0x07 Inspect 0x08 Local delivery Debug 0x09 Monitor 0x0a Log due to error in pkts***

Extra The next set of slides are for templates or extra information if needed

Text Slide Template

Image Slide Template

At-a-glance Block Template Lookup Rx Tx QM MR Parse MR Hdr Format Substr Decap Encap L1 L2 L3 Block Buffer Handle RBUF Ethernet Frame Len Port Text

Detailed Block Template VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size RBUF Buf Handle(32b) Port (8b) Reserved Eth. Frame Len (16b) Text

QM/Scheduler on Multiple MEs Header Format Input Hlpr (1 ME) QM/Schd (1 ME) Tx MR-1 MR-n . . . QM/Schd (1 ME) Tx QID(20b) IP Pkt Length (16b) Buffer Handle(32b) Rsv (4b) Reserved (16b) Port(8b) NN/Scratch Rings Buffer Handle(32b) Port(8b) Reserved (24b) NN Ring QID(32b): Reserved (8b) QM ID (3b) QID(17b): 1M queues per QM Input Hlpr would use QM ID to select Scratch ring on which to put request. QM/Sched then sends on its output NN/scratch ring to its associated Tx With 64 entries in Q-Array and 16 entries in CAM, max number of QM/Schds is probably 4 (2 bits). We’ll set aside 3 bits to give us flexibility in the future.

Packet Buffer Descriptor Tradeoffs Why use a Buffer Descriptor at all? QM needs something to link packets/buffers in queues ME-to-ME communications costs vs. SRAM access costs Specific to Radisys, from dl_meta.u

Packet Buffer Descriptor def Meta Data structure of Packet Buffers (LSB to MSB) buffer_next 32 bits Next Buffer Pointer (in a chain of buffers) offset 16 bits Offset to start of data in bytes BufferSize 16 bits Length of data in the current buffer in bytes header_type 8 bits type of header at offset bytes in to the buffer rx_stat 4 bits Receive status flags free_list 4 bits Freelist ID packet_size 16 bits (Total packet size across multiple buffers) output_port 16 bits Output Port on the egress processor input_port 16 bits Input Port on the ingress processor nhid_type 4 bits Nexthop ID type. reserved 4 bits Reserved fabric_port 8 bits Output port for fabric indicating blade ID. nexthop_id 16 bits NextHop IP ID color 8 bits Qos Color flow_id 24 bits QOS flow ID or MPLS label/flow id reserved 16 bits Reserved class_id 16 bits Class ID packet_next 32 bits pointer to next packet (unused in cell mode) Specific to Radisys, from dl_meta.u

Packet Buffer Descriptor Gets buffer_next: tx Offset: rx, tx, fwd BufferSize: tx, fwd header_type: tx, fwd rx_stat: NONE free_listpacket_size: NONE output_port: qm(?), tx input_port: rx, fwd nhid_type: NONE fabric_port: qm(?), tx nexthop_id color flow_id class_id packet_next Specific to Radisys, from dl_meta.u

Meta Data Caching Meta Data can be cached in one of three places: SRAM Xfer Registers DRAM Xfer Registers GPR Registers Size of Meta Data Cache is controlled by #define META_CACHE_SIZE Macro dl_meta_load_cache[] loads meta data cache buffer_handle: buffer handle for which meta data is to be fetched dl_meta: read transfer register prefix Xbuf_alloc[] should be used to allocate the needed registers signal_number: START_LW: starting long word for fetch NUM_LW: number of long words to fetch Each microengine (microblock?) can use Meta Data Caching differently. Specific to Radisys, from dl_meta.u

Meta Data Caching Specific to Radisys, from dl_meta.u In the ipv4_v6_forwarder sample app, dl_meta_load_cache() used in: Egress ethernet_arp.uc pkt_tx_16p.uc statistics_util.uc tx_helper.uc Ingress dl_meta_get_*[] used in: Ether.uc Ipv4_fwder.uc Ipv4_fwder_util.uc Ipv6_fwder.uc V6v4_tunnel_decap.uc V6v4_tunnel_encap.uc dl_meta_set_*[] used in: pkt_rx_init.uc pkt_rx_two_me_util.uc Specific to Radisys, from dl_meta.u

Buffer Handle

Buffer Descriptor Usage Is there a different Buffer Descriptor defn for LC and PE? Will we support Multi-Buffer Packets? If not, we do not need buffer_next(32b) or buffer_size(16b) QM uses packet_next for its packet chaining in qarray. Output Port and Input Port probably translate to TxMI and RxMI Next Hop fields (nhid_type(4b) and nexthop_id(16b)) probably can go away. QOS fields (color(8b) and flow_id(24b)) probably can go away. Two reserved fields 4b and 16b can go away. class_id(16b) (virtual queue id?) can probably go away. fabric_port can probably go away.

Buffer Descriptor Usage PE Buffer Descriptor: MR_ID (16b) TxMI (16b) VLAN (16b) buffer_next 32 bits Next Buffer Pointer (in a chain of buffers) offset 16 bits Offset to start of data in bytes BufferSize 16 bits Length of data in the current buffer in bytes header_type 8 bits type of header at offset bytes in to the buffer rx_stat 4 bits Receive status flags free_list 4 bits Freelist ID packet_size 16 bits (Total packet size across multiple buffers) output_port 16 bits Output Port on the egress processor input_port 16 bits Input Port on the ingress processor nhid_type 4 bits Nexthop ID type. reserved 4 bits Reserved fabric_port 8 bits Output port for fabric indicating blade ID. nexthop_id 16 bits NextHop IP ID color 8 bits Qos Color flow_id 24 bits QOS flow ID or MPLS label/flow id reserved 16 bits Reserved class_id 16 bits Class ID packet_next 32 bits pointer to next packet (unused in cell mode)

Buffer Descriptor Usage PE Buffer Descriptor: LW0: buffer_next 32 bits Next Buffer Pointer (in a chain of buffers) LW1: offset 16 bits Offset to start of data in bytes LW1: BufferSize 16 bits Length of data in the current buffer in bytes LW2: reserved 8 bits reserved/unused LW2: reserved 4 bits reserved/unused LW2: free_list 4 bits Freelist ID LW2: packet_size 16 bits (Total packet size across multiple buffers) LW3: MR_ID 16 bits Meta Router ID LW3: TxMI 16 bits Transmit Meta Interface LW4: VLAN 16 bits VLAN LW4: reserved 16 bits reserved/unused LW5: reserved 32 bits reserved/unused LW6: reserved 32 bits reserved/unused LW7: packet_next 32 bits pointer to next packet (unused in cell mode) Leave multi-buffer fields there as a template for the dedicated blade implementation of a jumbo-frame MR. Also reduces changes to Rx, Tx, and QM and reduces potential problems.

Multicast Alternatives At least Three Options Force MRs that need Multicast to be Dedicated Blade MRs and do their own Multicast For our short term goals this is probably sufficient and the best course. Perhaps longer term we can look at adding it to the CRF Treat as exception and send to Xscale Provide support in CRF for Multicast Use Multi-Hit Lookup capability of the TCAM MI Bit mask defined in Lookup Result Will put a bound on the number of MIs that can be supported on an MR because of the size of the lookup result. Has issues of mapping bits in the bit mask to actual MIs. Lookup Result contains an index into a table containing MI bit masks Allow but do not force MRs to provide code to interpret Lookup Result. This would also allow other possible extensions on an MR-specific basis This carries with it the problem of bounding the execution time of the MR-specific code in the Lookup block. For general multicast, this could be a serious issue. There are also issues with generating a QID based on an MI when the QID is not included in the Lookup Result. Other options?

CRF Support for Multicast Default/Unicast path MR Interp Parse Header Format MR-Specific Path Post Process Lookup MR-1 MR-1 MR-n . . . . . . MR-n DRAM Buf Ptr MR Id Input MI MR Ctrl Blk Ptr MR Mem Ptr DRAM Buf Ptr MR Id MR Lookup Key MR Ctrl Blk Ptr MR Mem Ptr DRAM Buf Ptr MR Id Output MI QID MR Specific Lookup Result Stats Index MR Ctrl Blk Ptr MR Mem Ptr DRAM Buf Ptr MR Id Output MI Buffer Offset QID

CRF Support for Multicast Default path MR Interp MR-Specific Path Post Process Lookup DRAM Buf Ptr DRAM Buf Ptr MR Id Output MI QID MR Specific Lookup Result Stats Index MR Ctrl Blk Ptr MR Mem Ptr Copy Cnt DRAM Buf Ptr MR Id Output MI QID MR Specific Lookup Result Stats Index MR Ctrl Blk Ptr MR Mem Ptr Copy Cnt=1 MR Id MR Lookup Key MR Ctrl Blk Ptr MR Mem Ptr We will need some kind of copy count or multicast bit and last copy bit to let TX know when it can release the DRAM buffer that holds the packet.

CRF Support for Multicast Default path MR Interp MR-Specific Path Post Process Lookup DRAM Buf Ptr MR Id Output MI QID MR Specific Lookup Result Stats Index MR Ctrl Blk Ptr MR Mem Ptr Copy Cnt DRAM Buf Ptr MR Id Output MI QID MR Specific Lookup Result Stats Index MR Ctrl Blk Ptr MR Mem Ptr Copy Cnt DRAM Buf Ptr MR Id Output MI QID MR Specific Lookup Result Stats Index MR Ctrl Blk Ptr MR Mem Ptr Copy Cnt DRAM Buf Ptr DRAM Buf Ptr Output MI Copy Cnt Output MI Copy Cnt MR Id MR Lookup Key Output MI Copy Cnt MR Lookup Key MR Specific Lookup Result MR Ctrl Blk Ptr MR Ctrl Blk Ptr MR Mem Ptr MR Mem Ptr We will need some kind of copy count or multicast bit and last copy bit to let TX know when it can release the DRAM buffer that holds the packet.

OLD The rest of these are old slides that should be deleted at some point.

Common Router Framework (CRF) Functional Blocks Parse Header Format Rx DeMux Lookup QM Tx MR-1 . . . MR-1 MR-n . . . MR-n VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size RBUF Buf Handle(32b) Rx: Function Coordinate transfer of packets from RBUF to DRAM Notes: We’ll pass the Buffer Handle which contains the SRAM address of the buffer descriptor. From the SRAM address of the descriptor we can calculate the DRAM address of the buffer data.

Common Router Framework (CRF) Functional Blocks Parse Header Format Rx DeMux Lookup QM Tx MR-1 . . . MR-1 MR-n . . . MR-n Buf Handle(32b) MR Id(16b) Input MI(16b) MR Mem Ptr(32b) Buf Handle(32b) DRAM Buf Ptr(32b) Buffer Offset(16b) VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size DeMux Function Read Pkt Header from DRAM Use VLAN from Ethernet header to determine destination MR in order to locate: MR Parse code MR specific memory pointers Write MR Id to Buffer Descriptor Write VLAN to Buffer Descriptor

Common Router Framework (CRF) Functional Blocks Parse Header Format Rx DeMux Lookup QM Tx MR-1 . . . MR-1 MR-n . . . MR-n MR Id(16b) Input MI(16b) MR Mem Ptr(32b) Buf Handle(32b) DRAM Buf Ptr(32b) Buffer Offset(16b) Buf Handle(32b) DRAM Buf Ptr(32b) VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size Buffer Offset(16b) MR Id(16b) Input MI(16b) MR Mem Ptr(32b) MR Lookup Key(16B) Parse Function MR-specific header processing Generate MR-specific lookup key (16 Bytes) from packet Need CRF functionality to managed multiple MRs in shared PE. Notes: Can Parse adjust the buffer/packet size and offset? Can Parse do something like, terminate a tunnel and strip off an outer header?

CRF Wrapper Around Parse MR-1 MR-n . . . MR Selector MR Id Input MI MR Mem Ptr Buf Handle(32b) DRAM Buf Ptr Buffer Offset MR Lookup Key MR Id Input MI MR Mem Ptr Buf Handle(32b) DRAM Buf Ptr Buffer Offset DRAM Buf Ptr Input MI MR Mem Ptr Buffer Offset MR Lookup Key Buffer Offset

Common Router Framework (CRF) Functional Blocks Parse Header Format Rx DeMux Lookup QM Tx MR-1 . . . MR-1 MR-n . . . MR-n MR Lookup Key(16B) MR Id(16b) Input MI(16b) MR Mem Ptr(32b) Buf Handle(32b) DRAM Buf Ptr(32b) Buffer Offset(16b) Buffer Handle(32b) MR Id(16b) Lookup Result(Nb) MR Mem Ptr(32b) DRAM Buf Ptr(32b) Buffer Offset(16b) VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size Lookup Function Perform lookup in TCAM based on MR Id and lookup key Result: Output MI QID Stats index MR-specific Lookup Result (flags, etc. ?) How wide can/should this be?

Common Router Framework (CRF) Functional Blocks Parse Header Format Rx DeMux Lookup QM Tx MR-1 . . . MR-1 MR-n . . . MR-n Buffer Handle(32b) Buffer Handle(32b) QID(16b) VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size DRAM Buf Ptr(32b) Header Format Function MR specific packet header formatting MR specific Lookup Result processing Drop and Miss bits Need CRF functionality to managed multiple MRs in shared PE. Pulls out QID, Length and Port from MR Result, etc. Checks for Drop and Miss bits and deals with those actions. Buffer Offset(16b) Size (16b) Port(8b) MR Id(16b) MR Mem Ptr(32b) Lookup Result(Nb) Includes drop and miss bits

CRF Wrapper Around Header Format MR-1 MR-n . . . Buffer Handle MR Selector Buffer Handle QID Size Port DRAM Buf Ptr(32b) Buffer Offset MR Id DRAM Buf Ptr Output MI MR Specific Lookup Result MR Mem Ptr Buffer Offset MR Mem Ptr Buffer Offset Gets written to Buffer Descriptor May also cause size(s) in Descriptor to be updated. (what about trimming data, What if it is a buffer’s worth Which would change the chaining, Can they add/trim at either end? Lookup Result

Common Router Framework (CRF) Functional Blocks Parse Header Format Rx DeMux Lookup QM Tx MR-1 . . . MR-1 MR-n . . . MR-n Buffer Handle(32b) Buf Handle(32b) QM Function CRF queue management for Meta Interface queues For performance reasons, QM may actually be implemented as multiple instances Each instance on a separate ME would support a separate set of Meta Interfaces. See next slide for more details… QID(16b) VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size Size (16b) Port(8b)

QM/Scheduler on Multiple MEs Header Format Input Hlpr (1 ME) QM/Schd (1 ME) Output Hlpr (1 ME) Tx MR-1 MR-n . . . . . . QM/Schd (1 ME) Buffer Handle(32b) QID(32b) Buf Handle(32b) Scratch Rings Size (16b) NN Ring NN Ring Port(8b) QID(32b): Reserved (8b) QM ID (4b) QID(20b): 1M queues per QM Input Hlpr would use QM ID to select Scratch ring on which to put request. Output Hlpr would process all Scratch rings coming from QM/Schd MEs and multiplex onto one NN ring to TX With 64 entries in Q-Array and 16 entries in CAM, max number of QM/Schds is probably 4 (2 bits). We’ll set aside 4 bits to give us flexibility in the future.

Common Router Framework (CRF) Functional Blocks Parse Header Format Rx DeMux Lookup QM Tx MR-1 . . . MR-1 MR-n . . . MR-n Buffer Handle(32b) TBUF VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size Tx Function Coordinate transfer of packets from DRAM to TBUF

Old Template Tx Function Rx DeMux Parse Lookup Header Format QM Tx Buffer Handle(32b) Port(8b) Reserved (24b) TBUF VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size Tx Function Coordinate transfer of packets from DRAM to TBUF

Old Rejected Overly Busy Slide Lookup Rx Tx QM MR Parse MR Hdr Format Substr Decap Encap L1 L2 L3 Block  Logical interface Data passing between layers Notes here  Physical format Actual Format of data Shows type of communication Scratch Ring NN Ring SRAM Rings  Buf Descriptor shows fields read/written Input Data Output Data Input Data Output Data VLAN Packet_Next MR_ID TxMI Free_List Packet_Size Buffer_Next Offset Buffer Descriptor Buffer_Size