1. 1 - John DeHart, Jing Lu - 3/8/2016 SRAM ONL NP Router Rx (2 ME) HdrFmt (1 ME) Parse, Lookup, Copy (3 MEs) TCAM SRAM Mux (1 ME) Tx (1 ME) QM (1 ME) xScale Assoc. Data ZBT-SRAM Plugin0Plugin1 Plugin2 Plugin3Plugin4 NN FreeList Mgr (1 ME) Tx, QM Parse Plugin XScale Stats (1 ME) QM Copy Plugins SRAM NN SRAM Ring Scratch Ring NN Ring NN SRAM 64KW Each

2. 2 - John DeHart, Jing Lu - 3/8/2016 MUX -> PLC Flags: Src: Source (2b): 00: Rx 01: XScale 10: Plugin 11: Undefined PT(1b): PassThrough(1)/Classify(0) Reserved (5b) Src (2b) PT (1b) 0 1 2 37 PLC Mux L3 (IP, ARP, …) Pkt Length (16b) Stats Index (16b) QID(16b) In Port (3b) Plugin Tag (5b) Flags (8b) Rsv (4b) Out Port (4b) Buffer Handle(24b)

3. 3 - John DeHart, Jing Lu - 3/8/2016 Flags(8b): Why pkt is being sent to XScale TTL(1b): TTL expired Options(1b): IP Options present NoRoute(1b): No matching route or filter NonIP(1b): Non IP Packet received ARP_Needed(1b): NH_IP valid, but no MAC NH_Invalid(1b): NH_IP AND NH_MAC both invalid Reserved(2b): currently unused PLC -> XScale PLC XScale xScale SRAM L3 (IP, ARP, …) Pkt Length (16b) Buffer Handle(24b) Stats Index (16b) QID(16b) In Port (3b) Flags (8b) Plugin Tag (5b) NH MAC DA[47:16] (32b) NH MAC DA[15:0] (16b) Unicast/MCast bits (16b) Rsv (8b) Reserved (16b) EtherType (16b) Same as ring_in Based on Parse & Lookup results 0 1 2 37 Reserved (2b) NR (1b) TTL (1b) Opt (1b) NI (1b) ARP (1b) NH INV (1b) 45

4. 4 - John DeHart, Jing Lu - 3/8/2016 PLC -> Plugins PLC Plugins 0-4 0 1 2 37 Reserved (2b) NR (1b) TTL (1b) Opt (1b) NI (1b) ARP (1b) NH INV (1b) 45 Flags(8b): Why pkt is being sent to XScale TTL(1b): TTL expired Options(1b): IP Options present NoRoute(1b): No matching route or filter NonIP(1b): Non IP Packet received ARP_Needed(1b): NH_IP valid, but no MAC NH_Invalid(1b): NH_IP AND NH_MAC both invalid Reserved(2b): currently unused xScale SRAM L3 (IP, ARP, …) Pkt Length (16b) Buffer Handle(24b) Stats Index (16b) QID(16b) In Port (3b) Flags (8b) Plugin Tag (5b) NH MAC DA[47:16] (32b) NH MAC DA[15:0] (16b) Unicast/MCast bits (16b) Rsv (8b) Reserved (16b) EtherType (16b) Same as ring_in Based on Parse & Lookup results

5. 5 - John DeHart, Jing Lu - 3/8/2016 PLC -> QM PLCQM Buffer Handle(24b) QID(16b) Rsv (4b) Out Port (4b) L3 (IP, ARP, …) Pkt Length (16b) Reserved(16b) Rsv (8b) Rsv (8b) Same as ring_in Based on Parse & Lookup results Maybe the added hdr buffer handle

6. 6 - John DeHart, Jing Lu - 3/8/2016 Types of Pkts Arriving at PLC From Rx: »Only have a payload buf, ref_cnt == 1. »Subject to classification except mal-formed IP pkts detected by Parse. From XScale/Plugins: »Passthrough (PT) pkt May/May not have a hdr buf, ref_cnt >= 1. Not processed by PLC. Copy sends it to QM. »Non-PT pkt Only have a payload buf at arrival, ref_cnt >= 1. Is IP pkt. Will be classified if it passes IP hdr validation done by Parse.

7. 7 - John DeHart, Jing Lu - 3/8/2016 PLC() { if (!ring_in.PT) { Parse(); if (dlNextBlock != BID_FREELISTMGR) Lookup(); } Copy(); } Inside Copy(), dl_sink() is called to enqueue pkts to downstream blocks.

8. 8 - John DeHart, Jing Lu - 3/8/2016 dl_sink Special design of dl_sink(first, last, try_action) »“first” == TRUE, means current pkt is the first one of a sequence of pkts to be sunk. »“last” == TRUE, means current pkt is the last one of a sequence of pkts to be sunk. »“try_action” == 1, drop the pkt if ring is full; otherwise try till succeeds. »dl_sink() has a return value to indicate if enqueue is successful or not. »If DL_ORDERED is defined and “first” == TRUE, thread waits for signal from previous context before enqueuing the first pkt. »If DL_ORDERED is defined and “last” == TRUE, thread passes signal to next context after euqueuing the last pkt.

9. 9 - John DeHart, Jing Lu - 3/8/2016 Parse Functions: »Do IP Router checks (wrong ver, Hlen, Plen, cksum), count error pkts. »Decrement TTL on pkts from Rx and recompute IP cksum »Detect exceptions (expired TTL, IP option, Non-IP) »Extract lookup key Input: »ring_in data Output: »lookup key, dlNextBlock, eth_type, DG QID. IP DAddr (32b)IP SAddr (32b)DPort (16b)SPort (16b) Proto (8b) TCP Flags (12b) Exceptions (16b) P (3b) P Tag (5b) 140 Bit Key: RL PF and AF

10. 10 - John DeHart, Jing Lu - 3/8/2016 Parse Operations: »If pkt is from Rx: Read eth_type in Ethernet hdr Ø If eth_type != 0x0800, set NIP in exception bits. –If eth_type == 0x0806, set ARP in exception bits. Ø Return. »eth_type <= 0x0800. »For pkt from Rx, offset <= 0x18E; for pkt from XScale/plugins, get offset from payload buf desc. »Read 20B IP hdr. Check for ver, Hlen, Plen. If fails, count and dlNextBlock <= BID_FREELISTMGR. Return. If Hlen > 5, set OPT in exception bits, read IP option. Verify cksum. If fails, count and dlNextBlock <= BID_FREELISTMGR. Return. If pkt is from Rx and TTL <= 1, or pkt is from plugins/XScale and TTL < 1, set TTL in exception bits. If pkt is from Rx and TTL > 1, decrement TTL, recompute cksum and write them back to dram. Form Datagram QID using DG QID = SA[9:8] SA[6:5] DA[6:5] (Used by Copy in case of a zero QID). Extract key from IP hdr. Ø If IP protocol is TCP/UDP, read 14B, and extract key from TCP/UDP hdr. Copy Plugin Tag and In Port to lookup key.

11. 11 - John DeHart, Jing Lu - 3/8/2016 Lookup Key and Results Formats IP DAddr (32b)IP SAddr (32b)DPort (16b)SPort (16b) Proto (8b) TCP Flags (12b) Exceptions (16b) P (3b) P Tag (5b) QID (16b) Stats Index (16b) UCast MCast (12b) V (4b) Prio (8b) D (1b) H (1b) M H (1b) Address (21b) 32 Bit Result in TCAM Assoc. Data SRAM: 96 Bit Result in QDR SRAM Bank0: PF QID (16b) Stats Index (16b) Uni Cast (8b) V (4b) Res (8b) D (1b) H (1b) M H (1b) Address (21b) AF QID (16b) Stats Index (16b) UCast MCast (12b) V (4b) Res (8b) D (1b) H (1b) M H (1b) Address (21b) RL S B (2b) Entry Valid (1b) NH IP Valid (1b) NH MAC Valid (1b) IP MC Valid (1b) NH_MAC (48b) NH_IP (32b)Res (16b) NH_MAC (48b) NH_IP (32b)Res (16b) NH_MAC (48b) NH_IP (32b)Res (16b) Multicast Copy Vector (11b) PPS (1b) If IP MC Valid = 1 D (1b) PPS (1b) UCast Out Port (3b) UCast Out Plugin (3b) Reserved (4b) If IP MC Valid = 0 140 Bit Key: RL PF and AF TCAM Ctrl Bits: D:Done H:HIT MH:Multi-Hit R e s (2b)

12. 12 - John DeHart, Jing Lu - 3/8/2016 Lookup Overview Initialization »Control Plane initializes TCAM and Route and Filter DBs Runtime Updates »Control Plane updates to Route and Filter DBs Design – in upcoming slides Processing – in upcoming slides Lookup will be written in C »There are many things about writing IXP code in “C” that I need to learn. Here are some of them: Performing multiple memory operations in parallel and waiting on a set of signals (If needed for performance reasons) Performing timestamp waits Calling IDT microcode macros

13. 13 - John DeHart, Jing Lu - 3/8/2016 Lookup: Design -- Databases Three Databases: »Route Lookup: Unicast Ø Sorted by DAddr Prefix Length Multicast Ø Exact match on DAddr and prefix of SAddr »Primary Filter Filters should be sorted in the DB with higher priority filters first »Auxiliary Filter Filters should be sorted in the DB with higher priority filters first Priority between Primary Filter and Route Lookup »A priority will be stored with each Primary Filter »A priority will be assigned to RLs (all routes have same priority) »PF priority and RL priority compared after result is retrieved. One of them will be selected based on this priority comparison. Auxiliary Filters: »If matched, cause a copy of packet to be sent out according to the Aux Filter’s result.

14. 14 - John DeHart, Jing Lu - 3/8/2016 Lookup: Design -- Results Use SRAM Bank 0 (2 MB per NPU) for Results »B0 Byte Address Range: 0x000000 – 0x1FFFFF 21 bits »B0 Word Address Range: 0x000000 – 0x1FFFFC 19 significant bits 2 trailing 0’s Store result in two parts: »32-bit Associated Data SRAM result for Address of actual Result: TCAM Control Bits (3b) Ø Done: 1b Ø Hit: 1b Ø MHit: 1b Priority: 8b Ø Present for Primary Filters, for RL and Aux Filters should be 0 SRAM B0 Word Address: 21b Ø 2 spare bitS if needed for anything else »3 Words (<= 96 bits) of Result in SRAM Bank0 Use Multi-Database Lookup (MDL) Indirect for searching all 3 DBs »Order of fields in Key is important. Each thread will need one TCAM context

15. 15 - John DeHart, Jing Lu - 3/8/2016 Lookup Processing write KEY to TCAM use timestamp delay to wait appropriate time make delay long enough that we are as sure as possible that we will have to read the 1 st word of the Results MB only once while !DoneBit // DONE Bit BUG Fix requires reading just first word read 1 word from Results Mailbox and check DoneBit done read words 2 and 3 from Results Mailbox If (PrimaryFilter AND RouteLookup results HIT) { PrimaryResult.Valid  TRUE compare priorities store higher priority result as Primary Result (read result from SRAM Bank0) } else if (PrimaryFilter results HIT) { PrimaryResult.Valid  TRUE PrimaryResults.*  PrimaryFilter.* (read result from SRAM Bank0) } else if (RouterLookup results HIT) { PrimaryResult.Valid  TRUE PrimaryResults.*  RouteLookup.* (read result from SRAM Bank0) } else PrimaryResult.Valid  False if (AuxiliaryFilter result HIT) { AuxiliaryResult.Valid  TRUE AuxiliaryResults.*  (read result from SRAM Bank0) } else AuxiliaryResult.Valid  FALSE

16. 16 - John DeHart, Jing Lu - 3/8/2016 Lookup Key and Results Formats IP DAddr (32b)IP SAddr (32b)DPort (16b)SPort (16b) Proto (8b) TCP Flags (12b) Exceptions (16b) P (3b) P Tag (5b) QID (16b) Stats Index (16b) UCast MCast (12b) V (4b) Prio (8b) D (1b) H (1b) M H (1b) Address (21b) 32 Bit Result in TCAM Assoc. Data SRAM: 96 Bit Result in QDR SRAM Bank0: PF QID (16b) Stats Index (16b) Uni Cast (8b) V (4b) Res (8b) D (1b) H (1b) M H (1b) Address (21b) AF QID (16b) Stats Index (16b) UCast MCast (12b) V (4b) Res (8b) D (1b) H (1b) M H (1b) Address (21b) RL S B (2b) Entry Valid (1b) NH IP Valid (1b) NH MAC Valid (1b) IP MC Valid (1b) NH_MAC (48b) NH_IP (32b)Res (16b) NH_MAC (48b) NH_IP (32b)Res (16b) NH_MAC (48b) NH_IP (32b)Res (16b) Multicast Copy Vector (11b) PPS (1b) If IP MC Valid = 1 D (1b) PPS (1b) UCast Out Port (3b) UCast Out Plugin (3b) Reserved (4b) If IP MC Valid = 0 140 Bit Key: RL PF and AF TCAM Ctrl Bits: D:Done H:HIT MH:Multi-Hit R e s (2b)

17. 17 - John DeHart, Jing Lu - 3/8/2016 Exception Bits in Lookup Key IP DAddr (32b)IP SAddr (32b)DPort (16b)SPort (16b) Proto (8b) TCP Flags (12b) Exceptions (16b) P (3b) P Tag (5b) Non-IP (1b) 140 Bit Key: RL PF and AF ARP (1b) IP Opt (1b) TTL (1b) Reserved (12b) Exception Bits: »TTL: TTL has expired. It was 0 or 1 on arriving packet »IP Opt: IP Packet contained Options »ARP: Ethertype field in ethernet header was ARP »Non-IP: Ethertype field in ethernet header was NOT IP NOTE: An ARP packet will have ARP bit and Non-IP bit set

18. 18 - John DeHart, Jing Lu - 3/8/2016 Lookup Block Diagram Setup Lookup Key Write Lookup Key to TCAM TimeStamp Delay Read 1W Result from AD SRAM Write: 5W SRAM Read: 1W 150 cycles mem access Latency Check Done Bit ctx_swap Read 2W Result from AD SRAM Read: 2W 150 cycles ctx_swap Read 2 Full Results from QDR SRAM Read: 3W 150 cycles ctx_swap SRAM Read: 3W 150 cycles Setup Results for Copy 315 cycles TOTAL (No optimization) 915 cycles

19. 19 - John DeHart, Jing Lu - 3/8/2016 Lookup File locations Code »src/applications/ONL_Router/src/plc/ONL/lookup.c Include Paths »src/applications/ONL_Router/src/dispatch_loop/ONL/ dl_source.h and dl_source.c Ø dl_source() and dl_sink() functions »src/IDT_NSE/data_place_IXP2XXX/include IDT IIPC defines and macros »others?

20. 20 - John DeHart, Jing Lu - 3/8/2016 Copy Functions: »Drop error pkts detected by Parse. »Count and send PT pkts to QM. »Process lookup results: When “control error” (NH_IP and NH_MAC both invalid or both valid), or “ARP request needed” (Uncast pkt with valid NH_IP but invalid NH_MAC), or “no route” (invalid PR and AR) is detected, pkt should be sent to XScale, or one of five plugins, or dropped based on user preference (dynamically configurable). Compute MAC DAddr for IP multicast pkts. Update total ref_cnt in paylod buf desc for each classified pkt. If total ref_cnt == 1 and pkt goes to QM, fill in payload buf desc with NH_MAC, Stats index and EtherType. If total ref_cnt > 1, add hdr buf to each pkt going to QM, and fill in hdr buf desc with buffer_next, packet_size, ref_cnt (=1), NH_MAC, Stats index, EtherType. For each copy, form ring_out data and enqueue it to the outgoing ring. Input: »ring_in data, exception bits, dlNextBlock, eth_type, DG QID, lookup results Output: »Ring_out data

21. 21 - John DeHart, Jing Lu - 3/8/2016 Copy Operations: »Error pkt detected by Parse (to freelist_mgr): Construct ring_out data to Freelist_mgr Call dl_sink(TRUE, TRUE, 0). Return. »PT pkt (to QM): dlNextBlock <= BID_QM Construct ring_out data to QM from ring_in data Call dl_sink(TRUE, TRUE, 0). Return. »Classified pkt: Pre-check: Ø Copy exception bits to flags. Ø If PR and AR are both invalid, this pkt has no route. –Set NR bit in flags. –Set dlNextBlock to user preference. –Construct ring_out data. –Call dl_sink(TRUE, TRUE, 1). Return. Pre-process lookup results, because: Ø Both PR and AR can generate pkt copies to QM and XScale/plugins. Ø Need to know total num of copies to –Decide whether hdr buffer decs should be added to copy going to QM. –Set “last” bit in dl_sink(). –Update total ref_cnt in payload buf desc using one atomic addition. Post-process lookup results, copy and send pkts to proper rings.

22. 22 - John DeHart, Jing Lu - 3/8/2016 Copy Pre-processing data structure: struct copy_plugins_prep{ unsigned int pkt_cnt; bool pr_arp; bool pr_nh_inv; bool pr_ucast; bool pr_mcast; bool ar_arp; bool ar_nh_inv; bool ar_ucast; } plugins; QID (16b) Stats Index (16b) UCast MCast (12b) V (4b) NH_MAC (48b) NH_IP (32b)Res (16b) QID (16b) Stats Index (16b) Uni Cast (8b) V (4b) S B (2b) NH_MAC (48b) NH_IP (32b)Res (16b) R e s (2b) PR: AR: Pre-process fields Post-process fields struct copy_qm_prep{ unsigned int pkt_cnt; bool pr_ucast; bool pr_mcast; bool ar_ucast; } qm; Only one can be TRUE

23. 23 - John DeHart, Jing Lu - 3/8/2016 Copy (pre-process PR) If PR is valid, »If NH_IP is valid and NH_MAC is invalid and IP_MC is invalid, this pkt needs ARP. Set ARP bit in flags. If pkt should be sent to XScale/plugins, Ø plugins.pkt_cnt ++; plugins.pr_arp <= TRUE. »Else if NH_IP and NH_MAC are both invalid or both valid, lookup entry is mis-configured. Set NH_INV bit in flags. If pkt should be sent to XScale/plugins, Ø plugins.pkt_cnt ++; plugins.pr_nh_inv <= TRUE. QID (16b) Stats Index (16b) UCast MCast (12b) V (4b) Entry Valid (1b) NH IP Valid (1b) NH MAC Valid (1b) IP MC Valid (1b) NH_MAC (48b) NH_IP (32b)Res (16b) Multicast Copy Vector (11b) PPS (1b) If IP MC Valid = 1 D (1b) PPS (1b) UCast Out Port (3b) UCast Out Plugin (3b) Reserved (4b) If IP MC Valid = 0

24. 24 - John DeHart, Jing Lu - 3/8/2016 Copy (pre-process PR) »Else, If MC_valid == 0 (unicast), Ø If Drop == 0, –If PPS == 0, qm.pkt_cnt ++, qm.pr_ucast <= TRUE. –Else (PPS == 1), plugins.pkt_cnt ++, plugins.pr_ucast <= TRUE. Else (MC_valid == 1 (Multicast)), Ø If PPS == 0, –qm.pkt_cnt <= total 1’s in high 5 bits in MCast_copyVector; –plugins.pkt_cnt <= total 1’s in low 6 bits in MCAST_copyVector. –If qm.pkt_cnt > 0, qm.pr_mcast <= TRUE. –If plugins.pkt_cnt > 0, plugins.pr_mcast <= TRUE. Ø Else (PPS == 1), –plugins.pkt_cnt <= total 1’s in low 6 bits in Mcast_copyVector; –If plugins.pkt_cnt > 0, plugins.pr_mcast <= TURE. Multicast Copy Vector (11b) PPS (1b) D (1b) PPS (1b) UCast Out Port (3b) UCast Out Plugin (3b) Reserved (4b)

25. 25 - John DeHart, Jing Lu - 3/8/2016 Copy (pre-process AR) If AR is valid, »If NH_IP is valid and NH_MAC is invalid, this pkt needs ARP. Set ARP bit in flags. If pkt should be sent to XScale/plugins, Ø plugins.pkt_cnt ++; plugins.pr_arp <= TRUE. »Else if NH_IP and NH_MAC are both invalid or both valid, lookup entry is mis-configured. Set NH_INV bit in flags. If pkt should be sent to XScale/plugins, Ø plugins.pkt_cnt ++; plugins.pr_nh_inv <= TRUE. QID (16b) Stats Index (16b) Uni Cast (8b) V (4b) S B (2b) NH_MAC (48b) NH_IP (32b)Res (16b) R e s (2b) Entry Valid (1b) NH IP Valid (1b) NH MAC Valid (1b) IP MC Valid (1b) D (1b) PPS (1b) UCast Out Port (3b) UCast Out Plugin (3b)

26. 26 - John DeHart, Jing Lu - 3/8/2016 Copy (pre-process AR) »Else, Each SB value is associated with a sampling rate rt(SB) that is dynamically configurable by users. Generate a random number rd. If rd <= rt(SB), Ø If PPS == 0, –qm.ar_ucast <= TRUE, qm.pkt_cnt ++. Ø Else (PPS == 1), –plugins.ar_ucast <= TRUE, plugins.pkt_cnt ++. D (1b) PPS (1b) UCast Out Port (3b) UCast Out Plugin (3b) Rsv (2b) SB (2b)

27. 27 - John DeHart, Jing Lu - 3/8/2016 Copy (post-process) If qm.pkt_cnt + plugins.pkt_cnt == 0, dlNextBlock <= BID_FREELISTMGR, dl_sink(TRUE, TRUE, 0). Return. Read ref_cnt from payload buf desc. If qm.pkt_cnt + plugins.pkt_cnt + (ref_cnt – 1) == 1, »If qm.pkt_cnt == 1, DO NOT add hdr buf desc. Based on qm.pr_ucast, qm.pr_mcast, qm.ar_ucast, Ø Fill in NH_MAC, stats index, and eth_type in payload buf desc. Ø Construct QM ring_out data. Ø dlNextBlock <= BID_QM. Ø dl_sink(TRUE, TRUE, 0). Return. »Else (plugins.pkt_cnt == 1), Based on plugins.*, Ø construct plugins/XScale ring_out data. Ø Set dlNextBlock to XScale or one of the five plugins. Ø dl_sink(TRUE, TRUE, 1). Return. struct copy_plugins_prep{ unsigned int pkt_cnt; bool pr_arp; bool pr_nh_inv; bool pr_ucast; bool pr_mcast; bool ar_arp; bool ar_nh_inv; bool ar_ucast; } plugins; struct copy_qm_prep{ unsigned int pkt_cnt; bool pr_ucast; bool pr_mcast; bool ar_ucast; } qm; QID (16b) Stats Index (16b) UCast MCast (12b) V (4b) NH_MAC (48b) NH_IP (32b)Res (16b) QID (16b) Stats Index (16b) Uni Cast (8b) V (4b) S B (2b) NH_MAC (48b) NH_IP (32b)Res (16b) R e s (2b) PR: AR:

28. 28 - John DeHart, Jing Lu - 3/8/2016 Copy (post-process) Else (qm.pkt_cnt + plugins.pkt_cnt + (ref_cnt – 1) > 1), »Add (qm.pkt_cnt + plugins.pkt_cnt – 1) to ref_cnt in payload buf desc. »If qm.pkt_cnt >0, for each copy to QM, add hdr buf desc. Based on qm.pr_ucast, qm.pr_mcast, qm.ar_ucast, Ø Fill in buffer_next, packet_size, ref_cnt (=1), NH, stats index, and eth_type in hdr buf desc. Ø Construct QM ring_out data. Ø dlNextBlock <= BID_QM. Ø Call dl_sink() qm.pkt_cnt times. –If plugins.pkt_cnt == 0, Set “last” bit in dl_sink() for the last copy and return. »If plugins.pkt_cnt > 0, Based on plugins.*, Ø Construct plugins/XScale ring_out data. Ø Set dlNextBlock to XScale or one of the five plugins. Ø Call dl_sink() plugins.pkt_cnt times. –Set “last” bit in dl_sink() for the last copy and return.

29. 29 - John DeHart, Jing Lu - 3/8/2016 PLC Typical Case Valid IP Pkt from Rx, no option, go to one port as a result of PR. AR entry is invalid. Read eth_type, IP, TCP/UDP hdr Aligned 10LW DRAM read 300 Operation Memory AccessCycle Count Write TTL, IP cksum Aligned 4LW DRAM write 300 Parse Write NH MAC, stats index, eth_type Aligned 3LW SRAM write 150 Enqueue pkt to QM 3LW Scratch Ring write 60 Copy Lookup 915 Lookup Total: 1785 Dequeue pkt from MUX 3LW Scratch Ring read 60 dl_sink dl_source