1. Design Slides for RAD Port to techX http://www. arl. wustl
Design Slides for RAD Port to techX some slides take from:
John DeHart

2. Virtual Networking – Basic Concepts
thru
channel
substrate link
virtual link
substrate router
virtual router
substrate links may run over Ethernet, IP, MPLS, . . .
virtual end-system

3. Virtual Router and Substrate
Virtual routers are constructed from
Virtual Interfaces (VI) that connect to virtual links
numbered sequentially from 0 for each VR
have specified bandwidth
may have “internal VLs” connecting VRs in same substrate router
Virtual Packet Processors (VPP)
numbered sequentially from 0
terminate VIs (one VPP may terminate multiple VIs)
one of several types
IXP 2850 type (full system of cluster of size 1, 2, 4, 8 MEs)
V2Pro P100 (full chip or half chip)
Virtual Switch (VS)
has numbered ports that connect to VPPs
ports have specified bandwidth (input, output may differ)
Control software maps virtual routers onto physical resources provided by substrate.

4. Substrate Virtual routers are interconnected via Virtual Links
Virtual Links exists within Substrate Links
Substrate Links are terminated in the Substrate of a Substrate Router
A Substrate Router may contain 0 or more Virtual Routers
A substrate link may contain Virtual Links that connect to several different Virtual Routers
The end of any particular virtual link is connected to one and only one Virtual Router

5. Packet Formats External packet format (Ethernet) Internal format
Ethernet type field identifies packet as IP packet or VNET packet
VNET packets carry Virtual Link Identifier (VLI) and length field
IP packets treated like VNET packets
Internal format
Virtual Router (VR)
Virtual Destination (VDST)
format field (3 bits)
VI, VPP, VPP pair, VPP range
priority (2 bits)
destination (2B - VI, VPP, VPP pair, VPP range)
Physical Destination (PDST)
Format field (3 bits)
port, port pair, port range
destination (2B - port, port pair, port range)
Header Error Check (HEC) detects hw faults
Lookup tables use VR+VDST to obtain PDST.
VNET IP
SRC adr
VLI
DST adr
LEN
CRC
payload
Padding
TYPE
SRC adr
DST adr
CRC
IP packet
Padding
TYPE VR
payload
HEC
VDST
PDST

6. Lookup Tables Line card mapping on input. Line card mapping on output
for VNET type, map VLI to (VR, VDST, PDST)
use VR=0 to distinguish “thru channels”
“thru channels” are ones ones that do not terminate here.
for IP type, map VLI=0 to (VR, VDST, PDST)
rate limit each outgoing stream as configured
Line card mapping on output
map (VR, VDST) to (output interface, Ethernet type, VLI)
rate limit each outgoing stream
PPC substrate mappings.
map (VR, VDST) to (local interface, PDST)
rate limit VI streams from PPs
Switch elements mapping
map VR to rate limit for outgoing stream
per VR queues at each output link
use PDST for forwarding decisions

7. Control Packets
We should support a general mechanism for getting control information to the VRs.
Our IPv4 VR will get them as ATM cells embedded in packets
The CCP right now operates on ATM cells so it will be easiest to maintain that and have the IN module strip the packet away from the embedded ATM Cell
How does the packet get to the VR?
How does the VR recognize something as control vs. data to be routed?
Proposal:
VI=0 Coming in from PPC?

8. Packet Passing Through techX System
VLI
Including
Switch Out Port VI VR …
VR, VDST, PDST
Phys Itf,
Substrate
Link,
VLI LC
Switch
IPP
Switch
OPP
Switch LC

9. Packet Passing Through techX SystemVR
VDST
HEC
PDST
Phys Loc of VPP
VR,VI LC
Switch
IPP
Switch
OPP
Switch LC

10. Packet Passing Through techX SystemVR
VDST
HEC
PDST
Substrate identifies
VR based on physical
Links/signalrs that pkt
Came to it on.
Including
Switch Out Port
VPP VR …
VR, VDST, PDST
VR+VPP LC
Switch
IPP
Switch
OPP
Switch LC

11. Packet Passing Through techX SystemVR
VDST
HEC
PDST
Phys Loc of VPP
VR,VPP LC
Switch
IPP
Switch
OPP
Switch LC

12. Packet Passing Through techX SystemVR
VDST
HEC
PDST
Including
Switch Out Port VI VR …
VR, VDST, PDST
VR,VI LC
Switch
IPP
Switch
OPP
Switch LC

13. Packet Passing Through techX SystemVR
VDST
HEC
PDST
VLI Phys Itf
VI,VR LC
Switch
IPP
Switch
OPP
Switch LC

14. Packet Passing Through techX System
VLI LC
Switch
IPP
Switch
OPP
Switch LC

15. IPv4 Virtual PP PP Card Substrate:
Identifies packet as for a VR on this card
Sends packet to PP/H VR
VDST
HEC
PDST
Phys Loc of VPP VI LC
Switch
IPP
Switch
OPP
Switch LC

16. IPv4 Virtual Packet Processor
SRAM
SRAM
TCAM
RSQ QM IN
CRL
from virtual links
OUT
to virtual links
PSM
SDRAM
Plugin Subsystem
Arriving packets received from multiple virtual links.
Classification & Route Lookup (CRL) implements route lookup and classification, using VLI+IP header fields – target 20 M lookups/s.
Packets stored off-chip by PSM, Queue Manager (QM) schedules packets for outgoing VLs – target 40 M enqueues/s.
Resequencer (RSQ) needed for medium, large systems.

17. IPv4 Virtual Packet Processor
SRAM
TCAM
SRAM
OUT IN
CRL QM
from virtual links
to virtual links
PSM
SDRAM
Plugin Subsystem
Design Changes in Target System:
Change where plugin subsystem connects
Have IN store packet payload to PSM/SDRAM
Have OUT retrieve packet payload from PSM/SDRAM
No RSQ in this design. It must reside in Switch Fabric or at its interface.
Also need to add CCP in to design

18. IPv4 Virtual Packet Processor
Control Cell Processor (CCP)
SRAM
TCAM
SRAM
OUT IN
CRL QM
from virtual links
to virtual links
PSM
SDRAM
Plugin Subsystem

19. Gigabit Ethernet Line CardGE
SUNI GE
FLASH GE
SUNI GE
Substrate V2Pro P100 (?) 4 GE
SUNI GE
Power GE
SUNI
Power GE

20. techX: Packet Processor Card
Power
SDRAM
QDR
2Mx18b
QDR
QDR
SDRAM
QDR
TCAM
Power
PP/H V2Pro P100
40625 (160 pins)
512MB
or 1GB
SDRAM 5
Substrate V2Pro P100
FLASH 10
102.5 G
SDRAM
PP/H V2Pro
P100
SDRAM 5
Power
40625 (160 pins)
TCAM
Power
QDR
QDR
QDR
QDR

21. Gigabit Ethernet Line Card
VNET Ethernet
Frame
Ethernet Frame
SRC adr
VLI
DST adr
LEN
CRC
payload
Padding
TYPE GE
SUNI
SRC adr
VLI
DST adr
LEN
CRC
payload
Padding
VNET VR
payload
HEC
VDST
PDST GE
FLASH GE
SUNI GE
Substrate V2Pro P100 (?) 4 4 GE
SUNI GE
Power GE
SUNI
Power GE

22. techX: Packet Processor Card
Power 10 5
Substrate V2Pro P100
SDRAM
PP/H V2Pro P100
QDR
2Mx18b
PP/H V2Pro
P100
FLASH
40625 (160 pins)
102.5 G
512MB
or 1GB
TCAM VR
payload
HEC
VDST
PDST VR
payload
HEC
VDST
PDST

23. IPv4 Virtual Packet Processor
SRAM
TCAM
SRAM
OUT IN
CRL QM
PSM
SDRAM
Plugin Subsystem

24. IPv4 Virtual Packet Processor
SRAM
TCAM
SRAM
OUT IN
CRL QM
PSM
SDRAM
Plugin Subsystem

25. IPv4 Virtual Packet Processor
SRAM
SRAM
OUT IN
CRL QM
PSM
SDRAM

26. IPv4 Virtual Packet ProcessorIN
CRL QM
OUT
SRAM
SDRAM
TCAM
PSM
RSQ
Plugin Subsystem

27. IPv4 Virtual Packet Processor
SRAM
SRAM
TCAM
RSQ QM IN
CRL
OUT
PSM
SDRAM
Plugin Subsystem

28. IPv4 Virtual Packet Processor
SRAM
SRAM QM IN
CRL
OUT
PSM
SDRAM

29. IPv4 Virtual Packet Processor
SRAM
SRAM IN
CRL QM
OUT
PSM
SDRAM

30. IPv4 Virtual Packet Processor
SRAM
SRAM VR
payload
HEC
VDST
PDST VR
payload
HEC
VDST
PDST IN
CRL QM
Packet Headers/References:
Do we want one uniform header to go through
Or specific header for each interface
Packet Header/Reference
Packet Payload
OUT
PSM
SDRAM

31. IN  CRL ISAR  CARL: 4 words of 64 bits each Flags(16b) IVIN(5b)
OVIN(5b)
PPN(3b)
QID(10b)
PktPtr(20b)
Saddr(32b)
Daddr(32b)
Sport(16b)
Dport(16b)
Protocol(8b)
MTP(8b)
TotalLength(11b)
IpOptionsWord1(32b) (used for LFS)
IpOptionsWord2(32b) (used for LFS) VR
payload
HEC
VDST
PDST IN
CRL
PSM
SDRAM

32. IN  CRL IN  CRL: Flags(16b) PktPtr(20b) Saddr(32b) Daddr(32b)
IVIN(5b)  Input VI
OVIN(5b)
PPN(3b)
QID(10b)
PktPtr(20b)
Saddr(32b)
Daddr(32b)
Sport(16b)
Dport(16b)
Protocol(8b)
MTP(8b)
TotalLength(11b)
IpOptionsWord1(32b) (used for LFS)
IpOptionsWord2(32b) (used for LFS)
WHAT ELSE DO WE NEED for CRL? IN
CRL
SDRAM
PSM VR
payload
HEC
VDST
PDST

33. CRL  QM CARL  QMGR: 3 words of 32 bits each Flags(16b), Mb(1b)
CopyCnt(2b)
QID(10b)
PPN(3b)
Ovin(5b)
PktPtr(20b)
LFS Rate2(8b)
LFS Rate1(8b)
TotalLength(11b)
CRL QM
SRAM

34. CRL  QM CRL  QM Plugin Bit Initial Copy bit Final Copy bit Drop bit
CopyCnt(2b) for monitoring?
Queue Set (Vitrual Interface) (3b)
QID(10b) : size?
PktPtr(20b)
TotalLength(11b)
WHAT ELSE DO WE NEED for QM?
CRL QM
SRAM

35. QM  OUT QMGR  OSAR: 4 words of 32 bits each Flags(16b),
LFS Rate2(8b)
LFS Rate1(8b)
QID(10b)
Mb(1b)
PktPtr(20b)
QueueLength(24b)
SRAM QM
OUT

36. QM  OUT QM  OUT Plugin Bit Free Space Bit (i.e. free PSM memory)
Single Chunk Bit
PluginFlowIndex (QID) (10b)
PktPtr(20b)
Virtual Interface #
WHAT ELSE DO WE NEED for OUT?
SRAM QM
OUT

37. Flags from NSP System Flags (8 bits) Internal Flags (8 bits)
DP: Drop Packet
RC: ReClassify packet
CARL should do its thing (possibly again)
NM: No Match
CARL found no match. Send packet somewhere for handling
EX: Exception packet
Some kind of exception found, like IP options. Send packet somewhere for handling
HO: Header Only
Special header only processing for SPC
HR: Header-only Return
FM: From LC/SW
Directional bit for where packet came from: Line Card or Switch
TO: To LC/SW
Directional bit for where packet is going to: Line Card or Switch
Internal Flags (8 bits)
DG: Datagram packet
Only CARL lookup result was route entry, put in datagram queue
SB: SPC bound packet
SR: SPC return packet
IC: Initial Copy
This is the first copy of the packet (may also be last and only copy) (CARL  QM)
FC: Final Copy
This is the last copy of the packet (may also be first and only copy) (CARL  QM)
LP: LFS option present
SC: Single chunk packet (why was this needed?)
spare

38. Flags for techX System Flags (8 bits) Internal Flags (8 bits)
DP: Drop Packet
RC: ReClassify packet
NM: No Match
EX: Exception packet
How do we want to handle packets that need special processing or that don’t match any route or filter?
HO: Header Only
HR: Header-only Return
FM: From LC/SW
TO: To LC/SW
Internal Flags (8 bits)
DG: Datagram packet
SB: SPC bound packet
SR: SPC return packet
IC: Initial Copy: do we need this?
FC: Final Copy
Tells QM/OUT when packet payload can be released from PSM.
LP: LFS option present
SC: Single chunk packet (why was this needed?)
Spare
Others ??

39. IN  CRL IN  CRL: Flags(16b) Input VI PktPtr(20b) Saddr(32b)
Daddr(32b)
Sport(16b)
Dport(16b)
Protocol(8b)
TotalLength(11b) IN
CRL
SDRAM
PSM VR
payload
HEC
VDST
PDST

40. CRL  QM CRL  QM Flags(16b) CopyCnt(2b) for monitoring?
QID(10b) : size?
Ovin(5b)  VDST?
PktPtr(20b)
TotalLength(11b)
CRL QM
SRAM

41. QM  OUT
QM  OUT
Flags(16b)
OVIN (5b)  VDST?
PktPtr(20b)
SRAM QM
OUT

42. Changes ISAR interface – remove some header fields
FIFOs change due to bram differences
Virtex: 4Kbits (up to 16 bits wide)
V2Pro: 16Kbits (up to 32 bits wide)
Block rams for CARL fifos will need to be regenerated
SRAM interface for CARL:
Asynchronous
125MHz internal rate
200 MHz DDR external rate
May need to redo the clock latency for SRAM access
Width issue
New accesses are 72 bits wide
Old access are 36 bits wide

43. Changes for QM SRAM interface:
Asynchronous
125MHz internal rate
200 MHz DDR external rate
QDR vs. SDR
May need to redo the clock latency for SRAM access
Width issue
New accesses are 72 bits wide
Old access are 36 bits wide
Support for multiple Virtual Links (max=8)
Each Virtual Link has:
A set of N queues (N=256 or 512)
64 Datagram
N-64 Reserved
One set of Plugin Subsystem queues:
256 or 512 queues.
Clean up the intermix of SPC and LC queue handling