System Architecture A Reconfigurable and Programmable Gigabit Network Interface Card Jeff Shafer, Hyong-Youb Kim, Paul Willmann, Dr. Scott Rixner Rice Computer Architecture: Hardware Research  What NIC architectures scale to support future high-performance offloading at multi-gigabit data rates?  Can reconfiguring the FPGA hardware provide a substantial performance improvement for different applications? Is this improvement greater than simply upgrading the NIC firmware?  What NIC architectures scale to support future high-performance offloading at multi-gigabit data rates?  Can reconfiguring the FPGA hardware provide a substantial performance improvement for different applications? Is this improvement greater than simply upgrading the NIC firmware? AbstractAbstract  In this project, we are designing and implementing a gigabit NIC on a FPGA. This NIC will support research in network server operating systems and network architecture design  Why are existing NICs insufficient for this research?  Current software programmable NICs (e.g. Alteon Tigon) are too slow for modern complex offloading tasks  Current ASIC-based NICs have limited functionality and leave most tasks to the host processor. This architecture does not scale well to high data rates  Commercial products often use locked or proprietary IP, limiting our design flexibility  Why do we need a FPGA-based NIC?  FPGA provides maximum flexibility for future research  For architecture research, we need a flexible NIC with hardware acceleration. This NIC is both reconfigurable (runs frequent / demanding tasks in hardware) and programmable (runs infrequent / flexible tasks in software)  In this project, we are designing and implementing a gigabit NIC on a FPGA. This NIC will support research in network server operating systems and network architecture design  Why are existing NICs insufficient for this research?  Current software programmable NICs (e.g. Alteon Tigon) are too slow for modern complex offloading tasks  Current ASIC-based NICs have limited functionality and leave most tasks to the host processor. This architecture does not scale well to high data rates  Commercial products often use locked or proprietary IP, limiting our design flexibility  Why do we need a FPGA-based NIC?  FPGA provides maximum flexibility for future research  For architecture research, we need a flexible NIC with hardware acceleration. This NIC is both reconfigurable (runs frequent / demanding tasks in hardware) and programmable (runs infrequent / flexible tasks in software) Hardware Design  Virtex-II Pro FPGA  MHz PowerPC processors  MAC controller - Complete  DDR memory controller – Complete  128MB - 1GB DDR Memory  Serial Port Debug – Complete  DMA engine – Pending  Checksum Offloading - Pending  Spartan-IIE FPGA  64-bit/66-MHz PCI interface – Complete  2MB SRAM - Complete  Event notification - Complete  Virtex-II Pro FPGA  MHz PowerPC processors  MAC controller - Complete  DDR memory controller – Complete  128MB - 1GB DDR Memory  Serial Port Debug – Complete  DMA engine – Pending  Checksum Offloading - Pending  Spartan-IIE FPGA  64-bit/66-MHz PCI interface – Complete  2MB SRAM - Complete  Event notification - Complete Software Design  PowerPC processor runs custom packet-handling firmware  Interfaces with MAC controller to send/receive packets  Interfaces with DDR controller to store bulk frame data  Interfaces with DMA controller to transfer data to/from host  Interfaces with SRAM over Spartan bridge to access PCI control registers  Custom device drivers written for Linux and FreeBSD  PowerPC processor runs custom packet-handling firmware  Interfaces with MAC controller to send/receive packets  Interfaces with DDR controller to store bulk frame data  Interfaces with DMA controller to transfer data to/from host  Interfaces with SRAM over Spartan bridge to access PCI control registers  Custom device drivers written for Linux and FreeBSD Device Utilization  Substantial headroom for future development ComponentUtilization Slice Registers8,257/ 27,39230% 4 input LUTs (Logic)8,357/ 27,39230% BRAMs100 / 13673% Occupied Slices8,573 / 13,69662% Global Clocks13 / 1681% Digital Clock Managers5 / 862% Total Gate Count:6,890,605 OS Research Development Platform Serial Port RJ-45 Port DDR Virtex FPGA Spartan FPGA PCI InterfaceEthernet PHY PCI / FPGA Development Board PCI Bridge (Spartan FPGA) PCI Bus (64b / 66 MHz) SRAM (2MB) 32b / 66 MHz PROM 64b / 66 MHz RS-232 (Debug) Ethernet PHY (10/100/1000) Full Duplex Ethernet 2 GBits/sec 8b / 125 MHz PowerPC 300MHz DDR Controller BRAM Block Spartan Bridge PLB – 64b / 100 MHz BRAM Block On-Chip Mem MAC Control PLB / OPB Bridge UART (OPB Bus) DDR-SDRAM (256 MB) Serial Port Memory / Control Ethernet PCI Debug 32b / 125 MHz 32b / 66 MHz Custom FPGA Design Custom FPGA Design  Connection Handoff to NIC  Offload network stack processing to NIC  Reconfigurable NIC allows exploration of hardware / software mechanisms to accelerate offloading process  Thread Parallelization of Network Stack  Parallel network stacks achieve high network throughput but have synchronization challenges  Control-synchronous parallel stacks use the OS scheduler to ensure connections are always managed by the same thread, eliminating some (but not all) locks  Exploring ways to use NIC to eliminate device queue lock to enable simultaneous communication among multiple threads in parallel stacks  Connection Handoff to NIC  Offload network stack processing to NIC  Reconfigurable NIC allows exploration of hardware / software mechanisms to accelerate offloading process  Thread Parallelization of Network Stack  Parallel network stacks achieve high network throughput but have synchronization challenges  Control-synchronous parallel stacks use the OS scheduler to ensure connections are always managed by the same thread, eliminating some (but not all) locks  Exploring ways to use NIC to eliminate device queue lock to enable simultaneous communication among multiple threads in parallel stacks