1. Cuenca, Ecuador – November 7, 2012 1 NetFPGA Informational Tutorial Presented by: Adam Covington (Stanford University) César Guerrero (Universidad Autónoma de Bucaramanga) IEEE LATINCOM 2012 Cuenca, Ecuador November 7, 2012 http://NetFPGA.org

2. Cuenca, Ecuador – November 7, 2012 2 Tutorial Outline Motivation –Introduction –The NetFPGA Platform Hardware Overview –NetFPGA 1G –NetFPGA 10G The Stanford Base Reference Router –Motivation: Basic IP review –Example: Reference Router running on the NetFPGA Extending the Reference Router –Motivation: Buffer Sizing –Example: Buffer Sizing Experiment running using the NetFPGA Community Contributions Concluding Remarks

3. Cuenca, Ecuador – November 7, 2012 3 Section I: Motivation

4. Cuenca, Ecuador – November 7, 2012 4 NetFPGA = Networked FPGA A line-rate, flexible, open networking platform for teaching and research

5. Cuenca, Ecuador – November 7, 2012 5 NetFPGA 1G Board NetFPGA consists of… Four elements: NetFPGA board Tools + reference designs Contributed projects Community NetFPGA 10G Board

6. Cuenca, Ecuador – November 7, 2012 6 NetFPGA 1GNetFPGA 10G 4 x 1Gbps Ethernet Ports4 x 10Gbps SFP+ 4.5 MB ZBT SRAM 64 MB DDR2 SDRAM 27 MB QDRII-SRAM 288 MB RLDRAM-II PCIPCI Express x8 Virtex II-Pro 50Virtex 5 TX240T NetFPGA Board Comparison

7. Cuenca, Ecuador – November 7, 2012 7 FPGA Memory 1GE NetFPGA board PCI CPU Memory NetFPGA Board PC with NetFPGA Networking Software running on a standard PC A hardware accelerator built with Field Programmable Gate Array driving Gigabit network links

8. Cuenca, Ecuador – November 7, 2012 8 FPGA Memory 1GE Running the Router Kit User-space development, 4x1GE line-rate forwarding PCI CPU Memory OSPFBGP My Protocol user kernel Routing Table Usage #1 IPv4 Router 1GE Fwding Table Packet Buffer “Mirror”

9. Cuenca, Ecuador – November 7, 2012 9 FPGA Memory 1GE Enhancing Modular Reference Designs PCI CPU Memory Usage #2 NetFPGA Driver Java GUI Front Panel (Extensible) PW-OSPF In Q Mgmt IP Lookup L2 Parse L3 Parse Out Q Mgmt 1GE Verilog modules interconnected by FIFO interfaces My Block Verilog EDA Tools (Xilinx, Mentor, etc.) 1.Design 2.Simulate 3.Synthesize 4.Download 1.Design 2.Simulate 3.Synthesize 4.Download

10. Cuenca, Ecuador – November 7, 2012 10 FPGA Memory 1GE Creating new systems PCI CPU Memory Usage #3 NetFPGA Driver 1GE My Design (1GE MAC is soft/replaceable) Verilog EDA Tools (Xilinx, Mentor, etc.) 1.Design 2.Simulate 3.Synthesize 4.Download 1.Design 2.Simulate 3.Synthesize 4.Download

11. Cuenca, Ecuador – November 7, 2012 11 Tools + Reference Designs 1G Tools: Compile designs Verify designs Interact with hardware Reference designs: Router (HW) Switch (HW) Network Interface Card (HW) Router Kit (SW) SCONE (SW)

12. Cuenca, Ecuador – November 7, 2012 12 Contributed Projects More projects: http://netfpga.org/foswiki/NetFPGA/OneGig/ProjectTable ProjectContributor OpenFlow switchStanford University Packet generatorStanford University NetFlow ProbeBrno University NetThreadsUniversity of Toronto zFilter (Sp)routerEricsson Traffic MonitorUniversity of Catania DFAUMass Lowell

13. Cuenca, Ecuador – November 7, 2012 13 Community Wiki Documentation –User’s Guide –Developer’s Guide Encourage users to contribute Forums Support by users for users Active community - 10s-100s of posts/week

14. Cuenca, Ecuador – November 7, 2012 14 International Community Over 1,000 users, using 2,000 cards at 150 universities in 40 countries

15. Cuenca, Ecuador – November 7, 2012 15 NetFPGA’s Defining Characteristics Line-Rate –Processes back-to-back packets Without dropping packets At full rate of Gigabit Ethernet Links –Operating on packet headers For switching, routing, and firewall rules –And packet payloads For content processing and intrusion prevention Open-source Hardware –Similar to open-source software Full source code available BSD-Style License –But harder, because Hardware modules must meeting timing Verilog & VHDL Components have more complex interfaces Hardware designers need high confidence in specification of modules

16. Cuenca, Ecuador – November 7, 2012 16 Test-Driven Design Regression tests –Have repeatable results –Define the supported features –Provide clear expectation on functionality Example: Internet Router –Drops packets with bad IP checksum –Performs Longest Prefix Matching on destination address –Forwards IPv4 packets of length 64-1500 bytes –Generates ICMP message for packets with TTL <= 1 –Defines how packets with IP options or non IPv4 … and dozens more … Every feature is defined by a regression test

17. Cuenca, Ecuador – November 7, 2012 17 Who, How, Why Who uses the NetFPGA? –Teachers –Students –Researchers How do they use the NetFPGA? –To run the Router Kit –To build modular reference designs IPv4 router 4-port NIC Ethernet switch, … Why do they use the NetFPGA? –To measure performance of Internet systems –To prototype new networking systems

18. Cuenca, Ecuador – November 7, 2012 18 Section II: Hardware Overview

19. Cuenca, Ecuador – November 7, 2012 19 NetFPGA-1G

20. Cuenca, Ecuador – November 7, 2012 20 Xilinx Virtex II Pro 50 53,000 Logic Cells Block RAMs Embedded PowerPC

21. Cuenca, Ecuador – November 7, 2012 21 Network and Memory Gigabit Ethernet –4 RJ45 Ports –Broadcom PHY Memories –4.5MB Static RAM –64MB DDR2 Dynamic RAM

22. Cuenca, Ecuador – November 7, 2012 22 Other IO PCI –Memory Mapped Registers –DMA Packet Transferring SATA –Board to Board communication

23. Cuenca, Ecuador – November 7, 2012 23 NetFPGA-10G A major upgrade State-of-the-art technology

24. Cuenca, Ecuador – November 7, 2012 24 NetFPGA 1GNetFPGA 10G 4 x 1Gbps Ethernet Ports4 x 10Gbps SFP+ 4.5 MB ZBT SRAM 64 MB DDR2 SDRAM 27 MB QDRII-SRAM 288 MB RLDRAM-II PCIPCI Express x8 Virtex II-Pro 50Virtex 5 TX240T Comparison

25. Cuenca, Ecuador – November 7, 2012 25 10 Gigabit Ethernet 4 SFP+ Cages NetLogic PHY 10G Support –Direct Attach Copper –10GBASE-R Optical Fiber 1G Support –1000BASE-T Copper –1000BASE-X Optical Fiber

26. Cuenca, Ecuador – November 7, 2012 26 Others QDRII-SRAM –27MB –Storing routing tables, counters and statistics RLDRAM-II –288MB –Packet Buffering PCI Express x8 –PC Interface Expansion Slot

27. Cuenca, Ecuador – November 7, 2012 27 Xilinx Virtex 5 TX240T Optimized for ultra high-bandwidth applications 48 GTX Transceivers 4 hard Tri-mode Ethernet MACs 1 hard PCI Express Endpoint

28. Cuenca, Ecuador – November 7, 2012 28 Beyond Hardware NetFPGA-10G Board Xilinx EDK based IDE Reference designs with ARM AXI4 Software (embedded and PC) Public Repository (GitHub) Public Wiki Reference Designs AXI4 IPs Xilinx EDK MicroBlaze SW PC SW PBWorks, GitHub, User Community

29. Cuenca, Ecuador – November 7, 2012 29 NetFPGA-1G Cube Systems PCs assembled from parts –Stanford University –Cambridge University Pre-built systems available –Accent Technology Inc. Details are in the Guide http://netfpga.org/static/guide.html

30. Cuenca, Ecuador – November 7, 2012 30 Rackmount NetFPGA-1G Servers NetFPGA inserts in PCI or PCI-X slot 2U Server (Dell 2950) Thanks: Brian Cashman for providing machine 1U Server (Accent Technology Inc.)

31. Cuenca, Ecuador – November 7, 2012 31 Stanford NetFPGA-1G Cluster Statistics Rack of 40 1U PCs with NetFPGAs Managed Power Console LANs Provides 4*40=160 Gbps of full line-rate processing bandwidth

32. Cuenca, Ecuador – November 7, 2012 32 Section III: Network review

33. Cuenca, Ecuador – November 7, 2012 33 Internet Protocol (IP) Data IP Hdr Eth Hdr Data IP Hdr Data to be transmitted: IP packets: Ethernet Frames: Data IP Hdr Data IP Hdr Eth Hdr Data IP Hdr Eth Hdr Data IP Hdr … …

34. Cuenca, Ecuador – November 7, 2012 34 Internet Protocol (IP) Data IP Hdr … 16 32 41 Options (if any) Destination Address Source Address Header ChecksumProtocolTTL Fragment Offset Flags Fragment ID Total Packet LengthT.Service HLen Ver 20 bytes

35. Cuenca, Ecuador – November 7, 2012 35 Basic operation of an IP router R3 A B C R1 R2 R4D E F R5 F R3E D Next HopDestination D

36. Cuenca, Ecuador – November 7, 2012 36 Basic operation of an IP router A B C R1 R2 R3 R4D E F R5

37. Cuenca, Ecuador – November 7, 2012 37 Forwarding tables EntryDestinationPort 1 2 ⋮ 2 32 0.0.0.0 0.0.0.1 ⋮ 255.255.255.255 1 2 ⋮ 12 ~ 4 billion entries Naïve approach: One entry per address Improved approach: Group entries to reduce table size EntryDestinationPort 1 2 ⋮ 50 0.0.0.0 – 127.255.255.255 128.0.0.1 – 128.255.255.255 ⋮ 248.0.0.0 – 255.255.255.255 1 2 ⋮ 12 IP address 32 bits wide → ~ 4 billion unique address

38. Cuenca, Ecuador – November 7, 2012 38 IP addresses as a line 0 2 32 -1 EntryDestinationPort 1234512345 Stanford Berkeley North America Asia Everywhere (default) 1234512345 All IP addresses North America Asia BerkeleyStanford Your computerMy computer

39. Cuenca, Ecuador – November 7, 2012 39 Longest Prefix Match (LPM) EntryDestinationPort 1234512345 Stanford Berkeley North America Asia Everywhere (default) 1234512345 Universities Continents Planet Data To: Stanford Matching entries: Stanford North America Everywhere Most specific

40. Cuenca, Ecuador – November 7, 2012 40 Longest Prefix Match (LPM) EntryDestinationPort 1234512345 Stanford Berkeley North America Asia Everywhere (default) 1234512345 Universities Continents Planet Data To: Canada Matching entries: North America Everywhere Most specific

41. Cuenca, Ecuador – November 7, 2012 41 Implementing Longest Prefix Match EntryDestinationPort 1234512345 Stanford Berkeley North America Asia Everywhere (default) 1234512345 Most specific Least specific Searching FOUND

42. Cuenca, Ecuador – November 7, 2012 42 Basic components of an IP router Control Plane Data Plane per-packet processing Switching Forwarding Table Routing Table Routing Protocols Management & CLI Software Hardware Queuing

43. Cuenca, Ecuador – November 7, 2012 43 IP router components in NetFPGA SCONE Routing Table Routing Protocols Management & CLI Output Port Lookup Forwarding Table Input Arbiter Output Queues SwitchingQueuing Linux Routing Table Routing Protocols Management & CLI Router Kit OR Software Hardware

44. Cuenca, Ecuador – November 7, 2012 44 Section IV: Example I

45. Cuenca, Ecuador – November 7, 2012 45 Operational IPv4 router Control Plane Data Plane per-packet processing Software Hardware Routing Table Routing Protocols Management & CLI SCONE Switching Forwarding Table Queuing Reference router Java GUI

46. Cuenca, Ecuador – November 7, 2012 46 Streaming video

47. Cuenca, Ecuador – November 7, 2012 47 Streaming video PC & NetFPGA (NetFPGA in PC) NetFPGA running reference router

48. Cuenca, Ecuador – November 7, 2012 48 Streaming video Video streaming over shortest path Video client Video server

49. Cuenca, Ecuador – November 7, 2012 49 Streaming video Video client Video server

50. Cuenca, Ecuador – November 7, 2012 50 Observing the routing tables Columns: Subnet address Subnet mask Next hop IP Output ports

51. Cuenca, Ecuador – November 7, 2012 51 Demo

52. Cuenca, Ecuador – November 7, 2012 52 Review NetFPGA as IPv4 router: Reference hardware + SCONE software Routing protocol discovers topology Demo: Ring topology Traffic flows over shortest path Broken link: automatically route around failure

53. Cuenca, Ecuador – November 7, 2012 53 Section V: Example II

54. Cuenca, Ecuador – November 7, 2012 54 Buffers in Routers Internal Contention Congestion Pipelining

55. Cuenca, Ecuador – November 7, 2012 55 Buffers in Routers Rx Tx

56. Cuenca, Ecuador – November 7, 2012 56 Extending the Reference Pipeline MAC RxQ MAC RxQ CPU RxQ CPU RxQ MAC RxQ MAC RxQ CPU RxQ CPU RxQ MAC RxQ MAC RxQ CPU RxQ CPU RxQ MAC RxQ MAC RxQ CPU RxQ CPU RxQ Input Arbiter Output Port Lookup MAC TxQ MAC TxQ CPU TxQ CPU TxQ MAC TxQ MAC TxQ CPU TxQ CPU TxQ MAC TxQ MAC TxQ CPU TxQ CPU TxQ MAC TxQ MAC TxQ CPU TxQ CPU TxQ Output Queues Rate Limiter Rate Limiter Event Capture

57. Cuenca, Ecuador – November 7, 2012 57 Topology PC & NetFPGA (NetFPGA in PC) NetFPGA running extended reference router nf2c2 eth1 nf2c1 eth2

58. Cuenca, Ecuador – November 7, 2012 58 Demo

59. Cuenca, Ecuador – November 7, 2012 59 Section V: Community Contributions

60. Cuenca, Ecuador – November 7, 2012 60 FPGA Memory 1GE Running the Router Kit User-space development, 4x1GE line-rate forwarding PCI CPU Memory OSPFBGP My Protocol user kernel Routing Table Usage #1 IPv4 Router 1GE Fwding Table Packet Buffer “Mirror”

61. Cuenca, Ecuador – November 7, 2012 61 FPGA Memory 1GE Enhancing Modular Reference Designs PCI CPU Memory Usage #2 NetFPGA Driver Java GUI Front Panel (Extensible) PW-OSPF In Q Mgmt IP Lookup L2 Parse L3 Parse Out Q Mgmt 1GE Verilog modules interconnected by FIFO interfaces My Block Verilog EDA Tools (Xilinx, Mentor, etc.) 1.Design 2.Simulate 3.Synthesize 4.Download 1.Design 2.Simulate 3.Synthesize 4.Download

62. Cuenca, Ecuador – November 7, 2012 62 FPGA Memory 1GE Creating new systems PCI CPU Memory Usage #3 NetFPGA Driver 1GE My Design (1GE MAC is soft/replaceable) Verilog EDA Tools (Xilinx, Mentor, etc.) 1.Design 2.Simulate 3.Synthesize 4.Download 1.Design 2.Simulate 3.Synthesize 4.Download

63. Cuenca, Ecuador – November 7, 2012 63 Section VI: What to do next?

64. Cuenca, Ecuador – November 7, 2012 64 To get started with your project 1.Get familiar with hardware description language http://www-netfpga.cl.cam.ac.uk/ Support provided by Redgate Software

65. Cuenca, Ecuador – November 7, 2012 65 To get started with your project 2.Prepare for your project b)Complete a hands-on tutorial a)Learn NetFPGA by yourself

66. Cuenca, Ecuador – November 7, 2012 66 Learn by Yourself Users Guide NetFPGA website (www.netfpga.org)

67. Cuenca, Ecuador – November 7, 2012 67 Learn by Yourself Developers Guide NetFPGA website (www.netfpga.org) Forums

68. Cuenca, Ecuador – November 7, 2012 68 Attend (or host) a hands-on tutorial Stanford Cambridge

69. Cuenca, Ecuador – November 7, 2012 69 Attend a hands-on tutorial Events NetFPGA website (www.netfpga.org)

70. Cuenca, Ecuador – November 7, 2012 70 Section VII: Conclusion

71. Cuenca, Ecuador – November 7, 2012 71 Conclusions NetFPGA Provides –Open-source, hardware-accelerated Packet Processing –Modular interfaces arranged in reference pipeline –Extensible platform for packet processing NetFPGA Reference Code Provides –Large library of core packet processing functions –Scripts and GUIs for simulation and system operation –Set of Projects for download from repository The NetFPGA Base Code –Well defined functionality defined by regression tests –Function of the projects documented in the Wiki Guide

72. Cuenca, Ecuador – November 7, 2012 72 Nick McKeown, Glen Gibb, Jad Naous, David Erickson, G. Adam Covington, John W. Lockwood, Jianying Luo, Brandon Heller, Paul Hartke, Neda Beheshti, Sara Bolouki, James Zeng, Jonathan Ellithorpe, Sachidanandan Sambandan, Eric Lo Acknowledgments NetFPGA Team at Stanford University (Past and Present): NetFPGA Team at University of Cambridge (Past and Present): Andrew Moore, David Miller, Muhammad Shahbaz, Martin Zadnik All Community members (including but not limited to): Paul Rodman, Kumar Sanghvi, Wojciech A. Koszek, Yahsar Ganjali, Martin Labrecque, Jeff Shafer, Eric Keller, Tatsuya Yabe, Bilal Anwer, Yashar Ganjali, Martin Labrecque Kees Vissers, Michaela Blott, Shep Siegel

73. Cuenca, Ecuador – November 7, 2012 73 Special thanks to our Partners: Other NetFPGA Tutorial Presented At: SIGMETRICS Ram Subramanian, Patrick Lysaght, Veena Kumar, Paul Hartke, Anna Acevedo Xilinx University Program (XUP) See: http://NetFPGA.org/tutorials/

74. Cuenca, Ecuador – November 7, 2012 74 Thanks to our Sponsors: Support for the NetFPGA project has been provided by the following companies and institutions Disclaimer: Any opinions, findings, conclusions, or recommendations expressed in these materials do not necessarily reflect the views of the National Science Foundation or of any other sponsors supporting this project.

75. Cuenca, Ecuador – November 7, 2012 75 Tutorial Survey www.netfpga.org/tutorial_survey.html

76. Cuenca, Ecuador – November 7, 2012 76 Group Discussion Your plans for using the NetFPGA –Teaching –Research –Other Resources needed for your class –Source code –Courseware –Examples Your plans to contribute –Expertise –Capabilities –Collaboration Opportunities