Getting ready for day 2 Yesterday’s tree was moved to NetFPGA-10G-live-BACKUP-Day1/ IF you edited code cp NetFPGA-10G-live-BACKUP-Day1/projects/crypto_nic/hw/ pcores/crypto/ hdl/verilog/crypto.v \ NetFPGA-10G-live/ projects/crypto_nic/hw/pcores/crypto/hdl/verilog/ nf10_crypto_v1_00_a.v Edit your nf10_crypto_v1_00_a.v Rename the module to nf10_crypto

NetFPGA Spring Camp Day 2 Presented by: Andrew W. Moore with Marek Michalski, Neelakandan Manihatty-Bojan, Gianni Antichi Georgina Kalogeridou, Jong Hun Han, Noa Zilberman aided by Yury Audzevich, Dimosthenis Pediaditakis Poznan University of Technology May 20 – 24, 2013 http://NetFPGA.org

Previously Covered The Life of a Packet Through the NetFPGA Infrastructure Tree Build System EDK environment The Life of a Packet Through the NetFPGA Hardware Datapath Interface to software: Exceptions and Host I/O Implementation Module Template Write Crypto NIC using a static key Simulation and Debug Write and Run Simulations for Crypto NIC

Tutorial Outline Build and Test Hardware Group Discussion Registers Explain Register System Use AXI Lite registers modules to implement register Add register access stimulus to define Crypto NIC encryption key Update Simulations Build and Test Hardware Build Explanation of Hardware Tests Write and run Hardware Tests Verify value: 0xFFFFFFFF Verify value: 0xFF00FF00 Verify value: 0x55555555 Group Discussion Project Ideas Scope of work that can be accomplished in 2-3 days Team up for Projects Team leaders will describe projects

Crypto Module State Diagram Detect Packet’s Header Header+ Payload Payload

Section I: Registers

Specifying the Key via a Register Can set the key via a register instead Need to understand the register system  Register system: Specify registers stimulus in the reg_stim.axi file Implement registers in module Use instances of ipif_regs and axi_lite_ipif_1bar Other instances available as well (2bar, 3bar, table)

Register bus WRITE WRITE READ READ S_AXI_CLK S_AXI_ARESETN Module S_AXI_ARESETN S_AXI_WREADY S_AXI_AWVALID S_AXI_BRESP S_AXI_WVALID S_AXI_BVALID S_AXI_BREADY WRITE S_AXI_AWREADY WRITE S_AXI_AWADDR S_AXI_ARREADY S_AXI_WDATA S_AXI_RRESP S_AXI_WSTRB S_AXI_RVALID READ S_AXI_ARVALID S_AXI_RDATA READ S_AXI_RREADY S_AXI_ARADDR

ipif_regs Module // -- IPIF REGS ipif_regs #( .C_S_AXI_DATA_WIDTH (C_S_AXI_DATA_WIDTH), .C_S_AXI_ADDR_WIDTH (C_S_AXI_ADDR_WIDTH), .NUM_WO_REGS (NUM_WO_REGS), .NUM_RW_REGS (NUM_RW_REGS), .NUM_RO_REGS (NUM_RO_REGS) ) ipif_regs_inst ( .Bus2IP_Clk ( Bus2IP_Clk ), .Bus2IP_Resetn ( Bus2IP_Resetn ), . . . .wo_regs ( wo_regs ), .rw_regs ( rw_regs ), .ro_regs ( ro_regs ) );

axi_lite_ipif_1bar (in every core) // -- AXILITE IPIF axi_lite_ipif_1bar #( .C_S_AXI_DATA_WIDTH (C_S_AXI_DATA_WIDTH), .C_S_AXI_ADDR_WIDTH (C_S_AXI_ADDR_WIDTH), . . . .C_BAR0_HIGHADDR (C_HIGHADDR) ) axi_lite_ipif_inst ( .S_AXI_ACLK ( S_AXI_ACLK ), .S_AXI_ARESETN ( S_AXI_ARESETN ), .S_AXI_AWREADY ( S_AXI_AWREADY ), // Controls to the IP/IPIF modules .Bus2IP_Clk ( Bus2IP_Clk ), .IP2Bus_Error ( IP2Bus_Error ) );

Additional Related Modules axi_lite_ipif_2bar Same as axi_lite_ipif_1bar but with support for 2 address ranges axi_lite_ipif_3bar Same as axi_lite_ipif_1bar but with support for 3 address ranges ipif_table_regs Read and write to a table

Adding Registers Logic (1) Registers are arranged in memory in the following order: Write only registers Read/Write registers Read only registers Define the number of registers used of each type, e.g.: localparam NUM_RW_REGS = 2; localparam NUM_RO_REGS = 1; Use these parameters for the ipif_regs module No need to define a parameter that is not used (e.g. there are no WO registers)

Adding Registers Logic (2) Ipif_regs exposes 3 registers busses: wo_regs, rw_regs, ro_regs Each bus is (NUM_REGS*C_S_AXI_DATA_WIDTH) wide Each register is assigned C_S_AXI_DATA_WIDTH bits Usage examples: assign ro_regs = {version_reg,counter_reg1,counter_reg2}; always @(posedge Bus2IP_Clk) if (~Bus2IP_Resetn) begin dummy_reg <= 'h0; end else begin dummy_reg <= rw_regs [C_S_AXI_DATA_WIDTH*(DUMMY_REG_ADDR) + 31 : C_S_AXI_DATA_WIDTH*(DUMMY_REG_ADDR) ];

Adding Registers Logic (3) Registers usage: RO, WO, RW refers to software access Write only registers can be written only by the software Read only registers are set by hardware and read by software or hardware Read/Write registers can set by software and read by software or hardware

Testing Registers with Simulation

Simulating Register Access 1. Define register stimulus 2. The testbench executes the stimulus system_axisim_tb reg_stim.axi reg_stim.log DUT 3. Simulation accesses are written to a log file compare 4. A script can compare expected and actual values And declare success or failure Legend: - DUT: Design Under Test - stim: stimulus - tb: testbench - sim: simulation == != PASS FAIL

Registers Stimulus (1) An example of write format : Address Data cd ~/NetFPGA-10G-live/projects/crypto_nic/hw vim reg_stim.axi An example of write format : Address Data Byte Enable strobe

Registers Stimulus (2) An example read format : Address cd ~/NetFPGA-10G-live/projects/crypto_nic/hw vim reg_stim.axi An example read format : Address

Registers Access Log WRITE READ Time cd ~/NetFPGA-10G-live/projects/crypto_nic/hw vim reg_stim.log WRITE READ Time

Replacing Static Key In the crypto project, replace the static key with the key from the registers Provide an Enable registers Update your simulations to set the key

To execute a register simulation cd ~/NetFPGA-10G-live/projects/crypto_nic/hw make simreg

Section II: Build and Test Hardware

Synthesis To synthesize your project CHECK your pao file: cd ~/NetFPGA-10G-live/projects/crypto_nic/hw/pcores/nf10_crypto_v1_00_a/data/ edit nf10_crypto_v2_1_0.pao For synthesis uncomment lines 39,40,43 comment out lines 41 and 42 cd ~/NetFPGA-10G-live/projects/crypto_nic/hw make

Hardware Tests Test compiled hardware Test infrastructure provided to Read/Write registers Read/Write tables Send Packets Check Counters

Example Hardware Tests Reference Router Send Packets from CPU Longest Prefix Matching Longest Prefix Matching Misses Packets dropped when queues overflow Receiving Packets with IP TTL <= 1 Receiving Packets with IP options or non IPv4 Packet Forwarding Dropping packets with bad IP Checksum update

Python Libraries Start packet capture on interfaces Clear all tables in hardware Create packets MAC header IP header PDU Read/Write registers Read/Write reference router tables Longest Prefix Match ARP Destination IP Filter

Hardware Test Examples from ~/NetFPGA-10G-live/<project> reference_nic Simple crossover test test/hw_external_crossover older 1Gbps Reference Router examples Packet Forwarding test/both_packet_forwarding Longest Prefix Match test/both_lpm_generic Send and Receive test/hw_send_rec

Creating a Hardware Test Useful functions: Register access: nftest_regwrite(addr, value) nftest_regread_expect(addr, expect) Packet generation: make_IP_pkt(…) – see documentation encrypt_pkt(key, pkt) decrypt_pkt(key, pkt) Packet transmission/reception: nftest_send_phy(interface, pkt) nftest_expect_phy(interface, pkt) nftest_send_dma(interface, pkt) nftest_expect_dma(interface, pkt) update

Creating a Hardware Test (2) Your task: edit ~/.bashrc and change NF_DESIGN_DIR to be crypto_nic reopen your Terminal (to force this change) cd ~/NetFPGA-10G-live/projects/ cp --archive reference_nic/test crypto_nic/. This creates a directory of hardware tests just like the reference_nic cd ~/NetFPGA-10G-live/projects/projects/crypto_nic/test cp hw_external_loopback hw_crypto_encrypt Now edit hw_crypto_encrypt/run.py to create your tests.

Running Hardware Tests Use command nf_test.py Required Parameter sim hw or both (right now only use hw) Optional parameters --major <major_name> --minor <minor_name> both_crypto_encrypt Run the command nf_test.py hw --major crypto --minor encrypt major minor

Section III: Interface with Software

NetFPGA-Host Interaction (recap) Register reads/writes via ioctl system call with wrapper functions: rdaxi(int address, unsigned *rd_data); wraxi(int address, unsigned *wr_data); eg: rdaxi(0x7d4000000, &val); Useful command line utilities cd ~/NetFPGA-10-live/projects/crypto_nic/sw/host/apps ./rdaxi 0x7d4000000 ./wraxi 0x7d4000000 0x1 Named registers are flakey

Recap Build a complete NetFPGA design Learn: Module creation (Verilog) Reference pipeline integration Verification via simulation Verification via hardware tests Interaction with software

Step 1. Program NetFPGA-10G Guidelines Prepare a bit file, nf10.ko driver for a NetFPGA card cd ~/NetFPGA-10G-live/projects/crypto_nic/sw/host/driver; make OR, do your make from the DESIGN directory cd ~/NetFPGA-10G-live/projects/crypto_nic; make Reference_nic driver is used for most projects. cd ~/NetFPGA-10G-live/projects/crypto_nic/bitfiles Load a bit file for programming FPGA ~/NetFPGA-10G-live/tools/scripts/impact_run.sh <bit_file_name.bit> eg ~/NetFPGA-10G-live/tools/scripts/impact_run.sh crypto_nic.bit Reboot machine (only required once each machine power-up) repeat step 4)

Step 2. Program NetFPGA-10G Guidelines If you are developing new DMA systems steps 6 onward apply To find out loaded bitfile image, run $ lspci –d *:4244 –vxx

Step 3. Program NetFPGA-10G Guidelines ~/NetFPGA-10G-live/tools/scripts/pci_save_restore.sh save dma Now, when the FPGA needs to be programmed, run only step 4) that programs FPGA, restores PCIE configuration, and loads nf10.ko driver.

Step 4. Program NetFPGA-10G Guidelines Go to ./NetFPGA-10G-live/projects/reference_nic/sw/host/driver Compile nf10 driver $ make Load nf10 driver by run insmod $ insmod nf10.ko

Step 5. Program NetFPGA-10G Guidelines Run dmesg to find out the kernel driver $ dmesg

Impact (the FPGA loader) Appendix Xilinx impact has a peculiar behaviour sometimes (this is a known Xilinx impact bug) If the batch mode of impact does not work, a GUI interface appears…. These next slides will guide you through using the GUI.

Step 1. Use of Impact GUI for Program FPGA Appendix Step 1. Use of Impact GUI for Program FPGA Impact GUI can be used for programming FPGA. This steps shows how programs FPGA using Impact GUI. Source Xilinx tools $ source /opt/Xilinx/13.4/ISE_DS/settings64.sh Go to NetFPGA-10G directory and run impact $ impact

Step 2. Use of Impact GUI for Program FPGA Appendix Step 2. Use of Impact GUI for Program FPGA Tick ‘create a new project’ and click OK.

Step 3. Use of Impact GUI for Program FPGA Appendix Step 3. Use of Impact GUI for Program FPGA Tick ‘Configure devices using Boundary-Scan(JTAG) and click OK

Step 4. Use of Impact GUI for Program FPGA Appendix Step 4. Use of Impact GUI for Program FPGA Click Yes

Step 5. Use of Impact GUI for Program FPGA Appendix Step 5. Use of Impact GUI for Program FPGA Select a bit file and click OPEN.

Step 6. Use of Impact GUI for Program FPGA Appendix Step 6. Use of Impact GUI for Program FPGA Click No.

Step 7. Use of Impact GUI for Program FPGA Appendix Step 7. Use of Impact GUI for Program FPGA Click Cancel.

Step 8. Use of Impact GUI for Program FPGA Appendix Step 8. Use of Impact GUI for Program FPGA Click Cancel.

Step 9. Use of Impact GUI for Program FPGA Appendix Step 9. Use of Impact GUI for Program FPGA Double click ‘Program’ This process ONLY loads the FPGA.

Section IV: Wrap-up

Thoughts for Developers Build Modular components Describe shared registers Consider how modules would be used in larger systems Define functionality clearly Through regression tests With repeatable results Disseminate projects Post open-source code Document projects on Web, Wiki Expand the community of developers Answer questions on the Email list Collaborate with your peers to build new applications

NetFPGA.org

Stuck for a NetFPGA project? Build an accurate, fast, line-rate NetDummy/nistnet element A flexible home-grown monitoring card Evaluate new packet classifiers (and application classifiers, and other neat network apps….) Prototype a full line-rate next-generation Ethernet-type Trying any of Jon Crowcrofts’ ideas (Sourceless IP routing for example) Demonstrate the wonders of Metarouting in a different implementation (dedicated hardware) Provable hardware (using a C# implementation and kiwi with NetFPGA as target h/w) Hardware supporting Virtual Routers Check that some brave new idea actually works e.g. Rate Control Protocol (RCP), Multipath TCP, toolkit for hardware hashing MOOSE implementation IP address anonymization SSL decoding “bump in the wire” Xen specialist nic computational co-processor Distributed computational co-processor IPv6 anything IPv6 – IPv4 gateway (6in4, 4in6, 6over4, 4over6, ….) Netflow v9 reference PSAMP reference IPFIX reference Different driver/buffer interfaces (e.g. PFRING) or “escalators” (from gridprobe) for faster network monitors Firewall reference GPS packet-timestamp things High-Speed Host Bus Adapter reference implementations Infiniband iSCSI Myranet Fiber Channel Smart Disk adapter (presuming a direct-disk interface) Software Defined Radio (SDR) directly on the FPGA (probably UWB only) Routing accelerator Hardware route-reflector Internet exchange route accelerator Hardware channel bonding reference implementation TCP sanitizer Other protocol sanitizer (applications… UDP DCCP, etc.) Full and complete Crypto NIC IPSec endpoint/ VPN appliance VLAN reference implementation metarouting implementation virtual <pick-something> intelligent proxy application embargo-er Layer-4 gateway h/w gateway for VoIP/SIP/skype h/w gateway for video conference spaces security pattern/rules matching Anti-spoof traceback implementations (e.g. BBN stuff) IPtv multicast controller Intelligent IP-enabled device controller (e.g. IP cameras or IP powermeters) DES breaker platform for flexible NIC API evaluations snmp statistics reference implementation sflow (hp) reference implementation trajectory sampling (reference implementation) implementation of zeroconf/netconf configuration language for routers h/w openflow and (simple) NOX controller in one… Network RAID (multicast TCP with redundancy) inline compression hardware accelorator for TOR load-balancer openflow with (netflow, ACL, ….) reference NAT device active measurement kit network discovery tool passive performance measurement active sender control (e.g. performance feedback fed to endpoints for control) Prototype platform for NON-Ethernet or near-Ethernet MACs Optical LAN (no buffers) Stuck for a NetFPGA project? Build an accurate, fast, line-rate NetDummy/nistnet element A flexible home-grown monitoring card Evaluate new packet classifiers (and application classifiers, and other neat network apps….) Prototype a full line-rate next-generation Ethernet-type Trying any of Jon Crowcrofts’ ideas (Sourceless IP routing for example) Demonstrate the wonders of Metarouting in a different implementation (dedicated hardware) Provable hardware (using a C# implementation and kiwi with NetFPGA as target h/w) Hardware supporting Virtual Routers Check that some brave new idea actually works e.g. Rate Control Protocol (RCP), Multipath TCP, toolkit for hardware hashing MOOSE implementation IP address anonymization SSL decoding “bump in the wire” Xen specialist nic computational co-processor Distributed computational co-processor IPv6 anything IPv6 – IPv4 gateway (6in4, 4in6, 6over4, 4over6, ….) Netflow v9 reference PSAMP reference IPFIX reference Different driver/buffer interfaces (e.g. PFRING) or “escalators” (from gridprobe) for faster network monitors Firewall reference GPS packet-timestamp things High-Speed Host Bus Adapter reference implementations Infiniband iSCSI Myranet Fiber Channel Smart Disk adapter (presuming a direct-disk interface) Software Defined Radio (SDR) directly on the FPGA (probably UWB only) Routing accelerator Hardware route-reflector Internet exchange route accelerator Hardware channel bonding reference implementation TCP sanitizer Other protocol sanitizer (applications… UDP DCCP, etc.) Full and complete Crypto NIC IPSec endpoint/ VPN appliance VLAN reference implementation metarouting implementation virtual <pick-something> intelligent proxy application embargo-er Layer-4 gateway h/w gateway for VoIP/SIP/skype h/w gateway for video conference spaces security pattern/rules matching Anti-spoof traceback implementations (e.g. BBN stuff) IPtv multicast controller Intelligent IP-enabled device controller (e.g. IP cameras or IP powermeters) DES breaker platform for flexible NIC API evaluations snmp statistics reference implementation sflow (hp) reference implementation trajectory sampling (reference implementation) implementation of zeroconf/netconf configuration language for routers h/w openflow and (simple) NOX controller in one… Network RAID (multicast TCP with redundancy) inline compression hardware accelorator for TOR load-balancer openflow with (netflow, ACL, ….) reference NAT device active measurement kit network discovery tool passive performance measurement active sender control (e.g. performance feedback fed to endpoints for control) Prototype platform for NON-Ethernet or near-Ethernet MACs Optical LAN (no buffers) Well I’m not sure about you but here is a list I created:

Project Ideas for the NetFPGA NetFPGA-10G Test Harness Drop 1-in-N packet module Rate-limited module Event capture module Statistics and Counters Measurement sketch Advanced OPL Input / Output scheduler 40G Port VLAN Tagging …. Ideas from NetFPGA-1G working test and verification harness drop 1-in-n packet module event capture module & rate-limiter module statistics and counters measurement sketches input arbiter queue scheduling VLAN tagging channel bonding 40Gbps MAC (Noa offers to supervise)

Visit http://NetFPGA.org

Wednesday restart from 9am This evening is time to try hw testing & synthesis crypto_nic design if you didn’t manage that Make your group, leader&project Specific, Realistic Time-bounded Modest is good…..

Section IX: Conclusion

Acknowledgments NetFPGA Team at Stanford University (Past and Present): Nick McKeown, Glen Gibb, Jad Naous, David Erickson, G. Adam Covington, John W. Lockwood, Jianying Luo, Brandon Heller, Paul Hartke, Neda Beheshti, Sara Bolouki, James Zeng, Jonathan Ellithorpe, Sachidanandan Sambandan, Eric Lo NetFPGA Team at University of Cambridge (Past and Present): Andrew Moore, David Miller, Muhammad Shahbaz, Martin Zadnik Matthew Grosvenor, Gianni Antichi, Neelakandan Manihatty-Bojan, Georgina Kalogeridou, Jong Hun Han, Noa Zilberman All Community members (including but not limited to): Paul Rodman, Kumar Sanghvi, Wojciech A. Koszek, Yahsar Ganjali, Martin Labrecque, Jeff Shafer, Eric Keller , Tatsuya Yabe, Bilal Anwer, Yashar Ganjali, Martin Labrecque Kees Vissers, Michaela Blott, Shep Siegel

Thanks to our Sponsors: Support for the NetFPGA project has been provided by the following companies and institutions Disclaimer: Any opinions, findings, conclusions, or recommendations expressed in these materials do not necessarily reflect the views of the National Science Foundation or of any other sponsors supporting this project. This effort is also sponsored by the Defense Advanced Research Projects Agency (DARPA) and the Air Force Research Laboratory (AFRL), under contract FA8750-11-C-0249. This material is approved for public release, distribution unlimited. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.