1. Getting ready for day 2
Yesterday’s tree was moved to NetFPGA-10G-live-BACKUP-Day1/
IF you edited code
cp NetFPGA-10G-live-BACKUP-Day1/projects/crypto_nic/hw/ pcores/crypto/ hdl/verilog/crypto.v \
NetFPGA-10G-live/ projects/crypto_nic/hw/pcores/crypto/hdl/verilog/ nf10_crypto_v1_00_a.v
Edit your nf10_crypto_v1_00_a.v
Rename the module to nf10_crypto

2. NetFPGA Spring Camp Day 2
Presented by:
Andrew W. Moore
with
Marek Michalski, Neelakandan Manihatty-Bojan, Gianni Antichi
Georgina Kalogeridou, Jong Hun Han, Noa Zilberman
aided by
Yury Audzevich, Dimosthenis Pediaditakis
Poznan University of Technology
May 20 – 24, 2013

3. Previously Covered The Life of a Packet Through the NetFPGA
Infrastructure
Tree
Build System
EDK environment
The Life of a Packet Through the NetFPGA
Hardware Datapath
Interface to software: Exceptions and Host I/O
Implementation
Module Template
Write Crypto NIC using a static key
Simulation and Debug
Write and Run Simulations for Crypto NIC

4. Tutorial Outline Build and Test Hardware Group Discussion
Registers
Explain Register System
Use AXI Lite registers modules to implement register
Add register access stimulus to define Crypto NIC encryption key
Update Simulations
Build and Test Hardware
Build
Explanation of Hardware Tests
Write and run Hardware Tests
Verify value: 0xFFFFFFFF
Verify value: 0xFF00FF00
Verify value: 0x
Group Discussion
Project Ideas
Scope of work that can be accomplished in 2-3 days
Team up for Projects
Team leaders will describe projects

5. Crypto Module State Diagram
Detect
Packet’s
Header
Header+
Payload
Payload

6. Section I: Registers

7. Specifying the Key via a Register
Can set the key via a register instead
Need to understand the register system 
Register system:
Specify registers stimulus in the reg_stim.axi file
Implement registers in module
Use instances of ipif_regs and axi_lite_ipif_1bar
Other instances available as well (2bar, 3bar, table)

8. Register bus WRITE WRITE READ READ S_AXI_CLK S_AXI_ARESETN
Module
S_AXI_ARESETN
S_AXI_WREADY
S_AXI_AWVALID
S_AXI_BRESP
S_AXI_WVALID
S_AXI_BVALID
S_AXI_BREADY
WRITE
S_AXI_AWREADY
WRITE
S_AXI_AWADDR
S_AXI_ARREADY
S_AXI_WDATA
S_AXI_RRESP
S_AXI_WSTRB
S_AXI_RVALID
READ
S_AXI_ARVALID
S_AXI_RDATA
READ
S_AXI_RREADY
S_AXI_ARADDR

9. ipif_regs Module
// -- IPIF REGS ipif_regs #( .C_S_AXI_DATA_WIDTH (C_S_AXI_DATA_WIDTH), .C_S_AXI_ADDR_WIDTH (C_S_AXI_ADDR_WIDTH), .NUM_WO_REGS (NUM_WO_REGS), .NUM_RW_REGS (NUM_RW_REGS), .NUM_RO_REGS (NUM_RO_REGS) ) ipif_regs_inst ( .Bus2IP_Clk ( Bus2IP_Clk ), .Bus2IP_Resetn ( Bus2IP_Resetn ), wo_regs ( wo_regs ), .rw_regs ( rw_regs ), .ro_regs ( ro_regs ) );

10. axi_lite_ipif_1bar (in every core)
// -- AXILITE IPIF axi_lite_ipif_1bar #( .C_S_AXI_DATA_WIDTH (C_S_AXI_DATA_WIDTH), .C_S_AXI_ADDR_WIDTH (C_S_AXI_ADDR_WIDTH), C_BAR0_HIGHADDR (C_HIGHADDR) ) axi_lite_ipif_inst ( .S_AXI_ACLK ( S_AXI_ACLK ), .S_AXI_ARESETN ( S_AXI_ARESETN ), .S_AXI_AWREADY ( S_AXI_AWREADY ), // Controls to the IP/IPIF modules .Bus2IP_Clk ( Bus2IP_Clk ), .IP2Bus_Error ( IP2Bus_Error ) );

11. Additional Related Modules
axi_lite_ipif_2bar
Same as axi_lite_ipif_1bar but with support for 2 address ranges
axi_lite_ipif_3bar
Same as axi_lite_ipif_1bar but with support for 3 address ranges
ipif_table_regs
Read and write to a table

12. Adding Registers Logic (1)
Registers are arranged in memory in the following order:
Write only registers
Read/Write registers
Read only registers
Define the number of registers used of each type, e.g.:
localparam NUM_RW_REGS = 2;
localparam NUM_RO_REGS = 1;
Use these parameters for the ipif_regs module
No need to define a parameter that is not used
(e.g. there are no WO registers)

13. Adding Registers Logic (2)
Ipif_regs exposes 3 registers busses:
wo_regs, rw_regs, ro_regs
Each bus is (NUM_REGS*C_S_AXI_DATA_WIDTH) wide
Each register is assigned C_S_AXI_DATA_WIDTH bits
Usage examples:
assign ro_regs = {version_reg,counter_reg1,counter_reg2};
Bus2IP_Clk)
if (~Bus2IP_Resetn) begin
dummy_reg <= 'h0;
end
else begin
dummy_reg <= rw_regs [C_S_AXI_DATA_WIDTH*(DUMMY_REG_ADDR) + 31 : C_S_AXI_DATA_WIDTH*(DUMMY_REG_ADDR) ];

14. Adding Registers Logic (3)
Registers usage:
RO, WO, RW refers to software access
Write only registers can be written only by the software
Read only registers are set by hardware and read by software or hardware
Read/Write registers can set by software and read by software or hardware

15. Testing Registers with Simulation

16. Simulating Register Access
1. Define register
stimulus
2. The testbench executes the stimulus
system_axisim_tb
reg_stim.axi
reg_stim.log
DUT
3. Simulation accesses are written to a log file
compare
4. A script can compare expected and actual values
And declare success or failure
Legend:
- DUT: Design Under Test
- stim: stimulus
- tb: testbench
- sim: simulation == !=
PASS
FAIL

17. Registers Stimulus (1) An example of write format : Address Data
cd ~/NetFPGA-10G-live/projects/crypto_nic/hw
vim reg_stim.axi
An example of write format :
Address
Data
Byte Enable strobe

18. Registers Stimulus (2) An example read format : Address
cd ~/NetFPGA-10G-live/projects/crypto_nic/hw
vim reg_stim.axi
An example read format :
Address

19. Registers Access Log WRITE READ Time
cd ~/NetFPGA-10G-live/projects/crypto_nic/hw
vim reg_stim.log
WRITE
READ
Time

20. Replacing Static Key
In the crypto project, replace the static key with the key from the registers
Provide an Enable registers
Update your simulations to set the key

21. To execute a register simulation cd ~/NetFPGA-10G-live/projects/crypto_nic/hw make simreg

22. Section II: Build and Test Hardware

23. Synthesis To synthesize your project CHECK your pao file:
cd ~/NetFPGA-10G-live/projects/crypto_nic/hw/pcores/nf10_crypto_v1_00_a/data/
edit nf10_crypto_v2_1_0.pao
For synthesis
uncomment lines 39,40,43
comment out lines 41 and 42
cd ~/NetFPGA-10G-live/projects/crypto_nic/hw
make

24. Hardware Tests Test compiled hardware Test infrastructure provided to
Read/Write registers
Read/Write tables
Send Packets
Check Counters

25. Example Hardware Tests
Reference Router
Send Packets from CPU
Longest Prefix Matching
Longest Prefix Matching Misses
Packets dropped when queues overflow
Receiving Packets with IP TTL <= 1
Receiving Packets with IP options or non IPv4
Packet Forwarding
Dropping packets with bad IP Checksum
update

26. Python Libraries Start packet capture on interfaces
Clear all tables in hardware
Create packets
MAC header
IP header
PDU
Read/Write registers
Read/Write reference router tables
Longest Prefix Match
ARP
Destination IP Filter

27. Hardware Test Examples
from ~/NetFPGA-10G-live/<project>
reference_nic
Simple crossover test
test/hw_external_crossover
older 1Gbps Reference Router examples
Packet Forwarding
test/both_packet_forwarding
Longest Prefix Match
test/both_lpm_generic
Send and Receive
test/hw_send_rec

28. Creating a Hardware Test
Useful functions:
Register access:
nftest_regwrite(addr, value)
nftest_regread_expect(addr, expect)
Packet generation:
make_IP_pkt(…) – see documentation
encrypt_pkt(key, pkt)
decrypt_pkt(key, pkt)
Packet transmission/reception:
nftest_send_phy(interface, pkt)
nftest_expect_phy(interface, pkt)
nftest_send_dma(interface, pkt)
nftest_expect_dma(interface, pkt)
update

29. Creating a Hardware Test (2)
Your task:
edit ~/.bashrc and change NF_DESIGN_DIR to be crypto_nic
reopen your Terminal (to force this change)
cd ~/NetFPGA-10G-live/projects/
cp --archive reference_nic/test crypto_nic/.
This creates a directory of hardware tests just like the reference_nic
cd ~/NetFPGA-10G-live/projects/projects/crypto_nic/test
cp hw_external_loopback hw_crypto_encrypt
Now edit
hw_crypto_encrypt/run.py
to create your tests.

30. Running Hardware Tests
Use command nf_test.py
Required Parameter
sim hw or both (right now only use hw)
Optional parameters
--major <major_name>
--minor <minor_name>
both_crypto_encrypt
Run the command
nf_test.py hw --major crypto --minor encrypt
major
minor

31. Section III: Interface with Software

32. NetFPGA-Host Interaction (recap)
Register reads/writes via ioctl system call with wrapper functions:
rdaxi(int address, unsigned *rd_data);
wraxi(int address, unsigned *wr_data);
eg:
rdaxi(0x7d , &val);
Useful command line utilities
cd ~/NetFPGA-10-live/projects/crypto_nic/sw/host/apps
./rdaxi 0x7d
./wraxi 0x7d x1
Named registers are flakey

33. Recap Build a complete NetFPGA design Learn: Module creation (Verilog)
Reference pipeline integration
Verification via simulation
Verification via hardware tests
Interaction with software

34. Step 1. Program NetFPGA-10G Guidelines
Prepare a bit file, nf10.ko driver for a NetFPGA card
cd ~/NetFPGA-10G-live/projects/crypto_nic/sw/host/driver; make
OR, do your make from the DESIGN directory
cd ~/NetFPGA-10G-live/projects/crypto_nic; make
Reference_nic driver is used for most projects.
cd ~/NetFPGA-10G-live/projects/crypto_nic/bitfiles
Load a bit file for programming FPGA
~/NetFPGA-10G-live/tools/scripts/impact_run.sh <bit_file_name.bit> eg
~/NetFPGA-10G-live/tools/scripts/impact_run.sh crypto_nic.bit
Reboot machine (only required once each machine power-up)
repeat step 4)

35. Step 2. Program NetFPGA-10G Guidelines
If you are developing new DMA systems steps 6 onward apply
To find out loaded bitfile image, run
$ lspci –d *:4244 –vxx

36. Step 3. Program NetFPGA-10G Guidelines
~/NetFPGA-10G-live/tools/scripts/pci_save_restore.sh save dma
Now, when the FPGA needs to be programmed, run only step 4) that programs FPGA, restores PCIE configuration, and loads nf10.ko driver.

37. Step 4. Program NetFPGA-10G Guidelines
Go to ./NetFPGA-10G-live/projects/reference_nic/sw/host/driver
Compile nf10 driver
$ make
Load nf10 driver by run insmod
$ insmod nf10.ko

38. Step 5. Program NetFPGA-10G Guidelines
Run dmesg to find out the kernel driver
$ dmesg

39. Impact (the FPGA loader)
Appendix
Xilinx impact has a peculiar behaviour sometimes (this is a known Xilinx impact bug)
If the batch mode of impact does not work, a GUI interface appears….
These next slides will guide you through using the GUI.

40. Step 1. Use of Impact GUI for Program FPGA
Appendix
Step 1. Use of Impact GUI for Program FPGA
Impact GUI can be used for programming FPGA. This steps shows how programs FPGA using Impact GUI.
Source Xilinx tools
$ source /opt/Xilinx/13.4/ISE_DS/settings64.sh
Go to NetFPGA-10G directory and run impact
$ impact

41. Step 2. Use of Impact GUI for Program FPGA
Appendix
Step 2. Use of Impact GUI for Program FPGA
Tick ‘create a new project’ and click OK.

42. Step 3. Use of Impact GUI for Program FPGA
Appendix
Step 3. Use of Impact GUI for Program FPGA
Tick ‘Configure devices using Boundary-Scan(JTAG) and click OK

43. Step 4. Use of Impact GUI for Program FPGA
Appendix
Step 4. Use of Impact GUI for Program FPGA
Click Yes

44. Step 5. Use of Impact GUI for Program FPGA
Appendix
Step 5. Use of Impact GUI for Program FPGA
Select a bit file and click OPEN.

45. Step 6. Use of Impact GUI for Program FPGA
Appendix
Step 6. Use of Impact GUI for Program FPGA
Click No.

46. Step 7. Use of Impact GUI for Program FPGA
Appendix
Step 7. Use of Impact GUI for Program FPGA
Click Cancel.

47. Step 8. Use of Impact GUI for Program FPGA
Appendix
Step 8. Use of Impact GUI for Program FPGA
Click Cancel.

48. Step 9. Use of Impact GUI for Program FPGA
Appendix
Step 9. Use of Impact GUI for Program FPGA
Double click ‘Program’
This process ONLY loads the FPGA.

49. Section IV: Wrap-up

50. Thoughts for Developers
Build Modular components
Describe shared registers
Consider how modules would be used in larger systems
Define functionality clearly
Through regression tests
With repeatable results
Disseminate projects
Post open-source code
Document projects on Web, Wiki
Expand the community of developers
Answer questions on the list
Collaborate with your peers to build new applications

51. NetFPGA.org

52. Stuck for a NetFPGA project?
Build an accurate, fast, line-rate NetDummy/nistnet element
A flexible home-grown monitoring card
Evaluate new packet classifiers
(and application classifiers, and other neat network apps….)
Prototype a full line-rate next-generation Ethernet-type
Trying any of Jon Crowcrofts’ ideas (Sourceless IP routing for example)
Demonstrate the wonders of Metarouting in a different implementation (dedicated hardware)
Provable hardware (using a C# implementation and kiwi with NetFPGA as target h/w)
Hardware supporting Virtual Routers
Check that some brave new idea actually works
e.g. Rate Control Protocol (RCP), Multipath TCP,
toolkit for hardware hashing
MOOSE implementation
IP address anonymization
SSL decoding “bump in the wire”
Xen specialist nic
computational co-processor
Distributed computational co-processor
IPv6 anything
IPv6 – IPv4 gateway (6in4, 4in6, 6over4, 4over6, ….)
Netflow v9 reference
PSAMP reference
IPFIX reference
Different driver/buffer interfaces (e.g. PFRING)
or “escalators” (from gridprobe) for faster network monitors
Firewall reference
GPS packet-timestamp things
High-Speed Host Bus Adapter reference implementations
Infiniband
iSCSI
Myranet
Fiber Channel
Smart Disk adapter (presuming a direct-disk interface)
Software Defined Radio (SDR) directly on the FPGA (probably UWB only)
Routing accelerator
Hardware route-reflector
Internet exchange route accelerator
Hardware channel bonding reference implementation
TCP sanitizer
Other protocol sanitizer (applications… UDP DCCP, etc.)
Full and complete Crypto NIC
IPSec endpoint/ VPN appliance
VLAN reference implementation
metarouting implementation
virtual <pick-something>
intelligent proxy
application embargo-er
Layer-4 gateway
h/w gateway for VoIP/SIP/skype
h/w gateway for video conference spaces
security pattern/rules matching
Anti-spoof traceback implementations (e.g. BBN stuff)
IPtv multicast controller
Intelligent IP-enabled device controller (e.g. IP cameras or IP powermeters)
DES breaker
platform for flexible NIC API evaluations
snmp statistics reference implementation
sflow (hp) reference implementation
trajectory sampling (reference implementation)
implementation of zeroconf/netconf configuration language for routers
h/w openflow and (simple) NOX controller in one…
Network RAID (multicast TCP with redundancy)
inline compression
hardware accelorator for TOR
load-balancer
openflow with (netflow, ACL, ….)
reference NAT device
active measurement kit
network discovery tool
passive performance measurement
active sender control (e.g. performance feedback fed to endpoints for control)
Prototype platform for NON-Ethernet or near-Ethernet MACs
Optical LAN (no buffers)
Stuck for a NetFPGA project?
Build an accurate, fast, line-rate NetDummy/nistnet element
A flexible home-grown monitoring card
Evaluate new packet classifiers
(and application classifiers, and other neat network apps….)
Prototype a full line-rate next-generation Ethernet-type
Trying any of Jon Crowcrofts’ ideas (Sourceless IP routing for example)
Demonstrate the wonders of Metarouting in a different implementation (dedicated hardware)
Provable hardware (using a C# implementation and kiwi with NetFPGA as target h/w)
Hardware supporting Virtual Routers
Check that some brave new idea actually works
e.g. Rate Control Protocol (RCP), Multipath TCP,
toolkit for hardware hashing
MOOSE implementation
IP address anonymization
SSL decoding “bump in the wire”
Xen specialist nic
computational co-processor
Distributed computational co-processor
IPv6 anything
IPv6 – IPv4 gateway (6in4, 4in6, 6over4, 4over6, ….)
Netflow v9 reference
PSAMP reference
IPFIX reference
Different driver/buffer interfaces (e.g. PFRING)
or “escalators” (from gridprobe) for faster network monitors
Firewall reference
GPS packet-timestamp things
High-Speed Host Bus Adapter reference implementations
Infiniband
iSCSI
Myranet
Fiber Channel
Smart Disk adapter (presuming a direct-disk interface)
Software Defined Radio (SDR) directly on the FPGA (probably UWB only)
Routing accelerator
Hardware route-reflector
Internet exchange route accelerator
Hardware channel bonding reference implementation
TCP sanitizer
Other protocol sanitizer (applications… UDP DCCP, etc.)
Full and complete Crypto NIC
IPSec endpoint/ VPN appliance
VLAN reference implementation
metarouting implementation
virtual <pick-something>
intelligent proxy
application embargo-er
Layer-4 gateway
h/w gateway for VoIP/SIP/skype
h/w gateway for video conference spaces
security pattern/rules matching
Anti-spoof traceback implementations (e.g. BBN stuff)
IPtv multicast controller
Intelligent IP-enabled device controller (e.g. IP cameras or IP powermeters)
DES breaker
platform for flexible NIC API evaluations
snmp statistics reference implementation
sflow (hp) reference implementation
trajectory sampling (reference implementation)
implementation of zeroconf/netconf configuration language for routers
h/w openflow and (simple) NOX controller in one…
Network RAID (multicast TCP with redundancy)
inline compression
hardware accelorator for TOR
load-balancer
openflow with (netflow, ACL, ….)
reference NAT device
active measurement kit
network discovery tool
passive performance measurement
active sender control (e.g. performance feedback fed to endpoints for control)
Prototype platform for NON-Ethernet or near-Ethernet MACs
Optical LAN (no buffers)
Well I’m not sure about you but here is a list I created:

53. Project Ideas for the NetFPGA
NetFPGA-10G Test Harness
Drop 1-in-N packet module
Rate-limited module
Event capture module
Statistics and Counters
Measurement sketch
Advanced OPL
Input / Output scheduler
40G Port
VLAN Tagging ….
Ideas from NetFPGA-1G
working test and verification harness
drop 1-in-n packet module
event capture module & rate-limiter module
statistics and counters
measurement sketches
input arbiter queue scheduling
VLAN tagging
channel bonding
40Gbps MAC (Noa offers to supervise)

54. Visit http://NetFPGA.org

55. Wednesday restart from 9am
This evening is time to try hw testing & synthesis crypto_nic design if you didn’t manage that
Make your group, leader&project
Specific, Realistic
Time-bounded
Modest is good…..

56. Section IX: Conclusion

57. Acknowledgments
NetFPGA Team at Stanford University (Past and Present):
Nick McKeown, Glen Gibb, Jad Naous, David Erickson,
G. Adam Covington, John W. Lockwood, Jianying Luo, Brandon Heller, Paul Hartke, Neda Beheshti, Sara Bolouki, James Zeng,
Jonathan Ellithorpe, Sachidanandan Sambandan, Eric Lo
NetFPGA Team at University of Cambridge (Past and Present):
Andrew Moore, David Miller, Muhammad Shahbaz, Martin Zadnik
Matthew Grosvenor, Gianni Antichi, Neelakandan Manihatty-Bojan,
Georgina Kalogeridou, Jong Hun Han, Noa Zilberman
All Community members (including but not limited to):
Paul Rodman, Kumar Sanghvi, Wojciech A. Koszek,
Yahsar Ganjali, Martin Labrecque, Jeff Shafer,
Eric Keller , Tatsuya Yabe, Bilal Anwer,
Yashar Ganjali, Martin Labrecque
Kees Vissers, Michaela Blott, Shep Siegel

58. Thanks to our Sponsors:
Support for the NetFPGA project has been provided by the following companies and institutions
Disclaimer: Any opinions, findings, conclusions, or recommendations expressed in these materials do not necessarily reflect the views of the National Science Foundation or of any other sponsors supporting this project.
This effort is also sponsored by the Defense Advanced Research Projects Agency (DARPA) and the Air Force Research Laboratory (AFRL), under contract FA C This material is approved for public release, distribution unlimited. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.