1. Design and Characterization of TMD-MPI Ethernet Bridge Kevin Lam Professor Paul Chow

2. Background In embedded development, Field Programmable Gate Arrays (FPGAs) allow mixing of hard and soft elements Issue: Communication between components MPI is a standard for inter-process communication in software parallel programming TMD-MPI implements a hardware interface for MPI Unified method of communication between all components Abstracts the communication medium

3. Typical Application without TMD-MPI FPGA Embedded CPU Soft Process 1 Hard Process 1 Soft Process 2 Hard Process 2 Processor Bus FIFO

4. TMD-MPI Application FPGA Embedded CPU Hardware Engines

5. Research Objective Attempt to create a module that routes MPI messages through on-board Ethernet hardware Design Objectives: Reliability High Bandwidth Low Latency

6. Multi-Board TMD-MPI system (your desk!)

7. Design Specification Behaviour FSL interface to on-board network (via NetIF module) Transparent inter-board communication via Ethernet FIFO behaviour Assumptions Only two systems connected, by short Ethernet cable Reliable, in-order delivery of packets

8. Design Specification FPGA FSL-Ethernet FSL FSL-Ethernet FSL Logical FSL

9. Design Specification Packet Format Basic Ethernet framing Source/Destination MAC address Type code – set to an unused value CRC checksum performed by Ethernet hardware Length header to account for smaller-than-minimum packets Set to 0 if data is longer than minimum packet length Followed by data payload, up to maximum Ethernet frame length No handshake protocol implemented

10. Implementation Base design around Xilinx echo example First send/receive data independently Test using PC as second ‘board’ Develop FSL interface, test with MicroBlaze Merge FSL interface to Ethernet module Add protocol headers, implement data packing

11. Board 2 Board 1 Testing Initial testing done using MicroBlaze processors High speed testing done using custom cores MicroBlaze 1MicroBlaze 2FSL-Enet UART PC

12. Testing Further testing using dedicated hardware core Testing high speed burst transfer from test core, UART displays all values received by MicroBlaze Board 2 Board 1 MicroBlazeTest CoreFSL-Enet UART PC MicroBlaze GPIO

13. Design Evaluation Reliability testing using two dedicated tester cores Transmitting sequence numbers, checking for mismatch/out of order Board 2 Board 1 Test Core 1Test Core 2FSL-Enet UART PC MicroBlaze GPIO MicroBlaze UART GPIO

14. Design Evaluation Bandwidth Uses 2 dedicated hardware cores Core A writes data to the FSL-Ethernet core whenever the FSL is not full Core B reads data from the FSL-Ethernet core whenever the FSL has data Core A counts the number of clock cycles elapsed to write a certain amount of data to the FSL Result read by MicroBlaze and displayed via UART Measured constant 962.5 Mbit/s data throughput rate

15. Design Evaluation Latency Uses 2 dedicated hardware cores Core A sends 1 word of data to Core B via FSL-Ethernet Core B replies with 1 word Core A measures the clock cycles elapsed between sending its data and receiving the response Result read by MicroBlaze and displayed via UART Several trials yield 598-599 cycle round trip (125 MHz) One-way latency is approximately 300 cycles or 2.4 microseconds

16. Conclusions Bandwidth sufficient for VGA video streaming Latency may be an issue for applications that pass data back and forth frequently No handshaking protocol Receiver in streaming applications must not be slower than the sender Messages cannot be sent until both boards are powered on and connected Future versions should attempt to implement handshaking Affect complexity and bandwidth Much more robust system

17. Acknowledgements Professor Paul Chow Vince Mirian Sami Sadaka Tony Zhou