Fast A/D sampler FINAL presentation Presented By: Tal Goihman, Irit Kaufman Instructor: Mony Orbach Winter 2012

Goals Project Goal: Design and implement an A/D system using Xilinx Virtex6 development board for sampling at highest possible rate. Sample to virtex6 development board DDR memory. Transfer the sampled data to PC memory through PCIe and save the data to disk.

PC H/W Block Diagram ML605 development board FMC125 Fast A/D FMC Conn PCI-E Connector FMC Conn Virtex6 FPGA ML605 development board DDR3 FMC125 Fast A/D A/D IC PC

ML605 development board

FMC125 Fast A/D

A/D Sampler The FMC125 is a Quad-Channel ADC that provides four 8-bit ADC channels enabling simultaneous sampling of 1, 2, or 4 channels @ 5 , 2.5 , 1.25Gsps respectively. Problems: 4DSP provides free of charge a reference design only for 4ch @ 1.25Gsps. A reference design for 1ch @ 5Gsps priced at 2300 EU.

4DSP reference design

Design Block Diagram “wormhole” AXI4 Implemented In XPS MicroBlaze DDR3 Memory Controller (MIG) AXI Master UART ADC Control FMC125 Aggregator CDMA PCI Express AXI Slave Mailbox Timer In ISE project Navigator “wormhole” AXI4-lite AXI4

Block Diagram: Main Data Channel FMC125 Aggregator AXI Master DDR3 Memory Controller (MIG) AXI Slave FMC125 delivers data using 4 128-bit lanes of a proprietary bus running at 125Mhz into the aggregator. Aggregator unites and synchronizes the different channels into a 512-bit bus. AXI Master sends the data over a 256-bit wide AXI bus running at 200Mhz to the AXI Slave interface of the Memory Controller. Memory controller handles Writes to DDR3. FIFO’s and H/W Flow control in every component’s input (and some components output) throughout the channel to achieve highest possible bandwidth. Throughput is measured using a timer from the start of the write operation until assertion of a write done signal from the AXI master 16 samples per clock per channel Total 16*4*125Mhz=8GB/s Aggregator synchronizes samples so that all channel’s sample 0 will come together Aggregator works at 125Mhz, has a 4bit channel enable input AXI master throughpuf of 256bit/8*200Mhz=6.25GB/s

AXI Bus background Xilinx has adopted AXI bus, which is a standard bus protocol from ARM used in modern ARM SoC. Characteristics Memory mapped, 32-bit addresses Write Address, write data, write response, read address, read data Variable width, clock & burst length over a single bus.

AXI write example

Memory & Memory controller 512MB DDR3 64-bit @ 400Mhz (800MT/s) Theoretical bandwidth of 800MT/s * 64bit / 8 = 6.25GB/s Main channel matched to this theoretical bandwidth (200Mhz * 256bit /8 = 6.25GB/s) Memory & controller isn’t perfect, has a utilization factor. We achieved 5.13GB/s Throughput (82% utilization!)

Design Block Diagram “wormhole” AXI4 Implemented In XPS MicroBlaze DDR3 Memory Controller (MIG) AXI Master UART ADC Control FMC125 Aggregator CDMA PCI Express AXI Slave Mailbox Timer In ISE project Navigator “wormhole” AXI4-lite AXI4

Block Diagram: memory to PCIe After the data is sampled to memory the system transfers the first chunk to the Host PC’s DMA buffer. The subsequent chunks are transferred upon receiving a command from the Host PC. Data is read from memory and transferred to PCIe by the DMA engine Throughput was matched to PCIe Transferred in 16MB chunks (the chosen DMA buffer size) Address translation occurs in the PCIe core Talk about address translation and the issues we had with it

Block Diagram: PCIe Xilinx Available solutions include: A PCIe integrated block with support for up to x8 gen2. Hard to work with: Need to know inner workings of PCIe Need to implement several proprietary interfaces with many rules and signals A wrapper for memory mapped AXI with support for up to x4 gen1 / x2 gen 2 Connects to a standard AXI4 bus Chosen in our design due to high ROI We chose to use the wrapper due to high ROI x4 gen1 configuration, up to 1GB/s One 64bit AXI BAR for sample data transfer with Configurable address translation, one PCIe BAR for mailbox communication.

Block Diagram: MicroBlaze MicroBlaze is a soft-core processor by Xilinx. Runs firmware written in C from a dedicated 64KB BRAM. Firmware communicates with Host PC’s software through a PCIe mailbox Supports a predefined set of commands to accomplish the functional use case and provide debug capabilities Initializes and configures the entire system per to the Host PC’s instructions. Provide visual status and information to the user through UART massages.

Block Diagram: ADC ctrl ADC ctrl is a custom core deigned to enable MicroBlaze to communicate with other custom cores Developed using the Create peripharel wizrd in XPS and implements mamory mapped register access through AXI4-lite Provides the following capabilities: Translate the wormhole transactions to / from the FMC125 core. Provide AXI Master with burst count, write start, test signals Provides the aggregator with enabled ADC channels

Software Written in C# .NET 4 in VS2012, GUI in WPF Enables simple interaction with the system User adjustable burst size, burst count, active channels, output file location and external trigger mode compatible with Windows 7/8 x64.

Software: Jungo WinDriver Easy driver creation Simple interface for DMA buffer allocation, read and write operations Slightly problematic from a .NET environment since it lacks proper documentation, and the wizard generates a non working solution Reference for .NET and DMA was found in a bundled example for PLX chips

Test Environment Verification of sampled data on DDR3 is accomplished by firmware on MicroBlaze comparing read data with predetermined written pattern FMC125 incoming data is observable through ChipScope. Bandwidth was calculated by measuring the time to write a chunk of data using a Timer. Data transfer from memory to PCIe is validated through observing the memory on system and comparing with the data received from PCIe manually.

Questions? Thanks

Backup Documantaion