1. Digital FX Correlator Nimish Sane Center for Solar-Terrestrial Research New Jersey Institute of Technology, Newark, NJ EOVSA Technical Design Meeting

2. Overview Nimish Sane, New Jersey Institute of Technology 2 No. of antennas16 No. of polarizations2 No. of frequency channels (subbands)4096 Integration time (ms)20 (possibly, tunable) IF (MHz)600 ADCF-EngineX-Engine P, P 2 Calculation

3. Hardware Roach-2 board [1] – Virtex-6 SX475T FPGA (XC6VSX475T-1FFG1759C) – PowerPC 440EPx stand-alone processor to provide control functions – 2 x Multi-gigabit transceiver break out card slots, supporting up to 8x10Ge links which may be CX4 or SFP+ – 4 x 36 * 2M QDR II+ SRAMs connected to the FPGA – A single 72-bit DDR3 RDIMM slot connected to the FPGA – 2 x ZDOKs – An FTDI FT4232H USB to JTAG, serial and IIC 8 boards with 2 antennas (dual-polarization) per board Nimish Sane, New Jersey Institute of Technology 3

4. ADC KAT ADC ASIAA ADC – 1 signal at 5 GSamples/sec (GS/s) – 2 signals at 2.5 GS/s – Used at CFA (Jonathan Weintroub, Rurik Primiani) 2 ADCs on the same board may cause cross- coupling issues. Nimish Sane, New Jersey Institute of Technology 4 Comments from discussion with Dan Werthimer and David MacMahon

5. F-Engine No. of F-engines per Roach board = 4 (2 antennas dual polarization) Output of an F-engine: Complex (even and odd channels), each with 18-bit real and 18-bit imaginary. Output data of F-engines per Roach board per FPGA clock cyle = 72 x 4 bits Total data rate of F-engines per Roach board = 7.2 Gbps (assuming FPGA clock period of 4 ns) Nimish Sane, New Jersey Institute of Technology 5 F – engine : 4096 Channel Fix 8_7 Ufix 36 even odd

6. X-Engine Each Roach board will have 1 X-unit Each X-unit will handle 4096/8 = 512 frequency channels (256 even and 256 odd channels) No. of complex multipliers in each X-unit correlator block = No. of visibilities x No. of polarizations x Simultaneous even and odd channels per F-unit = 120 x 2 x 2 = 480 X-unit output data per integration = 480 x 2 x 256 x 4 bits = 983040 bits Total data rate at the output of X-unit per Roach board = 983040 x 50 = 49.152 Mbps (assuming integration time of 20 ms) Nimish Sane, New Jersey Institute of Technology 6 Correlation Complex Multipliers = 480 Scaling and Quantization Vector Accumulation X – Unit UFix 36 Fix 4_3 (?) (32)(960)

7. F-engine Comments Coarse delay on FPGA Fine delay off-line Keep P, P 2 calculations separate from X-engine Phase switching is difficult on FPGA. Dan suggests doing it before ADC, and then undoing it on FPGA ATA Memo on fringe stopping after FFT GMRT does fringe stopping + coarse delay + fine delay + (possibly) phase switching, but not at 600 MHz Nimish Sane, New Jersey Institute of Technology 7

8. F and X-engine Connections Various architectures have been proposed in [2]. Current state of art: – F and X engines on different boards – F + X on the same board: (1) Can we fit the design? (2) Can F + X work in tandem? – Use full-duplex bidirectional capacity of 10 GbE link: Send output of F – engine to a switch that will distribute it to X – engines (even if F and X are on the same board) Nimish Sane, New Jersey Institute of Technology 8

9. X-engine Comments If CPUs are going to be used anyways, GPUs may be targets for X-engine No GPU correlator has been deployed yet. (PAPER and LEDA in progress). X-engine on FPGA is more straight-forward. Nimish Sane, New Jersey Institute of Technology 9

10. Other Issues How to perform 4-bit correlation? – Ensuring that data is well-distributed over 16 levels – How to use block RAM (BRAM) blocks for scaling each frequency channel? How to determine the gain values for each frequency channel? Synchronization with external clock Nimish Sane, New Jersey Institute of Technology 10

11. Nimish Sane, New Jersey Institute of Technology 11 References 1. https://casper.berkeley.edu/wiki/ROACH2https://casper.berkeley.edu/wiki/ROACH2 2. P. McMahon, et al. “CASPER Memo 017: Packetized FX Correlator Architectures,” September 2007.