xNTD/SKAMP/LFD Correlator 4th RadioNet Engineering Forum Workshop Next Generation Correlators for Radio Astronomy and Geodesy June 2006, Groningen, The Netherlands, John Bunton CSIRO ICT Centre, Sydney

New Generation Correlators Groningen June 2006 Common Design  Developing a set of technologies that will be used in xNTD, SKAMP3 and LFD  Common correlator, VHDL code and hardware elements  This talk mainly the correlator and xNTD beamformer  Team (SKAMP3, xNTD and LFD)  Uni Sydney/ ATNF - Ludi de Souza,  Uni Sydney Duncan Campbell-Wilson, John Russel, Chris Weimann, Adrian Blake  ATNF/ ICT Centre John Bunton, Jaysri Joseph  MIT Roger Cappallo, Brian Fanous  ANU Frank Briggs  and growing

New Generation Correlators Groningen June 2006 xNTD specification  Connected element telescope with antennas within ~10km area  Each antenna generates 32 dual pol. beams giving a field of view of ~30 square degrees – frequency independent  Dynamic range ~50dB (8-bit A/D) aided by low RFI environment at Mileura WA, >60dB filterbanks  Frequency range below 1.7 GHz with 300 MHz BW

New Generation Correlators Groningen June 2006 xNTD Correlator  FX correlator, FPGA based using polyphase filterbank  Large design with antennas x ~32 beams  Input to beamformers 1.5 THz, to correlator 500 GHz (16bits/Hz)  6,000-15,000 baselines, each full Stokes  ~1000 Tops/sec (mainly beamformer/filterbanks)  Output data rate 1-2 Gcorrelation/visability set  Full 300 MHz processed to full frequency resolution  ~50,000 channel – 6 kHz resolution (no options)  BUT  Low resolution and continuum by averaging channels  Daisy chain data for special processing  Later can trade bandwidth for higher resolution or time sampling

New Generation Correlators Groningen June 2006 Design  Beamformers and filterbank  ~200 inputs generating 32 dual pol. beams  Signal transport from feeds optical analogue (TBD)  Beamformer location TBD  Design similar to SKAMP3 filterbank (Ludi’s talk)  Correlator based on correlation cell  Highly configurable as to number inputs  If the beamformer is at the antenna crossconnect in fibre network and O/E interface  Needs data re-ordering (Ludi’s talk)

New Generation Correlators Groningen June 2006 xNTD Beamformer  Beamformer input  96 Dual polarisation inputs  300 MHz bandwidth  About 1 Terabit/sec input data rate per antenna  Each of 32 output beams is the sum of 20 to 96 inputs depending on frequency  Would like to have data from all inputs to each unit that generates a beam  Instead “Divide and Conquer”  Use the same topology as SKAMP3 filterbanks (Ludi’s talk)

New Generation Correlators Groningen June 2006 Filterbank to Beamformer

New Generation Correlators Groningen June 2006 Correlator  The challenge was to develop a correlator concept the could be used in xNTD, SKAMP3 and LFD , 192, and 512 antennas  Systolic array too inflexible  xNTD 760 correlations/beam  LFD 500,000 correlations  Approach developed – Correlation Cell  Combination of multiply-accumulate and storage  Each cell handles 256 correlations at a time  49,000 correlations per FPGA simultaneously  512 time sample short integration on chip

New Generation Correlators Groningen June 2006 Correlation Cell  Input 16+ pairs of data  4bit complex multiply in 18-bit multiplier  Accumulation to block RAM  Calculate 256 correlation, 512 successive time samples  Data reordering in filterbank  xNTD 4-7 cells for all correlations  MHz BW per FPGA  All baselines LFD 12, SKAMP3 1.5 FPGAs  MHz of bandwidth

New Generation Correlators Groningen June antenna xNTD configuration  Store 22 or 22 pairs per correlation cells  Within group 231 correlation  Between groups 484 correlations  Need two cells  48 sets of 22-antenna correlators per FPGA (Virtex4 SX35)  Clock cycles 242 for 44 inputs = 5.5 clocks per input  Input data rate 48/5.5 = 8.7 bytes/clock  As systolic array 192 multipliers gives 13x13 array  Input data rate 3 times higher at 26 bytes/clock

New Generation Correlators Groningen June 2006 Board Manufacture Simplification (1)  Manufacture of correlator board a major task  Examples SKAMP1, EVLA …  Correlation cell reduces input data rate into correlation chip  For xNTD correlation cell data rate 3 times less compared to systolic array  For individual correlation cell 2 sets of 16 inputs requires 256 clock cycles to process.  Data rate reduction up to a factor of 16  This value approached for SKAMP3 and LFD  For xNTD need ~5 byte wide busses into each correlator FPGA. (input data rate higher than correlator clock)

New Generation Correlators Groningen June 2006 Board Manufacture Simplification (2)  Correlation cell also reduces data duplication  SKAMP1 4x4 systolic array, EVLA 8x8 systolic array  Data duplication 8 in EVLA, higher in SKAMP1 due to array reuse  Each Correlation Cell process 256 correlations at once  Can reduce size of systolic array by sqrt(256)=16  No data duplication on board for up to 150 antennas  Data duplication none for xNTD, 1.5 for SKAMP3, and 3.5 for LFD, LFD 12 FPGAs data input 1/sqrt(12) of total  Correlation cell leads to a large simplification of correlator board

New Generation Correlators Groningen June 2006 Putting it Together – The SKAMP protoype Routing Autocorrelations Long Term Accumulations Correlations Correlator interface

New Generation Correlators Groningen June 2006 Input, Daisy Chain, Route, & Autocorrelate  Two FX20  Interconnection for high antenna number designs  Input 16 rocket I/O on unidirectional Infinband  Output 16 Rocket I/O unidirectional Infiniband  Can daisy chain modules for reuse of data in further processing modules or conform to standards, Infiniband … Two FX20

New Generation Correlators Groningen June 2006 Compute Engine  Six SX35 FPGAs  Input 24, Output 18 LVDS per FPGA  1152 Correlation cells total  Up to 294k correlations on board at a time (256 per cell)  Data re-ordering in filterbank to achieve  Process of 512 time values for each frequency channel  Then dump to LTA

New Generation Correlators Groningen June 2006 Long Term Accumulator  Number of Correlation require DRAM for storage  Data rate requires two DIMM modules for three SX35 Half of Correlation and LTA hardware Three SX35 Two LX15 + DDR2 DIMM

New Generation Correlators Groningen June 2006 Estimated Performance  Correlator board clock rate 330MHz, 192 cells/FPGA, 6 FPGAs  Board processing rate 400GCMACs/s = 2.8Tops/s  Power consumption ~100W  Power efficiency 0.25W/GCMAC (4bit FX)  Filterbank board 3.2Gsample/s, two polyphase filterbanks ~32 operations per sample  Board processing rate 100Gops/s (18 bit)  Power consumption ~60W  Power efficiency 0.6W/Gop

New Generation Correlators Groningen June 2006 Conclusion  Common hardware, hardware modules and VHDL for xNTD, SKAMP3 and LFD  SKAMP3 in the lead with filterbank and correlator hardware well on the way  Initial Manufacture this year  Correlator common to all using correlation cell to gain required flexibility  xNTD beamformer hardware starts July 06  LFD receiver will piggy back on this  Developing international project - distributed design team