1. xNTD/SKAMP/LFD Correlator 4th RadioNet Engineering Forum Workshop Next Generation Correlators for Radio Astronomy and Geodesy 27-29 June 2006, Groningen, The Netherlands, John Bunton CSIRO ICT Centre, Sydney

2. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 Common Design  Developing a set of technologies that will be used in xNTD, SKAMP3 and LFD  Common correlator, VHDL code and hardware elements  This talk mainly the correlator and xNTD beamformer  Team (SKAMP3, xNTD and LFD)  Uni Sydney/ ATNF - Ludi de Souza,  Uni Sydney Duncan Campbell-Wilson, John Russel, Chris Weimann, Adrian Blake  ATNF/ ICT Centre John Bunton, Jaysri Joseph  MIT Roger Cappallo, Brian Fanous  ANU Frank Briggs  and growing

3. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 xNTD specification  Connected element telescope with 20-30 antennas within ~10km area  Each antenna generates 32 dual pol. beams giving a field of view of ~30 square degrees – frequency independent  Dynamic range ~50dB (8-bit A/D) aided by low RFI environment at Mileura WA, >60dB filterbanks  Frequency range below 1.7 GHz with 300 MHz BW

4. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 xNTD Correlator  FX correlator, FPGA based using polyphase filterbank  Large design with 20-40 antennas x ~32 beams  Input to beamformers 1.5 THz, to correlator 500 GHz (16bits/Hz)  6,000-15,000 baselines, each full Stokes  ~1000 Tops/sec (mainly beamformer/filterbanks)  Output data rate 1-2 Gcorrelation/visability set  Full 300 MHz processed to full frequency resolution  ~50,000 channel – 6 kHz resolution (no options)  BUT  Low resolution and continuum by averaging channels  Daisy chain data for special processing  Later can trade bandwidth for higher resolution or time sampling

5. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 Design  Beamformers and filterbank  ~200 inputs generating 32 dual pol. beams  Signal transport from feeds optical analogue (TBD)  Beamformer location TBD  Design similar to SKAMP3 filterbank (Ludi’s talk)  Correlator based on correlation cell  Highly configurable as to number inputs  If the beamformer is at the antenna crossconnect in fibre network and O/E interface  Needs data re-ordering (Ludi’s talk)

6. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 xNTD Beamformer  Beamformer input  96 Dual polarisation inputs  300 MHz bandwidth  About 1 Terabit/sec input data rate per antenna  Each of 32 output beams is the sum of 20 to 96 inputs depending on frequency  Would like to have data from all inputs to each unit that generates a beam  Instead “Divide and Conquer”  Use the same topology as SKAMP3 filterbanks (Ludi’s talk)

7. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 Filterbank to Beamformer

8. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 Correlator  The challenge was to develop a correlator concept the could be used in xNTD, SKAMP3 and LFD - 20-30, 192, and 512 antennas  Systolic array too inflexible  xNTD 760 correlations/beam  LFD 500,000 correlations  Approach developed – Correlation Cell  Combination of multiply-accumulate and storage  Each cell handles 256 correlations at a time  49,000 correlations per FPGA simultaneously  512 time sample short integration on chip

9. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 Correlation Cell  Input 16+ pairs of data  4bit complex multiply in 18-bit multiplier  Accumulation to block RAM  Calculate 256 correlation, 512 successive time samples  Data reordering in filterbank  xNTD 4-7 cells for all correlations  30-70 MHz BW per FPGA  All baselines LFD 12, SKAMP3 1.5 FPGAs  1.2-1.5 MHz of bandwidth

10. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 22 antenna xNTD configuration  Store 22 or 22 pairs per correlation cells  Within group 231 correlation  Between groups 484 correlations  Need two cells  48 sets of 22-antenna correlators per FPGA (Virtex4 SX35)  Clock cycles 242 for 44 inputs = 5.5 clocks per input  Input data rate 48/5.5 = 8.7 bytes/clock  As systolic array 192 multipliers gives 13x13 array  Input data rate 3 times higher at 26 bytes/clock

11. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 Board Manufacture Simplification (1)  Manufacture of correlator board a major task  Examples SKAMP1, EVLA …  Correlation cell reduces input data rate into correlation chip  For xNTD correlation cell data rate 3 times less compared to systolic array  For individual correlation cell 2 sets of 16 inputs requires 256 clock cycles to process.  Data rate reduction up to a factor of 16  This value approached for SKAMP3 and LFD  For xNTD need ~5 byte wide busses into each correlator FPGA. (input data rate higher than correlator clock)

12. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 Board Manufacture Simplification (2)  Correlation cell also reduces data duplication  SKAMP1 4x4 systolic array, EVLA 8x8 systolic array  Data duplication 8 in EVLA, higher in SKAMP1 due to array reuse  Each Correlation Cell process 256 correlations at once  Can reduce size of systolic array by sqrt(256)=16  No data duplication on board for up to 150 antennas  Data duplication none for xNTD, 1.5 for SKAMP3, and 3.5 for LFD, LFD 12 FPGAs data input 1/sqrt(12) of total  Correlation cell leads to a large simplification of correlator board

13. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 Putting it Together – The SKAMP protoype Routing Autocorrelations Long Term Accumulations Correlations Correlator interface

14. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 Input, Daisy Chain, Route, & Autocorrelate  Two FX20  Interconnection for high antenna number designs  Input 16 rocket I/O on unidirectional Infinband  Output 16 Rocket I/O unidirectional Infiniband  Can daisy chain modules for reuse of data in further processing modules or conform to standards, Infiniband … Two FX20

15. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 Compute Engine  Six SX35 FPGAs  Input 24, Output 18 LVDS per FPGA  1152 Correlation cells total  Up to 294k correlations on board at a time (256 per cell)  Data re-ordering in filterbank to achieve  Process of 512 time values for each frequency channel  Then dump to LTA

16. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 Long Term Accumulator  Number of Correlation require DRAM for storage  Data rate requires two DIMM modules for three SX35 Half of Correlation and LTA hardware Three SX35 Two LX15 + DDR2 DIMM

17. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 Estimated Performance  Correlator board clock rate 330MHz, 192 cells/FPGA, 6 FPGAs  Board processing rate 400GCMACs/s = 2.8Tops/s  Power consumption ~100W  Power efficiency 0.25W/GCMAC (4bit FX)  Filterbank board 3.2Gsample/s, two polyphase filterbanks ~32 operations per sample  Board processing rate 100Gops/s (18 bit)  Power consumption ~60W  Power efficiency 0.6W/Gop

18. www.ict.csiro.au New Generation Correlators Groningen 26-29 June 2006 Conclusion  Common hardware, hardware modules and VHDL for xNTD, SKAMP3 and LFD  SKAMP3 in the lead with filterbank and correlator hardware well on the way  Initial Manufacture this year  Correlator common to all using correlation cell to gain required flexibility  xNTD beamformer hardware starts July 06  LFD receiver will piggy back on this  Developing international project - distributed design team