1. ATCA GPU Correlator Strawman Design ASTRONOMY AND SPACE SCIENCE Chris Phillips | LBA Lead Scientist 17 November 2015

2. ATCA Current CABB Backend 6 antenna 8 GHz analog IF, dual pol 2x2 GHz IF, dual pol 11 bit samplers, FPGA backend 2048 channel/1 MHz continuum mode Various spectral line modes with “Zoom Bands” –E.g. 4 MHz/2048 channels, 64 MHz/2048 channels Many designed modes never implemented Hardware unreliable and difficult to change modes No full bandwidth tied array mode DIFX 2015 – Chris Phillips 2 |

3. GPU replacement? Could we improve flexibility, reliability and long term support with GPU backend New samplers, which talk directly to optical fibre Use ASKAP “redback” boards for coarse filterbank (128 MHz) and Ethernet packetisation (10 GbE) Fine filterbank and cross correlation in GPU - Ethernet cross connect DIFX 2015 – Chris Phillips 3 |

4. DIFX 2015 – Chris Phillips 4 |

5. Samplers 12 bit Texas Instrument ADC12J4000 Transport JESD204B No FPGA required at sampler, optical interface 8 lanes @ 8 Gb/s DIFX 2015 – Chris Phillips 5 |

6. Redback ASKAP Beamformer/correlator board 6 Xilinx Kintex-420 Series-7 FPGAs 1U chassis 36x10 Gbps Ethernet output 2x2 GHz IF per redback 2 redback/telescope Coarse filterbank – 128 MHz Need to divide data 16 ways Re-quantize to 8 bits to reduce i/o load 16 bits would be better, but doubles backend cost 12 bit may have minimal cost overhead DIFX 2015 – Chris Phillips 6 |

7. Switch Commodity 40 Gbps Ethernet switch 64 port 40 GbE ~$30K Can run 4x10 GbE per 40 GbE port 8 bit system requires 56x40 GbE ports DIFX 2015 – Chris Phillips 7 |

8. GPU backend Need to frequency slice data at least 12 ways to avoid bottleneck on ingest (6 antenna, 2 pols) Use factor of 16 16 GPUs per IF, dedicated 40 GbE Ethernet per GPU 2 GPU/host plus 2 40 GbE NICs Don’t implement zoom bands – filter data to highest required spectral resolution then frequency average as appropriate DIFX 2015 – Chris Phillips 8 |

9. Costs: Assumptions ($A) SPF Copper$40 SPF Optical$200 Samplers$5,000 Redback$12,000 GPU$1,000 NIC$800 Server$8,000 Switch$30,000 DIFX 2015 – Chris Phillips 9 |

10. Costs: Total DIFX 2015 – Chris Phillips 10 | Antenna6 Samplers24$120,000 Redback12$144,000 SFP Copper192$7,680 SPF Optical64$2,560 Server16$128,000 GPU32$32,000 NIC32$25,600 Switch1$30,000 Total$489,840

11. Supported Modes Assuming GPUs have enough computational power basic interferometry modes relatively easy to implement High spectral resolution, short integration times Tied array – multiple simultaneous beams –i/o issues (but new GPU 2 copy engines and NIC Tx relatively empty) Pulsar binning Need to also extract noisecal DIFX 2015 – Chris Phillips 11 |

12. Exotic Modes Pulsar Coherence de-dispersion or binning Just requires extra compute Fast radio bursts – serendipitous Requires ~500 kHz spectral resolution, 64usec time resolution 246 Gbps visibility output (8 Gbps/node) –Not really viable, need to detect on 128 MHz data –Longer integration, lower spectral resolution? Transient buffer mode, external triggers 3 Gbytes/sec incoming rate Only buffer a few seconds – not viable without major cost (RAM) DIFX 2015 – Chris Phillips 12 |

13. Exotic Modes (cont) Nanosecond pulse detection (Lunaska) 2 GHz bandwidth gives 0.25 nsec sampling time But need full bandwidth in one location Could received 128 MHz channels on servers then recirculate in round-robin fashion. Not enough bandwidth to receive second copy of data without second NIC and more switching capability (dedicated infiniband?) Changed redback mode to buffer data, then round-robin full 2 GHz data to GPUs – need special redback FPGA firmware De-disperse antenna data, form tied array beams and look for pulses Can dump voltage data if candidate found DIFX 2015 – Chris Phillips 13 |

14. Astronomy and Space Science Dr Chris Phillips LBA Lead Scientist t+61 2 93724608 eChris.Phillips@csiro.au wwww.atnf.csiro.au ASTRONOMY AND SPACE SCIENCE Thank you