GSB : A real-time Software back-end for the GMRT Jayanta Roy National Centre for Radio Astrophysics Pune, India 12 th December 2008 Collaborators : Yashwant Gupta (NCRA) Ue-Li Pen (CITA) Jeff Peterson (CMU) Jitendra Kodilkar (NCRA) Sanjay Kudale (NCRA)

Acknowledgments  Analog group of GMRT : Ajitkumar and his group designed a new full bandwidth base-band receiver for GSB.  Computer management group of GMRT : Mangesh Umbarje and Sumit Mirajkar.  Mechanical support : By H S Kale and his group.  Real-time phase solution from Santaji Katore.  Clock and trigger distribution unit from Kevin Bandura (CMU)  Discussion and suggestions from other NCRA staff members : Jayaram Chengalur, Subhashis Roy, Rajaram Nityananda, Vasant Kulkarni

Desired operating modes  32 antennae x 2-pols 32 MHz  real-time software correlator  high time resolution incoherent and coherent array beams formation ( GSB is aimed to provide the specs of existing backend + more )  32 antennae x 2-pols 32 MHz base-band recording by real-time streaming to disk array with off-line read-back of recorded data and computation

Input data rate :  32 antennae x 2-pols base-band analog 16/32 MHz of bandwidth --> 2 GSamples/sec (using 16 ADC cards with 4 analog inputs in each card) Implementation details

Basic Methodology :  Run synchronous sampling on all 16 ADC boards – 16/32 MHz BW  Transfer data from ADC board to CPU unit (via interrupt driven 145 MB/s) in large blocks  For baseband recording mode, synchronous write to disk at each recording node  For correlations, distribute data from all antennas (using time division multiplexing) to all compute nodes -- each node handles 1/N time slice from each block  Carry out FFT, post-FFT fringe stop, Correlation and array beam formation at each compute node  Record integrated visibilities results and beam data to the “collector nodes”

The performance-per-watt capability of the GSB cluster ~ 260 Mflops/watt System Performance  Correlator specs : Spatial resolution = 32kHz – 125kHz Temporal resolution = 1s – 16s Max output data rate = 14 GB/hr  Beamformer : Incoherent/Coherent intensity mode : Temporal resolution = 60 us + Max output data rate = 57 GB/hr Coherent array voltage data : Temporal resolution = nyquist limited Max output data rate = 200 GB/hr  Base-band recorder : Temporal resolution = nyquist limited Max output data rate = 3.5 TB/hr

GSB with ONLINE Control and Monitor interface

GSBMON : GSB real-time data Monitoring

GSB current status 32 antennae x 2-pols 16 MHz  real-time software correlator running regularly  real-time array phasing  formation of high time resolution incoherent and coherent array beam 32 antennae x 1-pols 32 MHz  real-time software correlator being tested 32 antennae x 2-pols 16 MHz 4-bit base-band recording by real-time streaming to disk array with off-line read-back of recorded data and computation

Some results from GSB

Phases from GHBPhases from GSB Preliminary results from the GSB GSB gives ~ 1.5 times better rms than GHB GSB Image of NGC 4631 (made by J. Kodilkar)

GSB image over-plotted with data from Palmor observatory sky survey Image made by J. Kodilkar

Single pulses time series of PSR B GSB pulsar observations Pre detection dispersion removal “Presto” folded profile of B

Comparative study of GSB and DiFx pipeline Input is same baseband recorded data from GSB baseband recorder GSBC and DiFx produce similar results, but GSBC runs significantly faster than DiFx ! Auto spectra Cross spectra Spectral variation of RMS DiFx is Swinburne’s VLBI software correlator (Deller et al)

RFI removal Enhanced Performance from GSB : Pre and post detection RFI removal Power-line RFI removal  Time domain flagging on raw voltage samples  Spectral domain filtering of powerline harmonics in the visibility data Time domain RFI removal of power-line and any impulsive RFI  Median absolute deviation (MAD) based estimator

Summary  Real-time GSB is a highly optimized multi-threaded vectorized parallel pipeline, working ~ 90% of the theoretical peak FLOPS.  GSB is a fully real-time software back-end: First instance of such a real-time back-end for an intermediate size multi-element interferometer array like GMRT.  GSB is going to be the 24 x 7 default observatory back-end.  Radio frequency Interference: Severe at low frequencies; GSB processing algorithms are demonstrated to handle these issues.   GSB development cycle is only of 3 yrs.

Next steps  High BW real-time software back-end : GSB is easily reconfigurable Implement 4-bit 200 MHz BW 8 antennae back-end. Implement 4-bit 100 MHz BW 16 antennae back-end  GPU based high intensive pulsar processing : coherent dedispersion in real-time and RFI removal in real-time  Formation of Multiple phased array beams within the primary beam  24 x 7 piggy-back fast transient survey pipeline on GSB Software based processing likely to be a future growth path of radio astronomy signal processing : GSB can be a test bench for that

Multiple phased array beam formation

FFT cost/ node ~ 5 GFLOPS Beam-forming (# 400) cost/ node ~ 160 GFLOPS 1.6 x of real-time on 72 nodes itanium cluster Compute requirements : Real-time space

GMRT Multibeaming ……  Multi-element arrays can shape the beams to produce nulls in directions of known, spatially fixed sources of RFI spatially fixed sources of RFI  Signals from multiple beams (e.g. one beam on-target, another off-target) can be useful for implementing adaptive RFI cancellation techniques useful for implementing adaptive RFI cancellation techniques  Desirable to have a on-line calibration by having one beam continuously on the in-field calibrator in-field calibrator  Sensitivity and computational load balancing : need to synthesize ~ 500 multiple phased array beams, pointing in different directions within primary beam. phased array beams, pointing in different directions within primary beam. synthesize partially dephased fatter beams of lower sensitivity : reducing the synthesize partially dephased fatter beams of lower sensitivity : reducing the number of beams number of beams

24 x 7 Transient search using GSB : Study of dynamic radio sky Study of dynamic radio sky –Collaborative effort of NCRA and Swinburne University of Technology –Running in piggy-back mode simultaneously with other observations –Object of interest : Fast transients : nanosecond to 100's of millisecond –Event detection : based on the sensitivity of 8 antennae incoherent array beam over 32 MHz Phase space of the survey

Coincidence detections in all 4 incoherent beams : Discrimination real signals from RFI

 Search in Dispersion measure space : Discriminate out fast radio transient from RFI.  Coincidence or anti-coincidence filter : Multiple sub-array multiple beam coincidence filter reduces the false triggers due to direction-location dependent RFI.  Efficient RFI removal technique : MAD based technique (as described before.....) ….. Discrimination of real signals from RFI

24 x 7 Transient search pipeline

Summary  GSB + Itanium cluster with its compute and storage capability will open up the phase space of GMRT to enhance its productivity.  GSB is going to be the 24 x 7 default observatory back-end.   Total GSB development cycle (from first idea to complete realisation) has been only 3 yrs.

Thank you