E-field Parallel Imaging Correlator: An Ultra-efficient Architecture for Modern Radio Interferometers Nithyanandan Thyagarajan (ASU, Tempe) Adam P. Beardsley (ASU, Tempe) Judd Bowman (ASU, Tempe) Miguel Morales (UW, Seattle)

Outline Motivations for Direct Imaging Modular Optimal Frequency Fourier Imaging (MOFF) – A generic direct imaging algorithm EPIC implementation of MOFF in software EPIC imaging in action Imaging performance of EPIC vs. FX EPIC calibration in action (Adam Beardsley) EPIC on future large-N dense array layouts Time-domain capability of EPIC Testing GPU-based EPIC on HERA

Motivations for Direct Imaging Technological Scientific EoR studies Transient studies / Ionospheric monitoring Large collecting areas require large-N arrays Cost of the correlator scales as N2 Require fast writeouts Fast writeouts under technical Thornton et al. (2013)

Concept of Direct Imaging Antennas placed on a grid and perform spatial FFT of antenna voltages on grid to get complex voltage images Square the transformed complex voltage image to obtain real-valued intensity images Current implementation: 8x8 array in Japan (Daishido et al. 2000) 4x8 BEST-2 array at Radiotelescopi de Medicina, Italy (Foster et al. 2014)

Need for generic direct imaging Hurdles with current implementations MOFF algorithm Morales (2011) Uniformly arranged arrays have poor point spread functions – thus not ideal for imaging Aliasing of objects from outside field of view Assumptions of identical antennas => poor calibration Calibration still requires antenna correlations Antennas need not be on a grid but still exploit FFT efficiency Can customize to science needs Accounts for non-identical antennas Calibration does not require forming visibilities Can handle complex imaging issues - w-projection, time-dependent wide-field refractions and scintillations Optimal images

Mathematical basis for MOFF Measured visibility is the spatial correlation of measured antenna E-fields Antenna power pattern is the correlation of individual voltage patterns Visibility measurement equation is separable into antenna measurement equations Allows application of “multiplication route” in multiplication-convolution theorem of Fourier Transform (while visibility imaging uses “convolution route”) FFT efficiency leveraged by gridding E-fields using antenna voltage illumination pattern

EPIC implementation of MOFF imaging FX MOFF Parallelization for efficiency – not production-level

Modular EPIC software

EPIC Software Summary Object Oriented Python codes, emulate real-life telescope arrays Parallelized for efficiency Implements generic antenna layouts Accounts for non-identical antenna shapes Calibrates using only measured antenna voltages Contains E-field simulator and FX/XF-based imaging pipelines for reference Working towards production-level

Imaging with EPIC vs. FX Simulated Example: Nchan = 16 df = 100 kHz f0 = 150 MHz MWA core layout inside 150 m (51 antennas) Square antenna kernels

Imaging with EPIC vs. FX (zero spacing)

EPIC on actual LWA Data LWA1 TBN data with a total of 2s and 100 kHz Image obtained with 20 ms, 80 kHz Cyg A and Cas A prominently visible

Calibration *Need to apply calibration before gridding.

Calibration Problem E-field incident on ground related to sky by FT: Antenna integrates with voltage pattern: Further corruption modeled as complex gain and noise Goal is to solve for the gain factor

Calibration Requirements No visibilities MOFF algorithm mixes antenna signals ⇒ Must apply calibration at front end Scale <

Simple Case Single point source, centered on sky From Morales (2011): Sufficient to determine self-cal solutions

Calibration Loop An iterative approach Scales O(N)

Simple Case Simulate using EPIC Start with random “guess” for gains Use MWA core layout / beam pattern No noise for now Start with random “guess” for gains Attempt to recover “true” gains 40 kHz, 4 channels

Simple Case Gain Amplitudes Gain Phase Errors Each iteration consists of 20 finer iterations. Damping factor 0.35 (memory from previous gain solution), 0.65 from current solution

Simple Case Image Before Calibration Image After Calibration

Generalize… … Inspect this product Arbitrary sky, any direction Prime means current estimate and tracks the actual gain Arbitrary sky, any direction

Simulate more sources Gain Amplitudes Gain Phase Errors Each iteration = 400 timestamps totaling 10 ms, No receiver noise, only sky noise Remind about phase wrap Initial dip in amplitude is overcompensating for initial high gains

Simulate More Sources

Calibrating LWA data Same LWA data 150 channels ~ 29 kHz bandwidth Cal loop on 51.2 ms cadence total 1.5 s (30 cal. iterations) Model Cyg A and Cas A as point sources Need to handle auto correlation terms If you use the new plots from Evernote, 10 timestamps per cal update, 30 cal updates, total of 1.5s Get auto-subtraction term from Adam’s paper on github

Calibrating LWA data Gain Amplitudes Gain Phases Real data implies phases and gains converge to different values but are locked in

Calibrating LWA data Image Before Image After Reference Calibration Comment on image quality Image Before Calibration Image After Calibration Reference Image 55% increase in dynamic range

Further directions Sophisticated fits Direction dependence Self-cal loop: use images to update model and have a iterative feedback loop Sophisticated fits Direction dependence Deploy on arrays Self-cal loop

Implications from Scaling Relations EPIC FX Most expensive step – 2D spatial FFT at every ADC output cycle – O(Ng log Ng) For a given Ng, it does not depend on Na. e.g., dense layouts like HERA, LWA, CHIME Thus the array layout can get dense with no additional cost Most expensive step – FX operations on N2 pairs at every ADC output cycle – O(Na2) Accumulation in visibilities before imaging offers some advantage Advantage lost for large arrays requiring fast writeouts (due to fast transients, rapid fringe rate, ionospheric changes, etc.)

Current and future telescopes in MOFF-FX parameter space Top left is where MOFF is more efficient than FX Dashed line shows where expanded HERA will be Shaded area is where LWA will evolve to be Large-N dense layouts favor EPIC EPIC will benefit most of future instruments MOFF FX

Current and future telescopes in MOFF-FX parameter space

Writeout rates for Transients Data rate (EPIC) GB/s Data rate (FX/XF) GB/s Data rate ~Ng for MOFF with EPIC Data rate ~Na2 for visibilities to be written out MOFF using EPIC lowers data rates significantly in modern/future telescopes MOFF with EPIC also yields calibrated images on short timescales Ideal for bright, fast (FRBs) and slow transients with large-N dense arrays Telescope Assumes writeout timescale of 10 ms

Proposed EPIC demonstration on HERA HERA (Hydrogen Epoch of Reionization Array) B = 100MHz 1024 channels ~100 kHz channels FoV ~ 10 deg. at 150 MHz Compact hexagonal array 14m dishes Use HERA prototype GPU-backend as test bed HERA will use current PAPER F-engine & GPUs that comprise the X-engine Design a GPU-based, image-based transient search backend NSF-ATI proposal submitted Test machine set up at ASU to experiment the porting HERA-331 Cost-effective way to test it since we have already the hardware. Test machine set up at ASU to experiment the porting.

EPIC Summary EPIC is promising for most modern/future telescopes (HERA, LWA, CHIME, SKA1, MWA II core, etc.) EoR studies Large-N dense arrays for sensitivity to large scales Radio Transients Fast writeouts Economic data rates Calibrated images at no additional cost EPIC paper - Thyagarajan et al. (2016) in review (arXiv:1510.08318) Highly parallelized EPIC implementation in Python publicly available - https://github.com/nithyanandan/EPIC/ Results of calibration studies (EPICal - Beardsley et al. in prep.) coming soon!