1. E-field Parallel Imaging Correlator: An Ultra-efficient Architecture for Modern Radio Interferometers
Nithyanandan Thyagarajan (ASU, Tempe)
Adam P. Beardsley (ASU, Tempe)
Judd Bowman (ASU, Tempe)
Miguel Morales (UW, Seattle)

2. Outline Motivations for Direct Imaging
Modular Optimal Frequency Fourier Imaging (MOFF) – A generic direct imaging algorithm
EPIC implementation of MOFF in software
EPIC imaging in action
Imaging performance of EPIC vs. FX
EPIC calibration in action (Adam Beardsley)
EPIC on future large-N dense array layouts
Time-domain capability of EPIC
Testing GPU-based EPIC on HERA

3. Motivations for Direct Imaging
Technological
Scientific
EoR studies
Transient studies / Ionospheric monitoring
Large collecting areas require large-N arrays
Cost of the correlator scales as N2
Require fast writeouts
Fast writeouts under technical
Thornton et al. (2013)

4. Concept of Direct Imaging
Antennas placed on a grid and perform spatial FFT of antenna voltages on grid to get complex voltage images
Square the transformed complex voltage image to obtain real-valued intensity images
Current implementation:
8x8 array in Japan (Daishido et al. 2000)
4x8 BEST-2 array at Radiotelescopi de Medicina, Italy (Foster et al. 2014)

5. Need for generic direct imaging
Hurdles with current implementations
MOFF algorithm
Morales (2011)
Uniformly arranged arrays have poor point spread functions – thus not ideal for imaging
Aliasing of objects from outside field of view
Assumptions of identical antennas => poor calibration
Calibration still requires antenna correlations
Antennas need not be on a grid but still exploit FFT efficiency
Can customize to science needs
Accounts for non-identical antennas
Calibration does not require forming visibilities
Can handle complex imaging issues - w-projection, time-dependent wide-field refractions and scintillations
Optimal images

6. Mathematical basis for MOFF
Measured visibility is the spatial correlation of measured antenna E-fields
Antenna power pattern is the correlation of individual voltage patterns
Visibility measurement equation is separable into antenna measurement equations
Allows application of “multiplication route” in multiplication-convolution theorem of Fourier Transform (while visibility imaging uses “convolution route”)
FFT efficiency leveraged by gridding E-fields using antenna voltage illumination pattern

7. EPIC implementation of MOFF imagingFX
MOFF
Parallelization for efficiency – not production-level

8. Modular EPIC software

9. EPIC Software Summary
Object Oriented Python codes, emulate real-life telescope arrays
Parallelized for efficiency
Implements generic antenna layouts
Accounts for non-identical antenna shapes
Calibrates using only measured antenna voltages
Contains E-field simulator and FX/XF-based imaging pipelines for reference
Working towards production-level

10. Imaging with EPIC vs. FX Simulated Example: Nchan = 16 df = 100 kHz
f0 = 150 MHz
MWA core layout inside 150 m (51 antennas)
Square antenna kernels

11. Imaging with EPIC vs. FX (zero spacing)

12. EPIC on actual LWA Data LWA1 TBN data with a total of 2s and 100 kHz
Image obtained with 20 ms, 80 kHz
Cyg A and Cas A prominently visible

13. Calibration
*Need to apply calibration before gridding.

14. Calibration Problem E-field incident on ground related to sky by FT:
Antenna integrates with voltage pattern:
Further corruption modeled as complex gain and noise
Goal is to solve for the gain factor

15. Calibration Requirements
No visibilities
MOFF algorithm mixes antenna signals
⇒ Must apply calibration at front end
Scale <

16. Simple Case Single point source, centered on sky From Morales (2011):
Sufficient to determine self-cal solutions

17. Calibration Loop
An iterative approach
Scales O(N)

18. Simple Case Simulate using EPIC Start with random “guess” for gains
Use MWA core layout / beam pattern
No noise for now
Start with random “guess” for gains
Attempt to recover “true” gains
40 kHz, 4 channels

19. Simple Case Gain Amplitudes Gain Phase Errors
Each iteration consists of 20 finer iterations.
Damping factor 0.35 (memory from previous gain solution), 0.65 from current solution

20. Simple Case
Image Before Calibration
Image After Calibration

21. Generalize… … Inspect this product Arbitrary sky, any direction
Prime means current estimate and tracks the actual gain
Arbitrary sky, any direction

22. Simulate more sources Gain Amplitudes Gain Phase Errors
Each iteration = 400 timestamps totaling 10 ms,
No receiver noise, only sky noise
Remind about phase wrap
Initial dip in amplitude is overcompensating for initial high gains

23. Simulate More Sources

24. Calibrating LWA data Same LWA data 150 channels ~ 29 kHz bandwidth
Cal loop on 51.2 ms cadence
total 1.5 s (30 cal. iterations)
Model Cyg A and Cas A as point sources
Need to handle auto correlation terms
If you use the new plots from Evernote, 10 timestamps per cal update, 30 cal updates, total of 1.5s
Get auto-subtraction term from Adam’s paper on github

25. Calibrating LWA data Gain Amplitudes Gain Phases
Real data implies phases and gains converge to different values but are locked in

26. Calibrating LWA data Image Before Image After Reference Calibration
Comment on image quality
Image Before
Calibration
Image After
Calibration
Reference
Image
55% increase in dynamic range

27. Further directions Sophisticated fits Direction dependence
Self-cal loop: use images to update model and have a iterative feedback loop
Sophisticated fits
Direction dependence
Deploy on arrays
Self-cal loop

28. Implications from Scaling Relations
EPIC FX
Most expensive step – 2D spatial FFT at every ADC output cycle – O(Ng log Ng)
For a given Ng, it does not depend on Na. e.g., dense layouts like HERA, LWA, CHIME
Thus the array layout can get dense with no additional cost
Most expensive step – FX operations on N2 pairs at every ADC output cycle – O(Na2)
Accumulation in visibilities before imaging offers some advantage
Advantage lost for large arrays requiring fast writeouts (due to fast transients, rapid fringe rate, ionospheric changes, etc.)

29. Current and future telescopes in MOFF-FX parameter space
Top left is where MOFF is more efficient than FX
Dashed line shows where expanded HERA will be
Shaded area is where LWA will evolve to be
Large-N dense layouts favor EPIC
EPIC will benefit most of future instruments
MOFF FX

30. Current and future telescopes in MOFF-FX parameter space

31. Writeout rates for Transients
Data rate
(EPIC)
GB/s
Data rate
(FX/XF)
GB/s
Data rate ~Ng for MOFF with EPIC
Data rate ~Na2 for visibilities to be written out
MOFF using EPIC lowers data rates significantly in modern/future telescopes
MOFF with EPIC also yields calibrated images on short timescales
Ideal for bright, fast (FRBs) and slow transients with large-N dense arrays
Telescope
Assumes writeout timescale of 10 ms

32. Proposed EPIC demonstration on HERA
HERA (Hydrogen Epoch of Reionization Array)
B = 100MHz
1024 channels
~100 kHz channels
FoV ~ 10 deg. at 150 MHz
Compact hexagonal array
14m dishes
Use HERA prototype GPU-backend as test bed
HERA will use current PAPER F-engine & GPUs that comprise the X-engine
Design a GPU-based, image-based transient search backend
NSF-ATI proposal submitted
Test machine set up at ASU to experiment the porting
HERA-331
Cost-effective way to test it since we have already the hardware.
Test machine set up at ASU to experiment the porting.

33. EPIC Summary
EPIC is promising for most modern/future telescopes (HERA, LWA, CHIME, SKA1, MWA II core, etc.)
EoR studies
Large-N dense arrays for sensitivity to large scales
Radio Transients
Fast writeouts
Economic data rates
Calibrated images at no additional cost
EPIC paper - Thyagarajan et al. (2016) in review
(arXiv: )
Highly parallelized EPIC implementation in Python publicly available -
Results of calibration studies (EPICal - Beardsley et al. in prep.) coming soon!