1. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School Cross-correlators for Radio Astronomy ATP-NSI Brent Carlson, NRAO Synthesis Imaging Summer School 29 May 2012

2. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 2 Outline What is the purpose of the correlator? Simplified signal flow Basic correlator architectures -XF, FX, hybrid Technology -How do the electronics “work” The development process JVLA WIDAR correlator Now and the future

3. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 3 Purpose To calculate the integrated cross-power response for each pair of antennas “X” and “Y” in the array over some integration time “T”. Statistically independent signals integrate down, whereas the common radio source signal establishes some power level and the uncertainty in that level integrates down.

4. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School

5. 5 Purpose These are the visibilities—spatial Fourier components—for each baseline B in the u-v plane that are used to build the image. The fun begins: -As N increases, number of pairs of antennas goes as ~N 2 / 2. (e.g. JVLA has 27 antennas…378 baselines (including “auto” correlations). -As bandwidth and number of (spectral) channels increases, the speed of the electronics increases along with power and cost.

6. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 6 Purpose Analog correlators, operating on analog (continuous time) voltage signals have been built and do work… But nowadays the analog signal is quantized—in time and amplitude—as soon as possible for stability and to take advantage of “cheap” high-speed digital electronics. -Once the signal is digitized there are no more unknown/unquantifiable effects (well, unless something broke…)

7. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 7 Simplified signal flow STEP #1: -Receive and amplify the signal from the antenna. The LNA adds significant noise to the weak signal, and must be optimized to optimize sensitivity of the telescope. Additive noise stability (or calibrated as such) here is important for stable correlator output.

8. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 8 Simplified signal flow What does the signal “look” like? -Time domain (analog):

9. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 9 Simplified signal flow STEP #2: -down-convert (mix) and filter the signal…ready for digital sampling. Gain fluctuations here not so critical as long as additive noise is stable. It is the ratio of radio source signal to system noise that is important.

10. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 10 Simplified signal flow STEP #3: -quantize (digitize/sample) the signal. One of the most critical+difficult steps in the processing chain because it is where the quiet analog world meets the noisy digital world. Speed is important to maximize bandwidth.

11. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School “Flash” A-to-D converter diagram.

12. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 12 Simplified signal flow What does the signal look like? -Time domain (digital):

13. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 13 Simplified signal flow Sampling: -Nyquist sampling theorem: must sample at least 2X the highest frequency content to obtain all information about the signal. If less, leads to “aliasing” (confusion). -With noise input: -1-bit results in ~35% sensitivity loss. -2-bit results in ~12% sensitivity loss. -3-bit: 3.5% -- JVLA wideband samplers -4-bit: 1.5% -- JVLA correlator “internal samplers”

14. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 14 Simplified signal flow Sampling: -adding more bits/sample produces diminishing sensitivity returns for noise input and integrated output. -When narrowband interference is present, need more bits so as not to contaminate the spectrum with saturated sampler-generated harmonics. Get ~6 dB per bit dynamic range for a pure tone. dB=10log(x); if x is a power value. -For real-time signals (music/video) need lots of bits to accurately represent the real-time waveform (e.g. CD ~16-bit sampling=2 16 = 65,536 levels)

15. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 15 Simplified signal flow STEP #4: -Correct for antenna-dependent wavefront delay. Two phases: 1)pure digital delay to +/-0.5 samples. Cheapest is to use memory. Get up to +/-90 deg phase changes at the upper edge of the band…severe decorrelation, therefore need: 2)sub-sample delay to << +/- 0.5 samples. Various methods, sometimes analog, often digital…more later.

16. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 16 Simplified signal flow STEP #5: -Cross-correlate and accumulate. Must also correct for “fringe phase” due to the fact that wavefront delay compensation occurs at a different frequency (baseband) than where it originally occurred (at RF in free space). Various correlator methods to be discussed later…

17. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 17 Simplified signal flow What does the signal look like? -Frequency domain (10e6 samples integrated):

18. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 18 Simplified signal flow What does the signal look like? -Frequency domain…add some lines:

19. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 19 Simplified signal flow

20. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 20 Correlator architectures There are two basic methods for correlation: -“XF”: Cross-correlate in the time (tau) domain, then Fourier transform (after integration) to the frequency domain. a.k.a. “lag correlator” -“FX”: Fourier transform in the (real) time domain, then multiply and integrate in the frequency domain. “Convolution in one domain is multiplication in the other domain”

21. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 21 Correlator architectures Hybrid: -Combination of the two. JVLA WIDAR does this as does the ALMA correlator. -Coarse filter into sub-bands, XF each sub-band. -More details later.

22. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 22 Correlator architectures XF: -traditionally simpler to understand+implement— especially for 1-bit or 2-bit correlators (e.g. 1-bit correlator multiplier is XOR gate). Important in “earlier days” because of speed and logic availability. -Data N ant 2 distribution in correlator still uses the same few bits. -O(N chan x sample rate) multiplies/sec…but, very simple operations (multiply-accumulate).

23. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School

29. 29 Correlator architectures FX: -More complex, many-bit operations (FFT). (Has been) more difficult to implement. -O(log N chan ) x sample rate multiplies/sec…much more efficient…in principle. -Problems: 1.Have word-width expansion after FFT: (has been) 1 or 2-bit in, many bits out. 2.How to window the real-time data before FFT?

30. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School

31. 31 Correlator architectures FX problems: -Word-width expansion: -Keep full word width: large N ant 2 data distribution problem in correlator. -Reduced word width quasi-floating point. Original VLBA correlator did this. -Re-quantize to ~4 bits. Not much sensitivity loss. With strong interference just toss the bad channel. With strong source lines…measure power in each channel before re- quantization, correct levels afterwards. -Keep reasonable # bits to accurately represent the dynamic range of signals possible…no pre-requantizer power measurement required.

32. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 32 Correlator architectures FX problems: -How to window the real-time data before the FFT? -Don’t…use “box-car” with no overlap…large ringing/sidelobes (sin(x)/x = “sinc” function) but if signal is just noise it is ok. Reduces SNR slightly since all of the data isn’t being correlated. -Triangular 50% overlap…reduces ringing/sidelobes (sinc 2 ), no SNR loss since all data is used. Orig VLBA correlator did this. -Poly-phase FIR filter…excellent!

33. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 33 Correlator architectures Poly-phase FX: -Put poly-phase FIR (Finite Impulse Response) filter in front of the FFT. -Acts like a window/taper function that is longer than the FFT…achieves superior channel-to-channel isolation and spectral dynamic range. ~10 taps per frequency channel is required. -This performance cannot fundamentally be obtained with an XF correlator: window function limited to size of FFT, and causes broadening of spectral features. -~2X the amount of logic to implement over just the FFT. -Now not such a problem with modern high-performance programmable devices (FPGAs).

34. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School “Ringing” caused by truncated FFT. a.k.a. “Gibb’s phenomenon. Caused by convolving the true spectrum with a sinc() function. Ringing greatly reduced by tapered windowing before FFT but lines are broadened.

35. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School

38. 38 Correlator architectures More on poly-phase FX: -If many channels ~>1k required, must cascade filterbanks to avoid excessive numbers of poly-phase FIR taps (e.g. 1 stage 1Mchannels ~10Mtaps(!); 2 stage ~2k taps). -Each channel out of the FFT will have some leakage (aliasing) from the adjacent channel since it is critically sampled. -Therefore, must oversample the output. Performed by “barrel-rolling” the input across phases such that the output after FFT is oversampled. The next stage poly-phase-FFT of a particular channel will therefore not contain aliasing in regions of interest so that all taken together, clean alias-free spectra are obtained…

39. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 39 Correlator architectures See: Harris, Dick, Rice, “Digital Receivers and Transmitters using Polyphase Filterbanks for Wireless Communications”, IEEE transactions on microwave theory and techniques, Vol. 51, No. 4, April 2003. …for more detail on poly-phase filterbanks (great paper!)

40. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 40 Correlator architectures Hybrid FX-XF (a.k.a. “FXF”) -1 st stage: coarse filterbank (FIR or poly-phase FFT). -Useful as “digital BBC” for frequency-agile sub-band placement. -2 nd stage: XF. -Attractive as an efficient parallel processing method for wideband signals since no large multiplier operations are required (FIRs built with memory LUTs and adders). Has larger noise-power bandwidth in sub-band so can use fewer bits connecting to the ‘X’ part. -JVLA and ALMA correlators built this way (some slight differences in implementations). Probably the last of this breed!

41. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 41 Correlator architectures The actual signal processing operations are just one piece of the puzzle when putting a system together. Much of the logic and power in a system is consumed by transporting data around, synchronizing things, providing various modes of operation, error detection and recovery etc. Let’s look “under the hood” of the electronics…

42. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 42 Technology It all starts with fundamental physics…but moving up a level or two: -Transistor “switch”: the FET – Field Effect Transistor.

43. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 43 Technology N-MOS: applying a voltage to the Gate opens a conduction channel between the Drain and Source. P-MOS: applying a voltage to the Gate closes the conduction channel between the Source and the Drain.

44. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School MOS: Metal Oxide Semi-conductor. MO is the insulator between the gate and the conduction channel. Extremely sensitive to electro-static discharge (ESD). When the transistor is ON or OFF, no current flows from the gate to the conduction channel (unless it is blown…) Current (power) only flows when changing states, to charge/discharge the gate capacitance…faster state changes consumes more power.

45. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School CMOS: Complementary Metal Oxide Semiconductor. Output changes faster since it is being driven both high and low. Small amount of leakage current (power) when the conduction channel is switching states.

46. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School

48. A 4-bit 2’s complement multiplier F.A. (Full Adder)

49. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 49 Technology A logic gate “immediately” reflects changes on its input to its output. It can’t store a value. -A “bunch of gates doing something” is often referred to as “combinational logic”. A “Flip-Flop” transfers “Data in” to the output “Data out” only on the edge of its clock: A Flip-flop can therefore store a value…a single bit. A Flip-flop is actually a set of cross- coupled gates with feedback implementing an “asynchronous Finite State Machine”.

50. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 50 Technology In digital electronics, pretty much everything is “synchronous”. i.e. changes occur on the clock edge all “in step”. -It’s like a production line…the speed of the line is the clock speed and in each clock cycle each “worker” (bunch of gates doing some logic function) must get their step done before the next clock cycle starts. -As the clock speed increases, the logic “workers” must go faster to keep up.

51. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 51 Technology Together (and in the millions/billions), Flip-flops and gates (along with memory cells) form the bulk of all digital electronics. As feature sizes (transistors) get smaller, more gates can be packed on a chip, they run faster, and more can be done. -JVLA correlator implemented with 90 nm and 130 nm devices (c. 2005). -Industry currently shipping 28 nm devices…20 nm is next…

52. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School

58. 58 Development process Fundamentally, engineering a system uses a hierarchical top-down approach: -Take a big problem, hierarchically break into smaller and smaller problems until each small problem can be solved by a human. -Then, put everything together, and it works! (eventually) In reality there are several rounds of iteration and engineering may start at any point in the hierarchy, iterate, iterate…to eventually get a coherent picture to proceed with development.

59. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 59 Development process In logic design, at the “application level”, we don’t (or, rarely) design explicitly with gates and flip-flops. We write HDL – Hardware Description Language code that describes logic in a high-level fashion. -And there are higher-level approaches as well. Can (optionally) use hierarchical graphical design tools as well to improve the human’s ability to understand how it all fits together.

60. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School

62. 62 Development process In an HDL (VHDL, Verilog), all instructions are executed concurrently (i.e. on the clock edge), and so one must think concurrently when writing code. -If some output changes “now”, it isn’t picked up until the next clock edge. -Seems impossible to have program flow…but we create that with pipelining and Finite State Machines. -Logic doesn’t pop in and out of existence like functions/objects…it is always there and always active!

63. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 63 Development process “Synthesis” and “place and route” software then compiles our HDL logic design to get implemented on our target device. -For an “FPGA” – Field Programmable Gate Array, SRAM cells in the chip are downloaded at “run time” with the compiled “program” (firmware--”bitstream”) and set the chip to execute the program by setting connections/routes/use. -For an “ASIC” – Application Specific Integrated Circuit, the compiled program is used to plunk down gates, flip-flops, and memory and connect them with wires in the design software…which then gets built in a real chip foundry or “fab”. Wafers, die, …

64. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 64 JVLA WIDAR correlator Basic specs: -32 antennas, 8 GHz/polarization (in 2 GHz chunks 3-bit sampling; alternately 4 x 1 GHz 8-bit sampling). -128 independently tunable digital sub-bands; 128 MHz, 64 MHz, downto 31.25 kHz BW per sub-band. -Each sub-band can (will be able to) have a different delay center on the sky, within the antenna primary beam. ->60 dB (10e6:1) spurious spectral dynamic range for high spectral purity…within XF fundamental limitations. -16,384 to 4 million spectral channels per baseline…

65. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 65 JVLA WIDAR correlator Basic specs cont’d: -“Recirculation” provides a squared increase in spectral resolution with decrease in sub-band bandwidth. -Agile integration modes: normal, recirculation, pulsar phase binning, burst mode. -Able to flexibly tradeoff sub-band bandwidth for spectral resolution. -High time-resolution output; 10 msec minimum, 1 msec possible with 10G upgrade. -Phased-array output on all sub-bands…VDIF packets, and built-in autocorrelations.

66. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 66 JVLA WIDAR correlator Basic specs cont’d: -Correlator “configuration mapper” translates high-level observing requests from the JVLA Executor, into detailed correlator configurations. -The config mapper gets away from “mode-based” observing, to “goal-based” observing. -Performs mapping in flexible ways to maximize packing of resources, or maximize real-time performance (short integration times) by spreading out operations amongst available resources.

67. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 67 JVLA WIDAR correlator Simplified signal processing: 1.Offset the LO in each antenna by a small amount. 2.Filter wideband sampled signals into integer sub-band slots (Stage 1)…aliasing. (Other stages don’t have the integer slot restriction.) 3.Remove LO offset and “fringe phase” in complex XF correlator…causing aliasing to wash out with integration time. 4.Measure power before re-quantizer to allow wideband to be seamlessly “stitched together” from individually correlated sub-bands after correlation.

68. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School

72. 72 JVLA WIDAR correlator Due to aliasing, there is a factor of 2 sensitivity loss at the sub-band edge, tapering off to none at the rate of the transition band steepness. -Using a complex FIR filter and passband overlap there is an efficient way of eliminating this (always one more trick in the book…), but with some loss of bandwidth— the sum of all the transition band bandwidths—if implemented in the same hardware.

73. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School

74. 10e-3 passband ripple

75. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School

80. 80 JVLA WIDAR correlator Sub-sample delay tracking. -Wideband delay tracking to +/-0.5 samples in memory, produces a changing phase slope between +/-90 deg on the wideband signal. -Within a sub-band however, the phase slope change is 1/16 th of this and a changing phase that can be tracked (taken out) by the phase rotator. The residual very small amplitude and phase effects can be removed post-correlation. -Therefore known and stable sub-sample delay correction—and with it sensitivity—is achieved, further improving stability and minimizing systematic effects.

81. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School

82. 82 JVLA WIDAR correlator Due to cost and power constraints, the complex mixer/fringe rotator in the correlator chip approximates sine and cosine functions with 3 levels. -This reduces the sensitivity by ~5% and “splatters” that extra noise across the sub-band. -Provided the chosen LO offset frequencies are not related/don’t beat with each other, and provided the sampler doesn’t saturate to produce harmonics and intermodulation products of narrowband signals, this shows up as noise that integrates down, and generally it performs as well or better than with no fshift+mixer.

83. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School

85. No LO offset; 4-bit sampler saturating due to narrow-band tone power-to-noise power ratio (~300:1) too high (~28 dB DR).

86. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School LO offset with 3-level phase rotation; 4-bit sampler saturating due to narrow-band tone power-to-noise power ratio too high but most artifacts gone (~42 dB DR).

87. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School Here, 75% of power in sub-band is in the tone. No artifacts that don’t integrate down. >50 dB dynamic range…takes too long to simulate to achieve 60 dB!

88. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School

95. 95 JVLA WIDAR correlator Other features: -2 banks of 2000 phase bins for high time resolution (as low as ~12 usec for reduced spectral channels) “stroboscopic” observations of pulsars. -Recirculation up to a factor of 256 for up to 262,144 channels on each of 16 sub-bands. -Working on adding high time resolution burst mode for transient detection…possibly operating in the background all the time “striping” across correlator resources.

96. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School

102. 102 Now and the future Continued advances in semi-conductor technology are making correlator systems more “appliance-like” than ever before. Shipping now: 28 nm; next: 20 nm. A few COTS rack-mount CPUs can now do what a custom system used to have (to be engineered) to do. -The new VLBA “DiFX” correlator is a “software” correlator. FPGAs are more and more powerful. Latest available have ~3000 “DSP blocks”, 1-2M logic cells, useable 500 MHz clock rates, several tens of 10G and 28G transceivers…craziness!

103. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 103 Now and the future For any problem size, one has to look at capital vs operating cost (power) and decide what is the best technology. Certainly: -CPUs/GPUs for “small” to “medium” jobs. Relatively quick turnaround time/development effort (don’t forget s/w!). Power not such a concern. -FPGAs for “medium” to “large” jobs (can easily fit the entire VLBA correlator onto one FPGA now). Power starts to be a concern…consider ASIC migration. -ASICs for “very large” jobs where operating cost in terms of power is of primary concern. Probably only the SKA would ever need an ASIC again. -Poly-phase FX due to availability of large multipliers/adders and relatively inexpensive high-speed serial links.

104. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 104 Now and the future When comparing implementation concepts/ technologies and flexibility, must also bear in mind performance requirements and capabilities. -e.g. a few hundred MHz and a handful of antennas vs several 10s of GHz and hundreds or thousands of antennas. Must also bear in mind that a fully operational, shaken-down, facility-level system, no matter which way you cut it has software and testing overhead that takes people and time to get right. -“Wait a minute captain while I reprogram the computer to check for sub-space frequencies…”

105. Brent Carlson, 2012 NRAO Synthesis Imaging Summer School Questions? Thank-you