JIVE UniBoard Correlator External Review 3 September 2014 Jonathan Hargreaves, JIVE JIVE UniBoard Correlator External Review

JIVE UniBoard Correlator External Review Introduction The JUC correlator firmware Architecture Signal flow Future development Ask about implementation details JIVE UniBoard Correlator External Review

JIVE UniBoard Correlator External Review Specifications   Supported June 2014 Planned Stations 32 Polarizations 2 Bandwidth (real time) 64MHz (1 UNB) 64n MHz (n UNBs) Sub-bands (real time) 16MHz, 32MHz under test 1, 2, 4, 8, 16, 32, 64MHz Input resolution (max) 2 bits 1, 2, 4, 8 bits Integration time (all products) 0.022s – 1s 0.022s – 2s Correlation points 2112 incl cross, auto, and cross-polarization 2112 Frequency resolution 15.625kHz <1kHz spectral line mode >125kHz continuum mode Data Input Format VDIF VDIF mixed frame sizes Columns now and future Real time means processed slightly faster than the sample rate JIVE UniBoard Correlator External Review

JIVE UniBoard Correlator External Review Overall Signal Flow 8 field programmable gate arrays Fns and bns Fns do station based processing and channelisation. Bns receive ¼ of the bandwidth. Coloured line represent chunks of frequency bins JIVE UniBoard Correlator External Review

JIVE UniBoard Correlator External Review Control 1 At the start of every scan the control computer must set An application reset to all nodes The start second The bandwidth, frame length, integration time Which input streams are lower sidebands In the BNs, the products to export Incoming data are stored in a 4 second circular buffer The data stream (station, polarization & band) is determined by the UDP port number and IP address The time slot is determined by the VDIF time code and frame number JIVE UniBoard Correlator External Review

JIVE UniBoard Correlator External Review Control 2 Data processing begins after 2 seconds of data have been placed in the buffer The UniBoard does not check if all the data has arrived … it waits for the control computer to tell it Every ‘second’ the control computer tells the UniBoard how many integrations to process The UniBoard indicates via a status bit when it has finished processing The control computer interacts with FN0. FN0 tells the other FN1,2 & 3 when to start and stop processing data The UniBoard does not know if it is processing disk packs or eVLBI Gaps in the data are handled by the validity bits JIVE UniBoard Correlator External Review

JIVE UniBoard Correlator External Review Diagnostics Several diagnostic registers are read and logged by the control computer during a scan Number of frames received per input stream Number of unmatched frames received Number of delay model packets received Number of packets received and transmitted by 10GbE ports Counts of the FFT frames generated by each FN and received by each BN A sample statistics module has been written but not yet integrated into the FN firmware Signal taps JIVE UniBoard Correlator External Review

JIVE UniBoard Correlator External Review Architecture 32 stations x 2 pol All products are calculated but only the necessary ones are exported Stations are virtual: the control software can make a 16 or 8 station correlator with 2x or 4x the bandwidth Native mode is 4 bands of 16MHz Wider bands (32MHz, 64MHz) be supported with new firmware Narrower bands (8MHz, 4MHz …) can be processed at a multiple of real time Each band is channelized to 1024 frequency bins Other channelizations could be supported with new firmware The polyphase filter window function can be changed at run time Maximum integration time is 1s Limited by size of DDR3 module used for corner turning More unbs -> more bw JIVE UniBoard Correlator External Review

Front Node Hierarchical Design Implementation SOPC Physical Design 10Gb-Eth Packet Receiver Mixer Pre Filter Structure FFT Normalize Framer FBI Delay Model SOPC 1Gb-Eth ETH Switch Mesh Control Computer DDR Hierarchical Design 10Gb-Eth Delay Model SOPC Filter Bank Packet receive Framer DDR FBI 1Gb-Eth Normalize Physical Design Floor Plan

JIVE UniBoard Correlator External Review PFB Architecture JIVE UniBoard Correlator External Review

JIVE UniBoard Correlator External Review PFB Window Function Blackman harris has good interference rejection Other windows better in band shape Limit of 6 x fft due to on chip memory limits JIVE UniBoard Correlator External Review

Delay Model Evaluation Fifo   d1 + 1GbE port Delay rate register Integer sample delay UDP Offload 32 bits Packet decoder Output taps Fractional sample delay 31 59 51 20 28 27 24 Delay accumulator A new set of coefficients is loaded at the start of every integration Evaluates Delay = d0 + d1t once per FFT Offset between delay and rate increase dynamic range Coefficients are currently 32 bits, will be 48 bits The second derivative coefficient is stored but not currently used JIVE UniBoard Correlator External Review

Delay Model Application The integer (whole sample) part of the delay determines the address of the first sample in the integration The next 8 bits are used to calculate a phase correction to the nearest 1/256th of a sample The correction is applied to the frequency bins as they exit the FFT If the delay correction rolls over a whole sample during an integration, one sample is skipped or repeated and a 90 degree correction is applied to the phase model JIVE UniBoard Correlator External Review

JIVE UniBoard Correlator External Review Phase correction   + x2 P0 register P2 register P1 register Phase to mixer Evaluates Phase = p0 +Σ(p1 + p2 + 2Σp2) once per sample Phase is applied in a quadrature mixer at the input to the polyphase filter Coefficients currently 48 bits, will be 64 bits JIVE UniBoard Correlator External Review

(Containing 132 MAC cells) FBI Deframer Corner Turner Validity Accu Correlator Engine SOPC 1Gb-Eth Formatter 10Gb-Eth Mesh ETH Switch Control Computer DDR_I DDR_II Back Node Implementation Hierarchical Design 10Gb-Eth Formatter SOPC Corner Turner Deframer Correlator Engine (Containing 132 MAC cells) Validity Accumulator DDR_II DDR_I FBI 1Gb-Eth Floor Plan Physical Design

Correlator Multiply Accumulate Cell JIVE UniBoard Correlator External Review

JIVE UniBoard Correlator External Review Validity 1 bit per VDIF frame stored in FN A whole FFT is invalid if any contributing data are invalid First six FFTs in an integration are invalid because pre-filter structure is filling up Invalid FFTs are substituted by zeros so do not contribute to the products One validity bit per FFT is carried across to BN and corner-turned with the data Thirty-two bit validity accumulators calculate normalization factors for every product. JIVE UniBoard Correlator External Review

JIVE UniBoard Correlator External Review Current Developments Widen delay/phase coefficients from 32/48 to 48/64 bits Firmware written and under simulation/test Two bands of 32MHz bandwidth Under test Sample Statistics Firmware written, needs integration and verification VDIF Frame length So far only 5000 byte frames have been tested Mixed frame lengths can be supported with further work JIVE UniBoard Correlator External Review

JIVE UniBoard Correlator External Review Future Developments Support 1, 4 and 8 bit sampled data One bit can be supported by converting to 2 bit at the input 4 and 8 bit to be added when needed VDIF epoch One band of 64MHz bandwidth, other modes? Stagger output data to alleviate bursting Lower spectral resolution to reduce volume of output data 1024 -> 64 points UniBoard2 – 2x to 8x the bandwidth per board JIVE UniBoard Correlator External Review