1. PELICAN Imaging Framework Imaging on short timescales leads to very large correlator output data rates. In order to cope with these rates and produce updated calibration coefficients it is necessary to process the output data stream in real time. PELICAN b developed by the Oxford e-Research Center (OeRC) is a efficient and modular framework to process real time data streams. Data is split into parallel streams processed on CPU/GPUs to form images of the transiting sky and differential images for transient detection. Design of a 96 Element FX Correlator for the LOFAR-UK Station G. Foster 1,2 and K. Zarb Adami 1 1 University of Oxford 2 griffin.foster@astro.ox.ac.uk The LOFAR-UK Station The international LOFAR station at Chilbolton Observatory consists of a 96 element low band array (LBA) and a 96 element high band array (HBA) connected to a single digital backend. The station was completed in September 2010 and has been commissioned for operation. The current backend is designed to create beamlets from the station antennas to be beamformed and correlated at the LOFAR correlator in Groningen, Netherlands. A Single Station Correlator Each LOFAR station has a limited calibration correlator which has been used for single station, widefield images throughout station commissioning as a diagnostic tool and for developing the imaging pipeline. This correlator cycles through the individual subbands to produce a single channel correlation on second timescales. A dedicated correlator is in development which can process a selectable portion of the band (7 MHz per module), provide further subband channelization, and output correlations on subsecond timescales. A key science goal for this instrument will be, among others, monitoring and imaging of short timescale transient events. In addition to the FPGA based correlator a CPU/GPU realtime imaging pipeline will be necessary to cope with the large output data rates. This instrument will interface with the current LOFAR RSP such that commensal observations can be performed while the station is being used during international LOFAR operations. Completed development of the correlator and imaging pipeline is expect in early 2012. RSP Interface The current station digital backend uses a XAUI loop for forming beams and correlator calibration using the 24 RSP boards. Approximately 25% of the total bandwidth is unused. Each XAUI contains four lines, three will continue to be part of the main loop and the remaining line will be connected to the station correlator. An RSP firmware modification will allow a selectable 7 MHz of the band to be output over a single XAUI line. This firmware modification will be completed by ASTRON and be used in the SuperTERP correlator for the AARTFAAC c project. ROACH II Hardware The next generation ROACH-II board designed by CASPER a /KAT is based on a Xilinx Virtex 6 FPGA. The CASPER design tools are built around reusable DSP blocks. Designs are built and simulated using Simulink and the Xilinx toolflow. Traditional HDL can also be incorporated into designs. Each board can process up to 60 Gbps using CX-4 adapted cards. A modular design will be used to compute correlations subsets of the band across multiple boards. Two ROACH-II boards will be required to compute the full Stokes correlation of all 96 elements for the 7 MHz band. AP0 AP1 AP2 AP3 BP LCU Inter board interface (IBI) ring CEP serdes RSP Board...... RCU0 (Ant 0: pol X, pol Y) RCU95 (Ant 95: pol X, pol Y) RSP0 (Ant 0…4: pol X, pol Y) RSP23 (Ant 92…95: pol X, pol Y)...... x24 XAUI0 XAUI1 XAUI2 XAUI3 XAUI4 XAUI5 XAUI Subband Splitter XAUI Antenna Reorder Corner Turn Windowed X Engine 0 (48 Dual pol taps, 9 subbanbs) Quantization Equalization (4 bits) Windowed X Engine 1 (48 Dual pol taps, 9 subbanbs) Vector Accumulator 0 Vector Accumulator 1 10 GbE XAUI Subband Splitter XAUI Antenna Reorder Corner Turn Windowed X Engine 2 (48 Dual pol taps, 9 subbanbs) Quantization Equalization (4 bits) Windowed X Engine 3 (48 Dual pol taps, 9 subbanbs) Vector Accumulator 2 Vector Accumulator 3 10 GbE Subband Chunker RFI FlaggingCalibration Local Sky Model FFT ImagerCLEAN Image Database Differential Image Threshold Detection Stack Image … … … XAUI Line ROACH-II 1 ROACH-II 0 Correlator Specs Ant-pols192 Baselines18528 (Auto + Cross) Bandwidth7 MHz / XAUI RSP Boards24 (4 antennas w/ 2 pol per board) Subband Width200 kHz Data Format16 bit complex Integration Time~10-100 ms Integration Size141 MB Input Data Rate60 Gbps Windowed X Engine Taps1 auto + 48 cross (1/2 the number of antennas for a dual pol array) CMACs / Tap4 (16 4x4 multipliers and 4 accumulators) Multipliers / X Engine 784 Xilinx FPGA Device Virtex 6 SX475T Clock~300 MHz Logic Slices74400 DSP48e*2016 GTX IO36 (6.6 Gbps Max) *DSP48e contains a 25x18 multiplier and accumulator ROACH II IO/Memory FPGAVirtex 6 SX475T QDR4 x 36 bit x 2M QDR II+ DRAM144 bit DDR3 DRAM Interface 10 GbE(3 CX-4 or 4 SFP+) x 2 PELICAN Framework To interface the correlator with the current LOFAR digital backend a modification to the RSP firmware is required along with a passive XAUI combiner board. The current correlator design is implemented on 2 ROACH-II boards, each board has 3 CX-4 interfaces for input from the RSP and 4 SFP+ interfaces for interboard communications and 10 GbE output. The input data streams need to be reordered such that half of the subbands go into a single ROACH-II. A CASPER windowed X Engine is used to optimize resource utilization. The PELICAN pipeline will be used to form sky images along with providing a calibration routine which will be able to update the correlator phase and amplitude coefficients in real time. A local sky model will provide the initial calibration. A GPU based 2D FFT will be used to form the dirty image. For short integrations and low resolution, bright point sources will dominate the field. A short CLEAN loop can be used to isolate the sources based on the sky model. A differential comparison of images based on a number of time steps will be performed and a threshold detector will be used to find transient events. A slower stacking module will also be used to form a high dynamic range sky survey image. References a Collaboration for Astronomy Signal Processing and Electronics Research (http://casper.berkeley.edu) b Pipeline for Extensible, Lightweight Imaging and CAlibratioN (https://wiki.oerc.ox.ac.uk/svn/pelican/user/index.html, https://github.com/pelican/) c Amsterdam—ASTRON Radio Transients Facility and Analysis Centre (http://www.aartfaac.org/) XML Configuration DataBlob input data abstraction PELICAN C++ Module Class Auxiliary DataBlob DataBlob output data abstraction PELICAN Module Global Sky Model Phase/Amp Coefficients Subband Chunker Image by Jean-Mathisa Grassmeier, produced using data from LOFAR Station FR606 at Nancay ASTRON LOFAR-UK: Chilbotlon Observatory CASPER: ROACH-II