Digital Packaging Processor Gordon Hurford Jim McTiernan EOVSA PDR 15-March-2012

Role of Digital Packaging Processor To filter, average, partially calibrate and convert raw correlator output into a Miriad-compatible format that is written to Interim Data Base Real time, irreversible processing 1 1

DPP Interface Overview DPP State Frame ACC Correlator Start / End Scan Commands Scan-independent Calibration Parameters Scan Parameters Frame parameters Frame status report,, Correlations Interim Data Base Miriad format Internal RFI Database 1 s timing tick 0.02 s timing tick RFI results

DPP Task Timescales Data frame (20 ms) –filter, and frequency-average correlator output Spectral frame (1 s) –Assemble, pre-calibrate, reformat and write data to Interim database Scan initiation –Accept, store and preprocess scan-specific parameters Occasional – non operational –Accept, store and preprocess calibration parameters TBD –Format results and write to RFI database

DPP Software Architecture ACCState Frame Correlator IDB DPP_CONTROL I/O, data assembly, Miriad formatting, no numerical processing DPP_Process _Header DPP_Process_ Dataframe DPP_Process _Spectral _Frame Communication via Common Blocks Conventional, time-independent processing tasks with “Clean” inputs C1 C2 C3, C4 …. C2 Cn = core within a mullti-core processor or nodes in a cluster DPP RFI database DPP_Process_ Parameters C2

DPP_CONTROL -- Jim McTiernan -- Handles all external I/O Passes reformated header and calibration data on to processing tasks Assembles data frame input from correlator and calls processing task only if/when complete Assembles processed 20ms interval data and calls spectral frame processor only when all accumulations are available Converts output of spectral frame processor to Miriad format

DPP_PROCESS_DATAFRAME Tasks –Identify RFI-affected subbands as a function of frequency only. –Combine with pre-flagged subbands to generate a “destination vector” for each subband –Save RFI statistics – details TBD –Average subband data into spectral channels Challenge: 450 MB/s –Each visibility is handled only once –Multiple processors (up to 6) handle successive dataframes Initial Restriction: –No calibration at the subband level

DPP_PROCESS_SPECTRAL_FRAME Every spectral frame (1s) Convert antenna-based flags (e.g. slewing) from state frame to baseline-based, frequency-independent flags Apply time-independent complex gains Correct for attenuator settings Fine delay corrections Baseline corrections –(not necessary) Apply non-linearity corrections –Refinement to be added later, if necessary Correction for spectral simultaneity –not needed if using Miriad wide-band mode Conversion of visibility, uv and analysis-relevant state-frame data to Miriad-compatible format –May be a included here, in DPP_Control or as a separate Process

DPP-Correlator Interface 4096 x 2 x16^2 8-byte visibilities 4096 x 2 x 16 8-byte values  9 MB / data frame  450 MB/s Two Dedicated 10 Gb Ethernet link(s) Interface architecture driven by correlator design

Format of Correlator Output Format of interface based on 3-Feb-2012 EST memo from Nimish, as clarified in 12-March Features: –Accumulation divided into labeled packets, each with an identical header (except for sequence numbers) –Packets divided into ‘chunks’, each of which contain all power, power^2 and correlated data for a given subband –Data for a given subband has both polarizations interspersed, –All non-header data are scaled 4-byte integers Refinement: –Agreed that correlator output should always correspond to 16- antenna/2 polarizations. –DPP will discard unused antennas/baselines

DPP-Correlator Interface Open Issues Specifics of EST format generalization to EOVSA Byte order: –Correlator outputs Big-endian (MSB first) –Many machines (e.g. PC’s) use Little-endian (LSB first) How is the load to be divided between the two Ethernet interfaces? –Current format (and DPP software) assumes sequential input within a given accumulation

DPP Status Overall: –Software architecture and tasks defined –Platform to be acquired Interfaces: –IDB – Miriad format feasibility demonstrated with EST data –Correlator - as defined by Nimish with a few remaining issues –State frame - will follow protocol defined by GN, details TBD –ACC i – definition at an early stage –Timing ticks – TBD –RFI –TBD Milestones –Detailed software definition, coding and testing (with EST data) is underway –DPP_CONTROL has successfully associated tasks and processors –EST data loads correctly into DPP input common block –DPP_PROCESS_DATAFRAME well-defined and being coded. –EST to Miriad format conversion has been successfully demonstrated. –Immediate goal: End-to-end testing with EST data

Some DPP Open Issues Do we use spectral or wideband mode in Miriad? –Probably either would work –Initiall implementation will use wide-band mode (simpler, more flexible but less efficient) –analysis timing tests needed to see if this is an issue –easy to switch to spectral mode Is GNU Fortran 95 the best compiler choice? –Drawback is lack of support for STRUCTURE statements Workaround available for DPP_PROCESS_DATAFRAME Options for software development platforms –Independently develop DPP_CONTROL and processing routines on ‘personal’ machines, then test at OVRO? –Duplicate platform located in Bay area (JM, GH or SSL?) –Duplicate platform at NJIT ? Need to identify and purchase a multi-processor platform for DPP in near future.