Archive, Data Products, & Operations for the Murchison Widefield Array Ron Remillard (MIT); MWA Project Meeting, June 6, 2011

Outline Resources for the MWA Archive at MIT Standard Data Products NSF MRI Program Standard Data Products Incomplete definitions in early 2011 Strategy for progress Information Flow Diagram Commissioning Phase Needs for Commissioning Plan Needs for more sophisticated Monitoring Operations Phase Operations Plan and Needs for Update Plans for remainder of 2011

MWA Archive at MIT NSF MRI Program ($868 K: NSF + MIT contribution) Large equipment budget Advanced spending on archive requirements, data products Need to set aside majority of funds until late 2012 Archive Tasks (based on EoR, systematic error analyses, student projects) Capture raw data (in short bursts) Intermediate data capture (non-calibrated images; less resolution (n, t)) High timer resolution calibrated data Determine calibrations in parallel with RTS Calibrate data offline Study ionosphere parameters at 2 s Service MWA EoR Consortium, other EoR users, other outside projects

MWA Archive at MIT Gaps in Plan Redefine SOW for 128T (retain EoR focus; gain approval for NSF extensions) Complete Archive Requirements Document (project-wide scope) Method to obtain data from Curtin Method to provide data access for Users

Standard Data Products Science Data Eor Data Products (calibrated raw images) defined in MRI proposal SHI Needs raw data (pro-Correlator) and UV data from outlier tiles Transients & GEG groups need Images and other modes (light curves, transient searches (image diffs), wider images (low t res.), others?) “Meta Data” RTS {calibrator catalogs, calibrator solutions, ionospheric params., “peeled” sources} M&C {configuration parameters, hardware status, housekeeping data} Common RTS/M&C Database

Standard Data Products EOR Data Writing Requirements for 512T Images: HEALPix files, instrumental images (from back end of RTS): 25o radius ; 4 polarizations; 768 channels UV data samples: (from front end of RTS): visibilities for 100[TBR] Tile pairs  Foundation for MWA Calibrations and Data Processing High-Quality Calibrations and Supression of Systematic Errors Calibrator gain measurements; Ionosphere refraction data; Faraday rotation; beam fit parameters EOR Data Products are Basis for 3 Other Science Packages Imaging mode with variable {time, frequency-binning; spatial selection} for Transients Imaging (TLA mode) and GEG Imaging SHI: UV Data Samples + IPC Mode to be written Directly from Correlator Stand-alone processing for Transients discovery (Transient ASM mode) and other data products (e.g., flux-calibrated images; light curves)

MWA Information Processing of EOR Data M&C Housekeeping Configuration Receivers Command Writing mode Digitized Data RTS/RTC Calibration Subtract Calibrated Staging & Solutions Bri. Sources Images Processing Correlator UV Data (40 Gbs) MWA Data Archive (Perth) Catalogs: Calibration Sources Sources Subtracted Raw Data: UV Data [~100 tiles] Science Data: EOR Images Secondary processing Calibration Data: Beam Parameters Ionosphere Params. Flux Images Light Curves (study systematics) M&C MetaData: Configurations Housekeeping Additional MWA Archives (MIT, Melbourne)

MWA Information Processing and Data Products M&C Housekeeping Configuration Receivers Command Writing mode Digitized Data RTS/RTC Calibration Subtract Calibrated Staging & Solutions Bri. Sources Images Processing Correlator UV Data (40 Gbs) MWA Data Archive (Perth) Catalogs: Calibration Sources Sources Subtracted Raw Data: SHI: IPS Data [512] UV Data [~100] (for SHI & EOR) Science Data: EOR Images TLA Images GEG Images Secondary processing Calibration Data: Beam Parameters Ionosphere Params. Flux Images Light Curves Transient ASM Data M&C MetaData: Configurations Housekeeping Additional MWA Archives (MIT, Melbourne)

Archive & Standard Data Products for 128T Save Correlator output (larger Archive)? Built-in Capacity for UV Data analyses, Iterative Calibrations, Comparative Tests, Specialized Tests, Secondary Data Products, etc. Standard Data Products: No Significant Changes  Next efforts: Finalize design and track development of standard data products

Commissioning Phase (Checkout to Verification) Requirements & Verification Document Colin & RR first draft March 2011 Needs Adaptations for 128T Needs Attention and Inputs from Judd, Frank, etc. Gaps and Risk Verification Details are complicated; plans incomplete Insufficient strategy for automation and scripts Upgrade Plan for Monitoring Functions Status screens for hardware & housekeeping data for Installation/Checkout Well-defined metrics and useful error logs Take next step to rudimentary Interferometry monitoring Add smart scripts to monitor for Intermittent problems Examine all data Who? (Curtin Archive & Superscience help? ; MIT student project?)

Operations Phase Operations Management Plan Gaps and Risk Last update: Nov 2010 (Wayne Arcus) Operations succeeds end of Commissioning phase Engineering component – Dave Emrich (Commissioning Engineer) Science Performance Verification – Judd Bowman (Project Scientist) Initial Operations Manager – Tom Booler (Deputy Project Manager for Australia) Assumptions: move toward 24/7 operations ; proposals ; review ; schedule ; observe ; trend analyses ; archive verification ; TBD user support Gaps and Risk Currently No Operations funds Details for Proposals, Target Selection, Scheduling Plans during remainder of 2011 Revise Plan after work on: schedule, cost, monitoring, automation, data products, commissioning plan