1. Archive, Data Products, & Operations for the Murchison Widefield Array
Ron Remillard (MIT); MWA Project Meeting, June 6, 2011

2. Outline Resources for the MWA Archive at MIT Standard Data Products
NSF MRI Program
Standard Data Products
Incomplete definitions in early 2011
Strategy for progress
Information Flow Diagram
Commissioning Phase
Needs for Commissioning Plan
Needs for more sophisticated Monitoring
Operations Phase
Operations Plan and Needs for Update
Plans for remainder of 2011

3. MWA Archive at MIT NSF MRI Program ($868 K: NSF + MIT contribution)
Large equipment budget
Advanced spending on archive requirements, data products
Need to set aside majority of funds until late 2012
Archive Tasks (based on EoR, systematic error analyses, student projects)
Capture raw data (in short bursts)
Intermediate data capture (non-calibrated images; less resolution (n, t))
High timer resolution calibrated data
Determine calibrations in parallel with RTS
Calibrate data offline
Study ionosphere parameters at 2 s
Service MWA EoR Consortium, other EoR users, other outside projects

4. MWA Archive at MIT Gaps in Plan
Redefine SOW for 128T (retain EoR focus; gain approval for NSF extensions)
Complete Archive Requirements Document (project-wide scope)
Method to obtain data from Curtin
Method to provide data access for Users

5. Standard Data Products
Science Data
Eor Data Products (calibrated raw images) defined in MRI proposal
SHI Needs raw data (pro-Correlator) and UV data from outlier tiles
Transients & GEG groups need Images and other modes
(light curves, transient searches (image diffs), wider images (low t res.), others?)
“Meta Data”
RTS {calibrator catalogs, calibrator solutions, ionospheric params., “peeled” sources}
M&C {configuration parameters, hardware status, housekeeping data}
Common RTS/M&C Database

6. Standard Data Products
EOR Data Writing Requirements for 512T
Images: HEALPix files, instrumental images (from back end of RTS):
25o radius ; 4 polarizations; 768 channels
UV data samples: (from front end of RTS):
visibilities for 100[TBR] Tile pairs
 Foundation for MWA Calibrations and Data Processing
High-Quality Calibrations and Supression of Systematic Errors
Calibrator gain measurements; Ionosphere refraction data;
Faraday rotation; beam fit parameters
EOR Data Products are Basis for 3 Other Science Packages
Imaging mode with variable {time, frequency-binning; spatial selection} for Transients Imaging (TLA mode) and GEG Imaging
SHI: UV Data Samples + IPC Mode to be written Directly from Correlator
Stand-alone processing for Transients discovery (Transient ASM mode)
and other data products (e.g., flux-calibrated images; light curves)

7. MWA Information Processing of EOR Data
M&C
Housekeeping Configuration
Receivers
Command
Writing mode
Digitized Data
RTS/RTC
Calibration Subtract Calibrated Staging &
Solutions Bri. Sources Images Processing
Correlator
UV Data (40 Gbs)
MWA Data Archive (Perth)
Catalogs:
Calibration Sources
Sources Subtracted
Raw Data:
UV Data [~100 tiles]
Science Data:
EOR Images
Secondary
processing
Calibration Data:
Beam Parameters
Ionosphere Params.
Flux Images
Light Curves
(study systematics)
M&C MetaData:
Configurations
Housekeeping
Additional MWA Archives (MIT, Melbourne)

8. MWA Information Processing and Data Products
M&C
Housekeeping Configuration
Receivers
Command
Writing mode
Digitized Data
RTS/RTC
Calibration Subtract Calibrated Staging &
Solutions Bri. Sources Images Processing
Correlator
UV Data (40 Gbs)
MWA Data Archive (Perth)
Catalogs:
Calibration Sources
Sources Subtracted
Raw Data:
SHI: IPS Data [512]
UV Data [~100]
(for SHI & EOR)
Science Data:
EOR Images
TLA Images
GEG Images
Secondary
processing
Calibration Data:
Beam Parameters
Ionosphere Params.
Flux Images
Light Curves
Transient ASM Data
M&C MetaData:
Configurations
Housekeeping
Additional MWA Archives (MIT, Melbourne)

9. Archive & Standard Data Products for 128T
Save Correlator output (larger Archive)?
Built-in Capacity for UV Data analyses, Iterative Calibrations,
Comparative Tests, Specialized Tests, Secondary Data Products, etc.
Standard Data Products: No Significant Changes
 Next efforts: Finalize design and track development of
standard data products

10. Commissioning Phase (Checkout to Verification)
Requirements & Verification Document
Colin & RR first draft March 2011
Needs Adaptations for 128T
Needs Attention and Inputs from Judd, Frank, etc.
Gaps and Risk
Verification Details are complicated; plans incomplete
Insufficient strategy for automation and scripts
Upgrade Plan for Monitoring Functions
Status screens for hardware & housekeeping data for Installation/Checkout
Well-defined metrics and useful error logs
Take next step to rudimentary Interferometry monitoring
Add smart scripts to monitor for Intermittent problems
Examine all data
Who? (Curtin Archive & Superscience help? ; MIT student project?)

11. Operations Phase Operations Management Plan Gaps and Risk
Last update: Nov 2010 (Wayne Arcus)
Operations succeeds end of Commissioning phase
Engineering component – Dave Emrich (Commissioning Engineer)
Science Performance Verification – Judd Bowman (Project Scientist)
Initial Operations Manager – Tom Booler (Deputy Project Manager for Australia)
Assumptions: move toward 24/7 operations ; proposals ; review ; schedule ;
observe ; trend analyses ; archive verification ; TBD user support
Gaps and Risk
Currently No Operations funds
Details for Proposals, Target Selection, Scheduling
Plans during remainder of 2011
Revise Plan after work on:
schedule, cost, monitoring, automation, data products, commissioning plan