1 ANASAC Meeting – May 20, 2015 ALMA Pipeline Brian Glendenning (for Jeff Kern)

2 ANASAC Meeting – May 20, 2015 ALMA Pipeline Status Pipeline provisional acceptance May 2014 – Extensive punch list primarily on weblog and presentation issues Calibration pipeline put into operation Oct 2014 – Detailed parallel testing by ARC staff prior to operations We now have approximately 6 months of operational experience with the calibration pipeline

3 ANASAC Meeting – May 20, 2015 ALMA Pipeline Pipeline Performance and Use Cycle 1Cycle 2

4 ANASAC Meeting – May 20, 2015 ALMA Pipeline Pipeline Performance

5 ANASAC Meeting – May 20, 2015 ALMA Pipeline Manual Flagging by Reason

6 ANASAC Meeting – May 20, 2015 ALMA Pipeline Path Forward Internal Only Cycle 3 R1 (Jan 2015): – Cleanup of capabilities and feedback from early cycle2 – Infrastructure for automation – Bandwidth mapping for low SNR issues Cycle 3 R2 (~May 29) – Imaging Mechanics – Minor flagging changes – Infrastructure for Sessions Pipeline Face 2 Face: June – Pipeline WG (ISOpt) and Developers – Priority for imaging improvements – Total Power Pipeline Commissioning Status and Plans

7 ANASAC Meeting – May 20, 2015 ALMA Pipeline Path Forward (Continued) Cycle 3-R3 (Sept 2015) – Pipeline for use for cycle 3 – Including initial imaging – Increasing levels of testing leading to parallel testing of imaging and full acceptance.

8 ANASAC Meeting – May 20, 2015 ALMA Pipeline Planning Post Cycle 3 – Software engineering issues Automated testing Documentation – Expanded Scope Long Baselines Imaging Improvements Sessions Band Switching Calibration Low calibrator SNR Heuristics Polarization

9 ANASAC Meeting – May 20, 2015 ALMA Pipeline Steady State New capabilities initially offered in non-standard mode – Provides reasonable sample of data sets – Manual data reduction provides “best practice” heuristics – Comparison to manual reduction for pipeline validation. Move to standard mode in following cycle – Depends on how standard the mode is VLBI, Solar, Time Domain may never become standard Feedback from Data Reduction and Pipeline Operations – Improve heuristics and coverage

10 ANASAC Meeting – May 20, 2015 ALMA Pipeline Risks Image quality – Diverse expectations – Must deliver some capabilities to relive data reduction load on ARCs Sociology – Pipeline produced science products are new to radio interferometry, community acceptance will be mixed. – Errors in early days will have long lasting effects on confidence in the pipeline

11 ANASAC Meeting – May 20, 2015 ALMA Data Mining Toolkit (ADMIT) Update: May 11, 2015 Milestones #1 and #2 successfully completed. Task and data i/o infrastructure Work flow management (allows scripting of ADMIT tasks) ADMIT tasks: ingest, file, cubestats, cube spectrum, line-id, linecube, moment Automated build and regression testing Data browser prototype (url: carma.astro.umd.edu/admit-M2) On-line documentation: design, code docs, and user manuals (url: carma.astro.umd.edu/admit) Prototype pipeline for deployment on ALMA pipeline machines Tests performed on Cycle 0 and Cycle 1 data Milestone #3: July 31, 2015 Production of ADMIT pipeline data products on ALMA machines Ingestion of data products into archive Broad testing against Cycle 1 and 2 data Expansion of ADMIT tasks and exploration of new algorithms Creates data products for the archive which can be viewed and manipulated by the user. Data products can also be generated on local datasets using ADMIT/casa python environment. Data products are compatible for use by future ALMA archive enhancements

12 ANASAC Meeting – May 20, 2015 ALMA Data Mining Toolkit (ADMIT) Prototype ADMIT Data browser view displaying Cycle 1 NGC 253 data Chose web browser interface to minimize barrier for user adoption. Built on same browser resources as ALMA Pipeline Viewer to maintain consistent look and feel for user. Automatically updates as new data products are created locally. Allows organization data products by source using tabs Mitigates cross-platform incompatibilities that could arise from a custom GUI

13 ANASAC Meeting – May 20, science.nrao.edu The National Radio Astronomy Observatory is a facility of the National Science Foundation operated under cooperative agreement by Associated Universities, Inc.