1. ALMA Pipeline Heuristics F2F Meeting 2006 in Paris 1 SD Pipeline Heuristics Status Report ALMA Project, NAOJ George KOSUGI

2. ALMA Pipeline Heuristics F2F Meeting 2006 in Paris 2 E2E Pipeline Process Flow Scheduler Archive Offline Telcal ASDM (with raw data) MS2 Filler Result Engines Recipe Scripts Current Development Reference to Science Data Ref. to Calibrator in a necessary condition need to search other calibrator for the sufficient condition? Pipeline Parameters ? Pipeline

3. ALMA Pipeline Heuristics F2F Meeting 2006 in Paris 3 Current SD Pipeline Flow Chart SDFITS Extract Info Extract Data Re-Gridding FlagSpectrum Pointing Pattern Analysis Text Tabl e Text Tabl e numarray Pointing, StartTime, IntegrationTime, etc. BaselineFit Map numarray Integration over Channels > Single Pointing > Multi Pointing > Raster/OTF MS for input data AIPS++Table for Info Storage MS2 MS for output FITS FITS for image

4. ALMA Pipeline Heuristics F2F Meeting 2006 in Paris 4 Environment for SD Heuristics Pipeline  Python Packages –casapy with ASAP –numarray: work area for spectrum data analysis –Scipy: Kmean clustering algorism –Numpy: data for Kmean –PyLab: plotting –PyFits: for collecting info from SDFITS header

5. ALMA Pipeline Heuristics F2F Meeting 2006 in Paris 5 What is necessary for SD UT  MS input –Need sample MS –MSs for casa/ASAP UT in 2007/03 will be adequate  Store/Retrieve info into/from AIPS++Table  MS and FITS output ----------------- 2~3 FTE-w  Calibration script –Investigate TelCal Engine –Investigate scripts used in the casa/ASAP UT ----------------- 2~3 FTE-w ?  Additional plots ----------------- 1~2 FTE-w ?

6. ALMA Pipeline Heuristics F2F Meeting 2006 in Paris 6 Baseline Fit is done for each spectrum one by one  Determine Line Free Channels (if not specified) Determine Line Free Channels (if not specified)  Fit and Subtract Baseline Calculated from the Line Free Channels Fit and Subtract Baseline Calculated from the Line Free Channels

7. ALMA Pipeline Heuristics F2F Meeting 2006 in Paris 7

8. 8 Determine Line Free Channels (if not specified) –Subtract DC (bias) and gradient components –Accumulate several spectra in close vicinity in space to enhance S/N ratio for line detection –Line Detection by application of ASAP.asaplinefinder –Kmean clustering analysis is applied to the detected lines (parameters are line center and line width) –Select real (supposed to be) lines by examining continuities in space, distortion of the vector, density of the detection, and cluster size –Calculate 2-dimensional fit (ra, dec, line center), (ra, dec, line width) to each cluster member –Apply fit to spectra whose position is within the clustering radius

9. ALMA Pipeline Heuristics F2F Meeting 2006 in Paris 9 Fit and Subtract Baseline Calculated from the Line Free Channels –Determine fitting order for polynomial Search gaps in the time sequence fitting order is determined for each bunch of data (between 2 time gaps) fitting order is determined from frequencies of major components in the Fourier space –Calculate Fit to avoid jaggy due to the shortage of the accuracy in the calculation of higher-order power for polynomial, spectrum is divided into sub-spectra, if necessary. ASAP.asapfit was used for the fit, however it didn ’ t converge sometimes. So far python script is used for polynomial fit.

10. ALMA Pipeline Heuristics F2F Meeting 2006 in Paris 10 Spectrum based Flagging (not channel based)  Statistics are measured for each spectrum (emission line channels are excluded): –baseline rms before baseline subtraction after baseline subtraction –difference from the running mean before baseline subtraction after baseline subtraction –may need other criteria Tsys, etc...

11. ALMA Pipeline Heuristics F2F Meeting 2006 in Paris 11 Plot statistics

12. ALMA Pipeline Heuristics F2F Meeting 2006 in Paris 12 Gridding (Align to Lattice pattern)  Beam size and Grid spacing are given  Spectra at positions inside beam radius from the grid center are combined into one spectrum –weight: none, gaussian beam, or cone-shaped beam –combine: median (no weight), weighted mean, NmaxMmin rejected weighted mean

13. ALMA Pipeline Heuristics F2F Meeting 2006 in Paris 13 Line Intensity Map  Accumulate flux in given channels  Subtract flux outside given channels