NRAO VLA Archive Survey National Radio Astronomy Observatory 2008-06-25 SPIE: Observatory Operations: Strategies, Processes, and Systems Jared Crossley Loránt Sjouwerman Edward Fomalont Nicole Radziwill Wednesday, 25 June, 16:00 - 16:20 Our survey team consists of 4 people: I [Jared] provide scientific and software support and handle data analysis. Lorant Sjouwerman is our lead project scientist. Edward Fomalont offers scientific oversight. Nicole Radziwill is our project manager.

NRAO VLA Archive Survey (NVAS) Survey of the VLA Archive generated by an automated data pipeline. Under development since 2006. Data products include: images, calibrated data, diagnostics, logs, … Project ongoing, but data is already available on the Web. http://archive.nrao.edu/ The NRAO VLA Archive Survery, which I will call NVAS, is a survey of the VLA (raw) visibility data archive, generated by an automated data pipeline. The survey has been under development since 2006. The survey contains images, calibrated data, diagnostic plots, and log files. And, this is an ongoing project, but data is already available on the Web at this address.

Very Large Array Synthesis array of 27 antennas Located in New Mexico, USA Operational for 28+ years The instrument from which the raw data comes is the Very Large Array. It is a radio synthesis array located in New Mexico, USA. It was dedicated in 1980, and has been collecting data for over 28 years. The array configuration changes size on regular intervals; the maximum baseline length varies between 0.3 and 35 km. Frequency coverage ranges between 0.07 and 50 GHz. The VLA has been a very productive telescope, producing over 150 papers per year. The instrument outputs visibility measurements; most users want images. So, offline data reduction is necessary. Maximum baseline: 0.3 to 35 km Frequencies: 0.07 to 50 GHz Produces over 150 papers per year Outputs “visibility” measurements

“Typical” VLA Data Reduction Project scientists are responsible for all parts of the data reduction process. Download VLA raw data from the VLA Archive. Making images requires: Careful editing and calibration Fast Fourier transform processing Deconvolution methods To make data more easily accessible we have created the VLA Archive Survey Typically, the project scientist is responsible for all data reduction. The basic steps involved in producing an image are, acquiring the raw data from the archive, editing and calibrating the data, Fourier transforming to the image plane, and then using deconvolution to clean the image. The process can be time consuming, even for experts. To make VLA data more easily accessible to all astronomers, NRAO has created the VLA Archive Survey.

Present State of Survey Number of Images 72839 Unique Sky Positions 15590 Dates 1991 to 2003 Frequencies 1 to 50 GHz Observing Modes Continuum Array Configurations D,C,B, some A Presently, NVAS contains 72000+ images of 15000+ unique positions on the sky. We have processed most of the data between 1991 and 2003. Our frequency coverage is 1 to 50 GHz; below 1 GHz the necessary calibration methods become more difficult to automate; so we avoid the lowest frequency data. So far, we have only imaged continuum data. Later, we plan to image spectral line data too. We process data from all array configurations; because A-array data produces high-resolution images, this data takes longer to process, and we thus are proceeding at a slower rate through the A-array data.

Sky Coverage NVAS sky coverage as of June 2008. The map shows the 15000+ unique sky positions we have imaged.

NVAS Pipeline System Acquire Process Archive Validate 1. Makes use of preexisting components: Raw data archive and interface AIPS synthesis imaging software package Perl and shell scripts connect components to form the pipeline. NVAS data is incorporated in the raw data archive. One interface provides access to raw and processed data. Process 2. Archive 3. Validate 4.

NVAS Pipeline Stages Acquire Data 1. Acquire Perl script automates communication with Archive server by HTTP requests. Input start and stop dates for processing Script queries the server Algorithm selects one or more data files Download to local disk Data is loaded into AIPS, split by frequency, and prepared for processing

Data Processing Acquire Process NVAS Pipeline Stages 1. Acquire AIPS “runfile” (script) performs standard synthesis imaging steps (VLARUN) Output data products: Images Calibrated data Diagnostic plots Log files Metadata for Web interface 2. Process

Example image and diagnostic plots: Crab Nebula at 5 GHz Example image with diagnostic plots: Crab Nebula at 5 GHz.

Archiving Shell and Perl scripts Move data products to archive server NVAS Pipeline Stages Archiving 1. Acquire Shell and Perl scripts Move data products to archive server Generate HTML for survey Web interface 2. Process 3. Archive

Validation Human-interactive process NVAS Pipeline Stages Validation 1. Acquire Human-interactive process A private Web interface is used to remove low quality data. All NVAS data is reviewed by a human. Less than 5% of images have been removed during data validation. 2. Process 3. Archive 4. Validate

Pipeline System Performance Only data validation requires human interaction. All other pipeline stages are limited by computer and network data rates. Data processing requires the most time; high-resolution imaging may take many hours Survey data is 1.4 TB and growing

Access the survey from http://archive.nrao.edu/ Data Vault: 7019-47

Expect to finish these stages end of 2010. Future Development Finish imaging of VLA continuum data At the same time, continue algorithm upgrades: Self-calibration Better data editing Mosaic imaging Composite imaging of long-timescale projects Apply new algorithms to all survey data Process spectral line data Expect to finish these stages end of 2010.

NRAO VLA Archive Survey is… Summary NRAO VLA Archive Survey is… Automated survey of the full VLA archive Makes VLA data more accessible to all astronomers Provides reference images, calibrated data, diagnostics, logs, … Ongoing survey, expected completion in 2010 Currently available at: http://archive.nrao.edu/