1. NRAO VLA Archive Survey
National Radio Astronomy Observatory
SPIE: Observatory Operations: Strategies, Processes, and Systems
Jared Crossley
Loránt Sjouwerman
Edward Fomalont
Nicole Radziwill
Wednesday, 25 June, 16: :20
Our survey team consists of 4 people: I [Jared] provide scientific and software support and handle data analysis. Lorant Sjouwerman is our lead project scientist. Edward Fomalont offers scientific oversight. Nicole Radziwill is our project manager.

2. NRAO VLA Archive Survey
(NVAS)
Survey of the VLA Archive generated by an automated data pipeline.
Under development since 2006.
Data products include: images, calibrated data, diagnostics, logs, …
Project ongoing, but data is already available on the Web.
The NRAO VLA Archive Survery, which I will call NVAS, is a survey of the VLA (raw) visibility data archive, generated by an automated data pipeline. The survey has been under development since The survey contains images, calibrated data, diagnostic plots, and log files. And, this is an ongoing project, but data is already available on the Web at this address.

3. Very Large Array Synthesis array of 27 antennas
Located in New Mexico, USA
Operational for 28+ years
The instrument from which the raw data comes is the Very Large Array. It is a radio synthesis array located in New Mexico, USA. It was dedicated in 1980, and has been collecting data for over 28 years. The array configuration changes size on regular intervals; the maximum baseline length varies between 0.3 and 35 km. Frequency coverage ranges between 0.07 and 50 GHz. The VLA has been a very productive telescope, producing over 150 papers per year. The instrument outputs visibility measurements; most users want images. So, offline data reduction is necessary.
Maximum baseline: 0.3 to 35 km
Frequencies: 0.07 to 50 GHz
Produces over 150 papers per year
Outputs “visibility” measurements

4. “Typical” VLA Data Reduction
Project scientists are responsible for all parts of the data reduction process.
Download VLA raw data from the VLA Archive.
Making images requires:
Careful editing and calibration
Fast Fourier transform processing
Deconvolution methods
To make data more easily accessible we have created the VLA Archive Survey
Typically, the project scientist is responsible for all data reduction. The basic steps involved in producing an image are, acquiring the raw data from the archive, editing and calibrating the data, Fourier transforming to the image plane, and then using deconvolution to clean the image. The process can be time consuming, even for experts. To make VLA data more easily accessible to all astronomers, NRAO has created the VLA Archive Survey.

5. Present State of Survey
Number of Images
72839
Unique Sky Positions
15590
Dates
1991 to 2003
Frequencies
1 to 50 GHz
Observing Modes
Continuum
Array Configurations
D,C,B, some A
Presently, NVAS contains images of unique positions on the sky. We have processed most of the data between 1991 and Our frequency coverage is 1 to 50 GHz; below 1 GHz the necessary calibration methods become more difficult to automate; so we avoid the lowest frequency data. So far, we have only imaged continuum data. Later, we plan to image spectral line data too. We process data from all array configurations; because A-array data produces high-resolution images, this data takes longer to process, and we thus are proceeding at a slower rate through the A-array data.

6. Sky Coverage
NVAS sky coverage as of June The map shows the unique sky positions we have imaged.

7. NVAS Pipeline System Acquire Process Archive Validate1.
Makes use of preexisting components:
Raw data archive and interface
AIPS synthesis imaging software package
Perl and shell scripts connect components to form the pipeline.
NVAS data is incorporated in the raw data archive. One interface provides access to raw and processed data.
Process 2.
Archive 3.
Validate 4.

8. NVAS Pipeline Stages
Acquire Data 1.
Acquire
Perl script automates communication with Archive server by HTTP requests.
Input start and stop dates for processing
Script queries the server
Algorithm selects one or more data files
Download to local disk
Data is loaded into AIPS, split by frequency, and prepared for processing

9. Data Processing Acquire Process NVAS Pipeline Stages1.
Acquire
AIPS “runfile” (script) performs standard synthesis imaging steps
(VLARUN)
Output data products:
Images
Calibrated data
Diagnostic plots
Log files
Metadata for Web interface 2.
Process

10. Example image and diagnostic plots:
Crab Nebula at 5 GHz
Example image with diagnostic plots: Crab Nebula at 5 GHz.

11. Archiving Shell and Perl scripts Move data products to archive server
NVAS Pipeline Stages
Archiving 1.
Acquire
Shell and Perl scripts
Move data products to archive server
Generate HTML for survey Web interface 2.
Process 3.
Archive

12. Validation Human-interactive process
NVAS Pipeline Stages
Validation 1.
Acquire
Human-interactive process
A private Web interface is used to remove low quality data.
All NVAS data is reviewed by a human.
Less than 5% of images have been removed during data validation. 2.
Process 3.
Archive 4.
Validate

13. Pipeline System Performance
Only data validation requires human interaction.
All other pipeline stages are limited by computer and network data rates.
Data processing requires the most time; high-resolution imaging may take many hours
Survey data is 1.4 TB and growing

14. Access the survey from http://archive.nrao.edu/
Data Vault:

15. Expect to finish these stages end of 2010.
Future Development
Finish imaging of VLA continuum data
At the same time, continue algorithm upgrades:
Self-calibration
Better data editing
Mosaic imaging
Composite imaging of long-timescale projects
Apply new algorithms to all survey data
Process spectral line data
Expect to finish these stages end of 2010.

16. NRAO VLA Archive Survey is…
Summary
NRAO VLA Archive Survey is…
Automated survey of the full VLA archive
Makes VLA data more accessible to all astronomers
Provides reference images, calibrated data, diagnostics, logs, …
Ongoing survey, expected completion in 2010
Currently available at: