1. EOVSA Pipeline Processing System J. McTiernan EOVSA Prototype Review 24-Sep-2012

2. EOVSA pipeline processing: 1)DPP – IDB: The Digital Packaging Processor (DPP) creates the Interim Database (IDB). 2) IDB: MIRIAD datasets, 5 (TBD) minutes of data per dataset (estimated 400 to 500 Mbytes per dataset if 5 minutes based on 60 Gbyte per day estimate, and 12 hours of observing time). If the root data directory is /data, then the dataset name looks like this: /data/eovsa/idb/yyyy/mm/dd/scan_id/one_idb_dataset_yyyymmdd_hhmm The IDB contains minimally-processed data, see 4 EOVSA DPP document for details.

3. EOVSA pipeline processing: 3) IDB – ADB: The initial pipeline jobs are scripts of mostly MIRIAD commands that access the IDB, and create Application Data Bases (ADB). For example, to create a generic ADB, a script, called eovsa_create_adb.sh runs from a cronjob every N (5?) minutes, checking for new data. (If we want to wait until night to do pipeline processing, then we can adjust the crontab file to only run the script at night.) 4) Steps in create_adb script: 1)Check to see is a process is running; a lock file, /data/eovsa/locks/create_eovsa_lock is used to determine whether the process is running (If we want to allow multiple processes, we can create multiple lock files). If the file is present, then we are done. (If the file is old (how old? TBD), then send an email to alert the pipeline management personnel (or SOC, the Science Operations Center)). 2)If there is no lock file, create one. 3)Start processing. The adb results are stored in MIRIAD datasets (1 per scan?) : /data/eovsa/adb/yyyy/mm/dd/one_adb_dataset_yyyymmdd_hhmm

4. EOVSA pipeline processing: 5) Database Management: At each step, we create a database. For each dataset this contains (at least) the data start time, data end time and a status flag. The status flag tells us what step of processing has occurred for the given dataset. Here's a sample from RHESSI: HESSI Filedb File: Created: 2012-09-20T10:43:32.000 Number of Files: 1196 Filename Orb_st Orb_end Start_time End_time Status_flag Npackets Drift_start Drift_end Data source hsi_20120901_002900_003.fits 58017 58018 01-Sep-12 00:29:00 01-Sep-12 00:59:00 1 52180 -4.282 -4.282 Spacecraft:vc1_20120901191339.0,vc3_20120901191339.0... hsi_20120920_144440_000.fits 58314 58315 20-Sep-12 14:44:40 20-Sep-12 16:20:00 -1 273 -4.367 -4.367 Spacecraft:vc1_20120920150711.0 hsi_20120920_144440_001.fits 58314 58315 20-Sep-12 14:44:40 20-Sep-12 16:20:00 0 79146 -4.367 -4.367 Spacecraft:vc1_20120920164750.0,vc3_20120920164750.0 hsi_20120920_162000_000.fits 58315 58316 20-Sep-12 16:20:00 20-Sep-12 17:55:40 0 34130 -4.367 -4.367 Spacecraft:vc1_20120920164750.0,vc3_20120920164750.0 For RHESSI the status flag is 0 for files that have just been created and haven't had the next step (for RHESSI, that's qlook processing), 1 for files that have had the qlook process done, -1 for files that were overwritten with new data.

5. EOVSA pipeline processing: 5) Database Management (cont.): An IDB filedb entry might contain the start and end times, an identifier for the scan, and status_flag(s). We probably want to wait for completed scans before processing, so each database entry would correspond to a single scan, instead of individual IDB datasets. So the eovsa_create_adb script reads the filedb for the IDB, grabs the earliest dataset with status = 0, gets all of the data for that scan, processes, creates an ADB MIRIAD datset, sets the status flag for the IDB filedb entry to 1, creates a database entry for the new ADB dataset, with status = 0, and finishes. Rinse, repeat, for each pipeline process... 6) Ease of reprocessing: To reprocess a scan, access the filedb, set whatever status_flags = 0. We'll also provide an option for alternate reprocessing, via scan_id, which you might want to use

6. EOVSA pipeline processing summary: Interim DB (from DPP) Archive Process Application, File DBs Qlook Spec, FS Process Qlook Spec, Full Sun images The first Process takes the IDB; does time-dependent calibration, time and frequency averaging and creates the first of the Application databases. The next Process takes the ADB files, and creates spectrograms and full Sun images for browsing purposes and for use by the data analysis SW. The start point is the interim database, output from the DPP. This has had time-independent calibrations applied. Application databases are “Science” data, binned in time and frequency for use by the Data Analysis SW - in MIRIAD format (can also do FITS format if people want to use non-MIRIAD SW). We expect multiple datasets, magnetography, bursts, utility, type III. File databases contain information about the time range covered by each Application dataset. Quicklook Spectrograms are dynamic power spectra, functions of frequency and time, that can be used to choose time and frequency intervals by the data analysis SW. Full sun images could be used for choosing positions for high spatial resolution imaging.

7. Burst Process The next process examines the Qlook spectra and attempts to create a list of events. Also in this step, we look for intervals for magnetography, feeding back into the Mag. Database. Information from non-EOVSA sources may be helpful in identifying events. Event List Once we have a list of events, we go back and create more application databases, for flares, type IIIs. Event DBs The event database will be optimized in frequency and time resolution for burst studies. Event Images Images of bursts will hopefully be generated automatically, for distribution, and display on the cool EOVSA web pages. Event Archive Process Event Image Process Mag. DBModels Display

8. The EOVSA GUI is an IDL wrapper that will create MIRIAD scripts, which can be run locally or on a dedicated EOVSA server. The GUI borrows heavily from the RHESSI data analysis software. On initialization, the user chooses an observation time interval, and the dynamic spectrum for that time range is displayed. (Sample is FST data, but plot is simulated. ) The user will have the option of choosing time and frequency ranges graphically, or via pop-up widgets. Other settings, e.g., frequency binning, baseline choices, or frequency masks, are handled by selection widgets. Analysis SW: EOVSA IDL GUI

9. If you click the file pulldown: “Select Observation Time Interval” selects a time interval for which the Olook spectrogram will be plotted in the window. “Retrieve/Process Data” allows for choice of time interval and frequency bands for images and spectra. This is where most of the action is. You choose your parameters, press the “DO” button, MIRIAD commands are generated, and images or spectra are done. The rest of the buttons are for plotting, and exporting data: “Set Plot Preferences”, Sets plotting preferences for screen files and plot files, “Configure Plot File” allows the user to name the file, and output directory. “Print Plot” will print the plot, “Export Data” will allow you to export the data non- graphically, maybe in FITS format. “Exit” means exit. Currently only the Exit button is operational. The rest will be filled in.

10. Once the “DO” button is clicked, the IDL program generates a series of MIRIAD commands, which are written to a file. If the user has MIRIAD on his system, a process is spawned, and maps are created. If the user has no MIRIAD, then there is an option to upload the command file to our server, which will queue and then run the process. The user can choose to be notified via email that his process has been run, or the IDL shell can be configured to automatically look for results. Image maps are saved in (FITS, MIRIAD or FITS or both) format. The EOVSA program will be able to read and display the image files.