COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 1 of 24 Science Data Processing

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 2 of 24 General OPUS Requirements for Science Data Processing  OPUS will develop the COS pipeline based on the existing pipeline model including OTFR.  Level 0 data packetized and Reed-Solomon corrected by PACOR at GSFC  Receive science telemetry (level 1a data) at STScI as science instrument specific ‘pod files’ – Engineering snapshot included – No on-board compression for COS data  STScI processing on Compaq ALPHA/Tru64 UNIX platform

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 3 of 24 General OPUS Requirements for Science Data Processing (cont.)  OPUS must account for all scheduled exposures.  Convert telemetry to FITS format – Structure tables or data array – Populate header keywords > Keywords to provide metadata for archive catalog  Associate groups of exposures that must process further as a single unit  Execute calibration tasks in pipeline mode  Pass level 1b science data (pod files and uncalibrated science datasets) and jitter files to Hubble Data Archive

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 4 of 24 OPUS Requirements for Thermal Vac  OPUS will develop a COS science data processing pipeline capable of supporting Thermal Vac testing  No database access in Thermal Vac – No support schedule available in PMDB format – Necessary support schedule information to be read in from ASCII file  No associations  No calibration  Data will be archived to HDA  Processing on Sun / Solaris UNIX platform – Need to know which version of Solaris will be used?

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 5 of 24 OPUS Requirements for Thermal Vac (cont.) Thermal Vac OPUS pipeline delivery schedule – Earliest possible COS Thermal Vac currently scheduled for December 10, 2001 – OPUS Thermal Vac pipeline due about two months prior to Thermal Vac, September 24, – Beta version of OPUS pipeline due about five months prior to Thermal Vac, June 25, 2000.

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 6 of 24 OPUS science data processing pipeline for COS

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 7 of 24 OPUS Processes  Data Partitioning – segments the telemetry stream into standard EDT dataset – fill data inserted if telemetry drop-outs exist > event lost in Time-Tag mode > constructs a data quality image in ACCUM mode to ensure the subsequent science processing does not interpret fill data as valid science data  Support Schedule – gathers proposal information from PMDB – test proposals required for development > test version of PMDB must be populated by TRANS > Thermal Vac support schedule to be input from ASCII file

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 8 of 24 OPUS Processes (cont.)  Data Validation – decodes the exposure and engineering parameters in the telemetry and compares them to the planned values – internal header specification (from Ball) > PDB (EUDL.DAT, TDFD.DAT) must be fully populated and defined in DM-06 – flags and indicators need to be determined by the Instrument Scientists  World Coordinate System – implements a translation from telescope coordinates through the instrument light-path to an astronomically valid pointing – aperture positions must be defined

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 9 of 24 OPUS Processes (cont.)  Generic Conversion – Generic Conversion outputs uncalibrated data > rtg (Time-Tag); raw, rs1, rs2 (ACCUM); spt; trl – data will be output in standard FITS format with image extensions – primary header will contain keywords inherited by all extensions and a null data array – Image extensions > Time-Tag mode will be FITS binary tables > ACCUM mode will be images grouped by imsets consisting of science array and data quality array FUV data will contain separate imsets for each detector (a and b)FUV data will contain separate imsets for each detector (a and b) data quality array will be null if no telemetry dropoutsdata quality array will be null if no telemetry dropouts –calibration generates full data quality array with all other DQ flags

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 10 of 24 OPUS Processes (cont.)  Generic Conversion (cont.) – Required for development > DM-06 to develop algorithms for data formatting > keyword definitions (ICD-19) must be provided by the Instrument Scientists world coordinate definitionsworld coordinate definitions exposure time calculationsexposure time calculations calibration switches and selection criteriacalibration switches and selection criteria calibration file name keywordscalibration file name keywords

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 11 of 24 Keyword specification – keyword name – default value – possible values – units – datatype – short comment for header – long description – header position – DADS table – keyword source The following information must be provided by STScI Science Instrument team for all COS specific keywords using a standard form for keyword database input.

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 12 of 24 Science Instrument Configuration Keywords  Keyword: some possible values – OBSTYPE: Spectroscopy / Imaging – OBSMODE: ACCUM / TIME-TAG – EXPTYPE: ACQ, PEAKUP/XDISP, PEAKUP/DISP, SCI, WAVE, FLAT, DARK, PHA – DETECTOR: FUV, NUV – OPT_ELEM: G130M, G160M, G185M, G225M, G285M, G140L, G230L, MIRROR, MIRRORB – CENWAVE: 1298, 1309, 1320, etc. – APERTURE : PSA, BOA, WCA, FCA (Indicates a COS unique keyword or value)

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 13 of 24 OPUS Processes (cont.)  Data Collector – association table contains all information about product dataset > dataset self-documenting – only associations required for data processing with be constructed in the OPUS pipeline > FP splits > Repeat obs – OPUS will ensure all component exposures are present before processing further > association time-out rules need to be defined by Instrument Scientists > rules for processing incomplete associations need to be defined by Instrument Scientists

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 14 of 24 Wavecal considerations  Current plans for executing wavecals – Wavecal executed after each grating, central wavelength, mirror, or aperture change – Wavecal executed at least once each orbit  Wavecal exposures will be shared between associations – A single wavecal may go into multiple products > For example, wavecal at start of orbit can be shared by all observations in that orbit if it is applicable – Adds complexity to science data processing pipeline

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 15 of 24 Calibration  OPUS will use STSDAS calibration software  run on ALPHA / Tru64 UNIX platform in operations  expands size of dataset – converts integer raw data to real – possible expansion of data quality array for ACCUM mode  Need calibration reference files for testing (at least dummies)

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 16 of 24 Other Science Data Modes  requirements for data content of each of these other science data modes must be defined by Instrument Scientists – target acquisitions – microprocessor memory dump – engineering diagnostic data – cumulative detector image > MAMA cumulative image like STIS for NUV > Possible cumulative image for FUV detector

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 17 of 24 Engineering data processing  Receive engineering telemetry data from CCS at GSFC  Process Engineering data through FGS Data Pipeline  Generate data products to characterize jitter and pointing control information in support of science observations  COS jitter file association packaging will mimic science data associations  No other COS specific requirements

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 18 of 24 Archive Interface  OPUS will present to the archive: – Original data received from PACOR (binary pod files) – Possibly pod file data packaged by observation in FITS format – Output of Generic Conversion (uncalibrated science dataset) in FITS format – Jitter files in FITS format from the engineering telemetry – Data from other science modes (target acquisition, memory dump, engineering diagnostic data, cumulative MAMA image) in FITS format

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 19 of 24 OTFR  In OTFR, data retrieved from the archive are reprocessed from the pod file – Provides HST data user with optimal product at time of retrieval > Calibration updates, bug fixes, and new software features and algorithms available to archive users – OTFR pipeline uses the exact same code as current initial science data processing > Reduces software development and maintenance costs > No science instrument specific code developed for OTFR beyond what is necessary for initial data processing – Adds negligible time for retrievals

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 20 of 24 OTFR

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 21 of 24 CODE REUSE  Core OPUS system (OPUS 12.1) – ~236,000 lines of code – 100% reuse  COS specific processes – Based on FUSE study (Rose et al. 1998, “OPUS: The FUSE Data Pipeline”, > 5076 lines of code > 71% reuse of existing OPUS modules  Expect > 99% reuse of existing data processing software for COS, based on lines of code. – All SI complexity contained in relatively few lines of code.  Efficient use of existing system!

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 22 of 24 Operational Considerations  No throughput issues anticipated based on COS data processing requirements  OPUS sizing based on SSR limitations in ACS era – COS data volumes should be easily accommodated Hence,  OPUS pipeline throughput for COS could be handled with existing operational environment  No new system requirements for running CALCOS in OPUS  No additional hardware specifications

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 23 of 24 Major Science Data Processing Requirements Summary  Internal header specification – DM-06 to document content and format of science internal header – PDB (EUDL.DAT, TDFD.DAT) defined and fully populated  Keyword definitions  Flags and indicators for Data Validation  Aperture definitions

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 24 of 24 Test Data Requirements  Test data from detectors on optical bench expected in March 2001 and from integrated instrument in August 2001  Test data to be provided by IDT/Instrument Scientists and Engineers should include all science modes  Test data must include – PMDB population and PDB definition – spectra that enable OPUS to determine proper orientation – list of possible error conditions to simulate – data that simulate error conditions – enough data for throughput test – engineering data to test jitter file production