WFC3 PIPELINE CDR Jim Rose October 16, 2001OPUS Science Data Processing Space Telescope Science Institute 1 of 13 Science Data Processing

WFC3 PIPELINE CDR Jim Rose October 16, 2001OPUS Science Data Processing Space Telescope Science Institute 2 of 13 OPUS Science Data Processing

WFC3 PIPELINE CDR Jim Rose October 16, 2001OPUS Science Data Processing Space Telescope Science Institute 3 of 13 OPUS Processes  Data Partitioning – segments the telemetry stream into standard EDT dataset – appends multiple dumps into a single exposure file > uses an index file to avoid sorting packets > flags packet errors caught by PACOR  Support Schedule – gathers proposal information from PMDB – test proposals required for development > test version of PMDB must be populated by TRANS

WFC3 PIPELINE CDR Jim Rose October 16, 2001OPUS Science Data Processing Space Telescope Science Institute 4 of 13 OPUS Processes (cont.)  Data Validation – decodes the exposure and engineering parameters in the telemetry and compares them to the planned values – internal header specification (from Ball) > PDB (EUDL.DAT, TDFD.DAT) must be fully populated and defined in DM-06 [DONE] – Planned value comparisons (“flags and indicators”) need to be determined by the Instrument Scientists [09/2002]  World Coordinate System – implements a translation from telescope coordinates through the instrument light-path to an astronomically valid pointing – aperture positions must be defined [09/2002]

WFC3 PIPELINE CDR Jim Rose October 16, 2001OPUS Science Data Processing Space Telescope Science Institute 5 of 13 OPUS Processes (cont.)  Generic Conversion – Generic Conversion outputs uncalibrated data > raw, asn, spt, trl – data will be output in standard FITS format with image or table extensions – primary header will contain keywords inherited by all extensions and a null data array – Extensions > Images grouped by imsets consisting of science array, data quality array, and null error array. > data quality array will be null if no telemetry dropouts

WFC3 PIPELINE CDR Jim Rose October 16, 2001OPUS Science Data Processing Space Telescope Science Institute 6 of 13 OPUS Processes (cont.)  Generic Conversion (cont.) – Required for development > DM-06 to develop algorithms for data formatting [DONE] > keyword definitions (ICD-19) are currently being worked by the Instrument Scientists world coordinate definitions[09/02]world coordinate definitions[09/02] calibration switches and selection criteria[09/02]calibration switches and selection criteria[09/02] calibration file name keywords[09/02]calibration file name keywords[09/02]

WFC3 PIPELINE CDR Jim Rose October 16, 2001OPUS Science Data Processing Space Telescope Science Institute 7 of 13 OPUS Processes (cont.)  Data Collector – OPUS will ensure all component exposures are present before processing further – association table contains information about the product dataset > dataset self-documenting – only associations required for data processing will be constructed in the OPUS pipeline > CR splits > Repeat obs

WFC3 PIPELINE CDR Jim Rose October 16, 2001OPUS Science Data Processing Space Telescope Science Institute 8 of 13 Calibration  OPUS will use STSDAS calibration software  run on ALPHA / Tru64 UNIX platform in operations  expands size of dataset – converts integer raw data to real – possible expansion of data quality array. – expansion of the error array.  Need calibration reference files for testing (at least dummies)

WFC3 PIPELINE CDR Jim Rose October 16, 2001OPUS Science Data Processing Space Telescope Science Institute 9 of 13 Other Science Data Modes  data content & keywords for other science data modes must be defined by Instrument Scientists – microprocessor memory dump[09/02] – engineering diagnostic data[09/02]

WFC3 PIPELINE CDR Jim Rose October 16, 2001OPUS Science Data Processing Space Telescope Science Institute 10 of 13 CODE REUSE  Core OPUS system (OPUS 13.2) – ~90,000 lines of code – 100% reuse  WFC3 specific processes – Based on FUSE study (Rose et al. 1998, “OPUS: The FUSE Data Pipeline”, > 5076 lines of code > 71% reuse of existing OPUS modules (ACS, NICMOS)  Major effort in functional and throughput testing

WFC3 PIPELINE CDR Jim Rose October 16, 2001OPUS Science Data Processing Space Telescope Science Institute 11 of 13 Major Science Data Processing Requirements Summary  Internal header specification – DM-06 to document content and format of science internal header [DONE] – PDB (EUDL.DAT, TDFD.DAT) defined. [DONE]  Keyword definitions [DONE]  Planned values “flags and indicators” for Data Validation  Aperture definitions

WFC3 PIPELINE CDR Jim Rose October 16, 2001OPUS Science Data Processing Space Telescope Science Institute 12 of 13 Test Data Requirements  Test data from detectors on ops bench expected in March 2002 and from integrated instrument in June 2002 – including all science and engineering modes  SMGT & IPT test data must include: – PMDB population and PDB definition – images that enable OPUS to determine proper orientation – list of possible error conditions to simulate – data that simulate those error conditions – enough data for throughput test

WFC3 PIPELINE CDR Jim Rose October 16, 2001OPUS Science Data Processing Space Telescope Science Institute 13 of 13 Thermal Vac Support .SDI files will trigger full dataset name: – ipppssoot_csiiyydddhhmmss – allows reuse of SMS  Optical Stimulus Snapshots – named the same as the image file: csiiyydddhhmmss.oss – allows corresponding OSC data to populate keywords  Support Schedule defaults – since exposures will not have PMDB support, a default support schedule will be used. Auxilliary keyword values can be supplied in a separate exposure-specific ASCII file.