Data Management Subsystem Jeff Valenti (STScI)

DMS Context PRDS - Project Reference Database PPS - Proposal and Planning OSS - Operations Scripts FOS - Flight Operations WSS - WFS&C Software DMS - Data Management Science & Operations Center Ground Segment JWST Element Segment Mission Subsystem

Science Data Flow - The Big Picture Focal Plane Array Application Specific I ntegrated Circuit Focal Plane Electronics Focal Plane Array Processor Solid State Recorder Deep Space Network NASA I ntegrated Services Network Flight Operations Subsystem Data Management Subsystem High Gain Antenna ISIM Spacecraft Common Systems Science & Operations Center Observatory Ground Segment Element

What is the DMS? “The DMS provides the data processing, archive, catalog, calibration, distribution and analysis functions required to support the science program and maintenance of Observatory performance.” Ground Segment Requirements Document (JWST-RQMT-1056, Level 2)

Mission Operations Concept Document (JWST-OPS ) General MO-599Archive science data from JWST. Distribute data to users. MO-601Backup data onto removable media. Store offsite. Engineering Data MO-602Do not processes engineering data (handled by FOS). MO-603Obtain processed engineering data from FOS. Engineering Images MO-609Archive guide star acquisition images. MO-1086Calibrate wavefront sensing images within 3 hours. MO-1089Send calibrated wavefront sensing images to WSS.

Mission Operations Concept Document (JWST-OPS ) Science Data MO-600Process and archive science data. Notify users when processed science data are available. MO-602Defines “data processing levels” for science data. MO-604Receive science data from FOS, process, and archive. Upon request retrieve, process, and distribute science data. MO-605Remove instrumental signatures and observation artifacts. MO-606Make science data available within 5 days of observation. Archive raw data (229 Gbits/day) plus derived data. Archive users must register. Protect proprietary science data. MO-607Process science data upon receipt. Verify data integrity. Record pipeline parameters and reference file names. Archive metadata (preview, object lists, NIRSpec image map). Upon request, reprocess data “on-the-fly”. Deliver FITS files. Protect proprietary data using archive accounts. Distribute public data to mirror archives.

Mission Operations Concept Document (JWST-OPS ) Science Data (cont.) MO-608Recorded science data identifies exposure and observation. Use timestamps to merge engineering data with science data. MO-610Decompress, calibrate, format, and archive science data. Assess dataset completeness. Report incomplete processing. Produce calibrated science data suitable for analysis. Notify users when processed science data are archived. Distribute data via network, unless physical media requested. Store data online to allow “quick” data access. MO-611Base initial pipeline reference files on ground tests. Calibration scientists use flight data to update reference files. Use best reference files in pipeline and provide to community.

Data Processing Levels Level 0 Recorded Science Data - Downlinked files containing packets of compressed pixel data, as stored by ISIM on the SSR. Data from a particular integration may be spread across multiple files. Level 1 Raw Exposure Data – FITS file containing up-the-ramp image data for one detector and one exposure, which may consist of multiple integrations. Relevant packets are extracted from recorded science data files, uncompressed, and assembled into one image sequence per integration. Engineering telemetry and calibration keywords are added to the FITS header. Level 2 Count Rate Images – Up-the-ramp image sequences are fitted to create a count rate image, an uncertainty image, and a data quality image for each integration. The slope fitting procedure removes cosmic rays, bias drifts, and dark current. Level 3 Processed Science Exposures – Count rate images are processed to remove instrumental signatures, extract spectra, and convert to physical units. Measurements from contemporaneous calibration exposures (e.g., science target locations in the reference image) are used when creating processed science exposures. Data from multiple integrations are combined. Level 4 Combined Science Exposures – Associated science exposures (e.g., dithers) are combined.

DMS Data Flow Recorded Science Data Unpacking Science Header Creation Recorded Science Data Collection FOS Calibration Metadata Creation Up-The- Ramp Processing Science Exposure Processing Science Exposure Combination PPS / DS FOS / EDW Science Data Archive Science Header Service Science Data Pipeline Pipeline Executive (and OTFR) Backup Manager Calibration Metadata Update Calibration Reference Database Calibration Data Processing Science Header Update Calibration Metadata Update Level 3 Processed Science Exposures Level 4 Combined Science Exposures Level 2 Count Rate Images Level 1 Raw Exposure Data Level 0 Recorded Science Data Science Header Update OTFR 1 OTFR 2 OTFR 3 0 Science Archive Interface Reference File Creation Reference File Server Science Data Ingest

Jeff’s Rule for DMS Design Reusing the Hubble DMS design and tools will usually be easier and faster (and hence cheaper) than creating new tools.

Reasons to Modify HST DMS JWST requirements not met by HST DMS design New modes, e.g. automatically extract multi-object spectra New data models, e.g. engineering data warehouse More data, e.g. can we routinely distribute raw exposure data? Component of HST DMS will become obsolete Rewrite synphot in Python because SPP is obsolete [in progress] Reduce overall cost of operating JWST Automatic calibration data processing (e.g., dark processing) Update science headers and calibration metadata via server Improve value of data products and services Source lists, object classification, etc. [e.g., Hubble Legacy Archive]