1. JWST Calibration Pipeline Software Requirements Walkthrough June 10, 2010 Robert Jedrzejewski Science Software Branch/STScI

2. Calibration Software Requirements Walkthrough 2 Purpose of this Walkthrough Demonstrate to the JWST project that STScI can specify and architect a system to calibrate all JWST data Specify: - WHAT the calibration pipeline software system shall do Architect: - What pieces the system will need and how will they fit in with the rest of the Data Management Subsystem

3. Calibration Software Requirements Walkthrough 3 Purpose… Calibrate: Calibration Calibrated data Raw data

4. Calibration Software Requirements Walkthrough 4 JWST Mission System Tree JWST Mission Observatory Segment Ground Segment Launch Segment Spacecraft Integrated Science Instrument Module Science & Operations Center Common Systems Institutional Systems Launch Site Services Payload Adaptor Launch Vehicle Optical Telescope Element Data Management Subsystem (DMS) Proposal Planning Subsystem (PPS) Flight Operations Subsystem (FOS) Project Reference Database Subsystem (PRDS) Mission Subsystem Element Segment WFS&C Software Subsystem (WSS) Data Processing Calibration Pipeline Data Ingest Archive Component

5. Calibration Software Requirements Walkthrough 5 Calibration Overview Calibration is the process whereby the imperfect data from the JWST detectors is improved by the application of algorithms and reference files that quantify our understanding of the detectors and instruments: Raw JWST data suffers from many effects that are introduced everywhere along the observatory chain - Science target photons are accompanied by unwanted background signals - OTE does not reflect all the photons, shapes the Point-Spread Function (PSF), introduces geometric distortion and adds to the unwanted background - Instruments do not propagate all the desired photons to the detector, introduce their own geometric distortion and adds their own background signal - Detectors do not detect all photons, add background, sub-sample the data, and add both Poisson and cosmetic noise, as well as efficiently detecting cosmic rays SI Teams and Instrument Scientists characterize the instruments by analyzing data taken under specific conditions - This allows them to model the behavior of the instruments - They develop algorithms for correcting each of the unwanted effects - They generate Calibration Reference Files that quantify the unwanted effects The combination of algorithms and reference files provides a way to ‘undo’ many of the undesired effects introduced in the observatory It is the job of the calibration software to apply the calibration algorithms using the necessary calibration reference files

6. Calibration Software Requirements Walkthrough 6 Motivation for Calibrating the Data Most astronomers are not experts on instruments or detectors Nor should it be necessary for them to be Doing the calibration in one place ensures consistent, well- documented products Also avoids duplication of effort Having STScI scientists involved in the calibration process ensures that the required expertise is developed to enable the support of the user community A well-developed and tested calibration pipeline available when the first science data are collected allows for quick turnaround of scientific results, and improves the quality of science proposed for in later cycles The limited duration of the mission means that we cannot afford to spend a significant fraction of the early mission years figuring out how to calibrate the data

7. Calibration Software Requirements Walkthrough 7 More Calibration Discussion Calibration is not ‘all or nothing’ – it is a process of continuous improvement of data quality that starts before the instruments are designed, continues through fabrication, testing, launch, commissioning, science operations and post-Mission Calibration does not directly affect Mission Operations, it primarily interacts with the Science Requirements Although the use of calibrated products in the Wavefront Sensing and Control element is a Mission Critical aspect Understanding the calibration needs can help STScI ensure that adequate calibration data is collected If we are to meet the Science Requirements, it presupposes that the necessary level of calibration can be achieved Can we calibrate the data to achieve the consistency and predictability of results for standard targets to ensure that data for science targets can be held to the same level of confidence? Do our algorithms model the observatory and does the calibration software execute the algorithms?

8. Calibration Software Requirements Walkthrough 8 What is Calibrated Data? Working definition: Calibrated data is what would have been received if an ideal JWST with ideal instruments and ideal detectors had observed the chosen field in the chosen observing mode. Using calibrated data it will be possible to - Determine the flux incident on each pixel, along with its uncertainty (relative and absolute flux determinations limited by different effects) - Determine the relative locations of each pixel to very high accuracy (limited by knowledge of the geometric distortions in the telescope + instruments) - Determine the absolute locations of the observed field to high accuracy (limited by absolute Guide Star accuracies) - Determine what effects may have contributed to degraded calibration quality for each pixel

9. Calibration Software Requirements Walkthrough 9 What are “Requirements”? JWST has 4 science instruments, each of which can be used in several ways for science (“Modes”) NIRSpec Multi-Object Spectroscopy Long-slit spectroscopy Integral Field Spectroscopy MIRI Imaging Coronography Long-slit Spectroscopy Integral Field Spectroscopy FGS-TFI Imaging Coronography Non-redundant Mask Imaging NIRCam Imaging Coronography Grism Spectroscopy Pupil Imaging

10. Calibration Software Requirements Walkthrough 10 A Deeper Look at Calibration All JWST data is a representation of a celestial scene, as processed by the OTE and instrument to create an image on the detector The detector turns that image into a digital dataset (imperfectly!) JWST OTE Instrument Detector Data Celestial Scene Image formed by telescope Image/spectrum formed by instrument

11. Calibration Software Requirements Walkthrough 11 Modes Each Mode is just a different way that a JWST instrument renders the celestial scene onto the detector(s) Imaging modes transfer the whole scene to the detector through a filter that isolates a specific range of wavelengths Spectroscopic modes disperse the light from part of the scene and transfer the resulting spectra onto the detector Coronographic modes are like imaging modes, except that part of the scene is occulted and the point-spread function is changed to increase the contrast between the wings of the PSF and neighboring targets But all Modes involve reading the detector(s) There is a natural separation of the calibration into 2 parts: detector calibration (Mode independent) and Instrumental calibration (Mode dependent)

12. Calibration Software Requirements Walkthrough 12 Decomposing the Calibration Pipeline Detector Calibration Instrument Calibration Imaging Calibration Spectroscopic Calibration Coronographic Calibration

13. Calibration Software Requirements Walkthrough 13 More on Decomposing the Pipeline The Spectroscopic Calibration can be further decomposed Multi-slit calibration Longslit calibration Integral-field calibration Grism calibration Decomposing the pipeline into Mode-specific behavior is preferable to decomposing it into Instrument-specific behavior Calibrating similar Modes will involve sharing much of the code Even though each instrument implements the behavior slightly differently, there will be many primitive operations that will be shared - E.g. Re-mapping a spectrum into linear space, correcting for aperture throughput…

14. Calibration Software Requirements Walkthrough 14 Detector Calibration The detector calibration is simplified because we will only have 2 types of detectors on JWST 15 Hawaii 2RG HgCdTe detectors on NIRCam, NIRSpec, FGS 3 Ga:As detectors on MIRI Even so, the description of the calibration steps will be the same E.g., Both types of detectors have a Reference Pixel Correction step But the best algorithm for doing the correction will often be different It’s unlikely that the Reference Pixel Correction will be the same for Ga:As and HgCdTe detectors And often the same The Nonlinearity Correction algorithm may well be the same for all detectors – we’ll see

15. Calibration Software Requirements Walkthrough 15 Calibration Software Architecture The JWST Calibration Pipeline will inherit many of the successful architectural features from the HST Calibration Pipelines, while dropping some of those that are less useful or problematic The good: Existence! Data driven, using keywords in the data and reference files pointed to by keywords Flexible: users can choose at runtime what steps to run and which reference files to use Open source Distributed so users can run locally The not-so-good: ‘Stovepipe’ architecture, separate pipelines for each instrument Duplication of code Mixed languages (spp, C, Python) Dependence on IRAF

16. Calibration Software Requirements Walkthrough 16 The Fundamental Concept of the JWST Calibration Pipeline The JWST Calibration Pipeline will be set up as a sequence of independent Calibration Steps, where the output from each step becomes the input for the following step: Input (from previous step) Reference File/Table Calibration Step Output (to next step) May include control parameters

17. Calibration Software Requirements Walkthrough 17 Calibration Steps and Observing Modes Each calibration step will correct a single imperfection in the data Calibrating a given Mode will be done by chaining together a sequence of these calibration steps The order of the calibration steps will be the reverse of the order in which the imperfections occur Calibration is the process of transforming the data to how it would have looked if the detectors, instruments and telescope worked ‘perfectly’ Detector Readout  Detection  Instrument Each calibration step can be applied and tested independently of the others Although it will be necessary to make sure the appropriate data is fed into the calibration step

18. Calibration Software Requirements Walkthrough 18 Where do we start? At this time, our understanding of the calibration steps that will be necessary is limited Most of our experience is with engineering-grade detectors and non-flight readout electronics Some data taken with instrument Engineering and Test Units - E.g. MIRI VM1 and VM2 test campaigns Recently, some data has been taken with flight detectors and flight electronics The definitive test/characterization data will come from Flight Model tests, scheduled for FY11 We do have NICMOS and WFC3-IR on orbit taking IR data We can use our experience with these instruments as a starting point in determining what steps we will need

19. Calibration Software Requirements Walkthrough 19 There’s so much we DON’T know… As testing of the detectors and instruments progresses, we will learn of the existence of new imperfections, that will require new calibration steps to mitigate their effect The calibration pipeline must be flexible enough to allow the addition of new calibration steps We will also refine our understanding of the best methods of correcting these imperfections The calibration pipeline must be flexible enough to allow the swapping in and out of different algorithms to accomplish a given calibration step as the design progresses Sometimes the ‘best’ algorithm will be different for different detectors (e.g. MIRI vs. HgCdTe) The calibration pipeline must be flexible enough to support different algorithms for different detectors

20. Calibration Software Requirements Walkthrough 20 JWST Data Primer Every time a JWST detector is read out, we get a FRAME:

21. Calibration Software Requirements Walkthrough 21 JWST Data Primer… An exposure is made up of a sequence of frames:

22. Calibration Software Requirements Walkthrough 22 JWST Data Primer Each frame results from a longer integration time, so the signal-to- noise ratio improves with frame number

23. Calibration Software Requirements Walkthrough 23

24. Calibration Software Requirements Walkthrough 24 MIRI Calibration Software Requirements Requirements are classified as: Interface: Specifying the origin and form of inputs, and destination and form of outputs Functional: Specifying what the software will do Performance: Specifying execution time, memory constraints, and algorithm faithfulness Security: Specifying constraints on who may access data at each level of processing

25. Calibration Software Requirements Walkthrough 25 Back To Requirements… The task: Calibrate JWST data - Transform data to ‘how it would look on the sky if observed by a perfect JWST and perfect instrument’ - Linear geometric scale: Each pixel corresponds to a constant number of arcseconds (imaging) or constant number of microns (spectroscopic) - Linear photometric scale: Counts in each pixel corresponds to a fixed, known flux - Data Quality Information: Errors associated with count rate presented, pixels known to have calibration issues flagged

26. Calibration Software Requirements Walkthrough 26 Requirements… To calibrate the data, we will need: The uncalibrated data (provided by the Data Processing Component of the Data Management Subsystem) Calibration Reference Files, and a means for determining which reference files to use in each calibration step (provided by the Calibration Reference File System component of the Data Management Subsystem) A means for determining which exposures may be processed together to provide an improved product (provided by the Data Association component of the Data Management Subsystem) A means of storing the calibrated data products and associated metadata (provided by the Archive component of the Data Management Subsystem) A means of delivering calibrated data products to users (provided by the Archive)

27. Calibration Software Requirements Walkthrough 27 Requirements… Design Issues: We already know that we don’t know all the steps we will need to do - We will find out as we go along - We will need a flexible architecture that allows plugging new calibration steps - We know some of the steps already from our experience with HST and other IR instruments We already know that we don’t know what algorithms to use - We will find out as we go along - We will need a flexible architecture that allows swapping in and out different algorithms for each step, and provide a means for calibration scientists to test the efficacy of each step - We know what algorithms worked for HST IR instruments

28. Calibration Software Requirements Walkthrough 28 A starting point For each of the Science Modes of JWST, we can lay out what calibration steps will be needed Based on what we know now Each step will correct for a single data imperfection Each distinct calibration step will be used to derive a functional requirement for the calibration pipeline It will state what capability the calibration pipeline software will need to have in order to calibrate the data The calibration algorithms are not addressed by requirements They are covered by DESIGN If the calibration pipeline is able to perform the step, it will meet the requirements

29. Calibration Software Requirements Walkthrough 29 Requirements REQUIREMENTS map to CALIBRATION STEPS Requirements describe WHAT the system will do We expect the requirements to grow in number as new data imperfections are discovered, but we don’t expect individual requirements to change much DESIGN maps to CALIBRATION ALGORITHMS Algorithms describe HOW the system will do it We expect algorithms to change as our understanding of the instruments and detectors improves

30. Calibration Software Requirements Walkthrough 30 How Well Should the Calibration be Performed? How do we address the QUALITY of the calibration? Should the requirements state the ACCURACY to which a calibration shall be performed? I argue NO The calibration pipeline software is a ‘best effort’ endeavor STScI is a stakeholder in the success of the calibration Specifying required accuracies will discourage further exploration of calibration improvements There is a place for a process to explore where to place effort in improving the calibration, but the requirements is not it We will almost certainly discover effects that will require our expertise in calibrating A Working Group/Advisory Board will be much better placed to make these decisions

31. Calibration Software Requirements Walkthrough 31 Example Proposed Calibration Pipeline NIRCam Imaging: Data Quality & Error Initialization Coarse Ramp Jump Detection Reference Pixel Correction Detector Dark Current Correction Nonlinearity Correction Fine Ramp Jump Correction Ramp Slope Determination Flatfield Correction Persistence Correction Absolute Calibration Geometric Distortion Correction Image Combination