1. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 1 COS Training Series IV: COS Post-observation M.E. Kaiser P. E. Hodge 28 February 2008 COS Spectrograph (Colorado, BASG)

2. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 2 COS Pipeline Verification STScI : Mary Elizabeth Kaiser Phil Hodge Tony Keyes Tom Ake Cristina Oliveira Dave Sahnow Brittany Shaw Alessandra Aloisi Rosa Diaz Charles Proffitt Colorado: Jim Green Stephane Beland Steve Penton Eric Burgh Cyndi Froning Steve Osterman Eric Wilkinson

3. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 3 COS Pipeline Outline Pipeline DescriptionPipeline Description PythonPython Data Formats and File StructureData Formats and File Structure Calibration Reference Files & TablesCalibration Reference Files & Tables Reduced Data Products, Formats, etc.Reduced Data Products, Formats, etc. CumCumulative Image & Pulse Height Maps Headers and KeywordsHeaders and Keywords

4. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 4 CALCOS Foundation CALCOS data requirements specified by AV03 (COS Calibration Requirements & Procedures) -Outlines final data products for COS observations -Specifies the pre-flight calibration requirements -Specifies the reference files required -Specifies the calibration steps required to obtain the COS data product Reminder: DetectorWave Range (A) Plate Scale (  m/arcsec) Active Area (mm) Active Area (pixels) Pixel Size (  m) Resel Size (pixels) XDL (FUV)L: 1230 - 2378 M: 1132 - 1798 ~2652 @ 85 x 10 2 @ ~ 14160 x 400 6 x 246 x 10 MAMA (NUV)L: 1334 - 3560 4 osms, 3 stripes, Non-contiguous ~97025.6 x 25.61024 x 102425 x 253 x 3 CALCOS: Python program to calibrate COS FUV & NUV TIME_TAG event lists & ACCUM images, producing a wavelength & flux calibrated spectrum.

5. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 5 CALCOS Products TIME-TAG data: Raw events table: time, x, y, pulse height amplitude (FUV)Raw events table: time, x, y, pulse height amplitude (FUV) Produce corrected events table: time, corrected x, corrected y, weight, dq, phaProduce corrected events table: time, corrected x, corrected y, weight, dq, pha Produce calibrated ACCUM image from corrected event listProduce calibrated ACCUM image from corrected event list ACCUM data: Produce calibrated ACCUM image from raw ACCUM imageProduce calibrated ACCUM image from raw ACCUM image Produce flux-calibrated 1-D spectrum (wavelength calibrated)Produce flux-calibrated 1-D spectrum (wavelength calibrated) Combine FP-POS or REPEATOBS spectraCombine FP-POS or REPEATOBS spectra 2-d data: FITS images - single imset consisting of science, error, & data quality images 1-d data and TIME-TAG lists: FITS tables - extracted spectra contain fluxes, wavelengths, errors, data quality, and other related quantities (e.g. background)

6. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 6 Input Files and Products All associated data (e.g., wavecals, FP-POS components, REPEATOBS, PHA) use separate files (unlike STIS) & are linked via an association table. CALCOS files Science spectrum (suffix designates detector)Science spectrum (suffix designates detector) “_a” (or “_b” ): FUV detector segment A (or B); NUV: no “_” Pulse-height histogram if FUV ACCUM (A and B)Pulse-height histogram if FUV ACCUM (A and B) Target-acquisition imageTarget-acquisition image Support file (contains engineering information)Support file (contains engineering information) Association file controls calibration processing (science exposures, wavecals)Association file controls calibration processing (science exposures, wavecals) Primary TIME-TAG file products: “rawtag” (raw photon list), “corrtag” (corrected photon list), “flt” (2-D corrected count-rate image), “x1d” (1-D flux-calibrated extracted spectrum). Primary ACCUM file products: “rawimage” (raw image), “flt”, “x1d” 1-D spectrum for each FP-split location or repeatobs exposure; Sum of 1-D spectra

7. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 7 What is Python & Why should we use it? CALCOS is Written in Python Extremely productive (Many powerful libraries available)Extremely productive (Many powerful libraries available) Easy to learn and read & well-documented.Easy to learn and read & well-documented. Available on just about any computing platform. Portable with robust implementationAvailable on just about any computing platform. Portable with robust implementation Wide user and developer baseWide user and developer base Supports both procedural and object-oriented coding.Supports both procedural and object-oriented coding. Unlike Tcl or Perl, scales well to larger programming projectsUnlike Tcl or Perl, scales well to larger programming projects Extendable with C, C++, or Fortran.Extendable with C, C++, or Fortran. The Science Software Group successfully developed a new Command Language for IRAF using Python.The Science Software Group successfully developed a new Command Language for IRAF using Python. Programs more easily written, tested, and modified than C programs because of Python’s interpreted, dynamic and concise naturePrograms more easily written, tested, and modified than C programs because of Python’s interpreted, dynamic and concise nature Programs easier to adapt to unanticipated cal requirements than C programs.Programs easier to adapt to unanticipated cal requirements than C programs. Better at bookkeeping and set-up than CBetter at bookkeeping and set-up than C Easier to diagnose Python errors than C errors (esp. memory and pointer errors).Easier to diagnose Python errors than C errors (esp. memory and pointer errors). Python is an Open Source, freely available “scripting” language for astronomy data reduct’n & analysis

8. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 8 CALCOS Overview COS ASSOCIATIONS: will generally follow NICMOS design - associated obs are in separate datasets linked via an association table Each spectroscop sci expos will have at least 1 wavecal assoc wavecals may be shared by diff sci exps Only formal associat’ns in pipeline process & HAD: science/wavecal groupings, FP-POSs, & REPEATOBS User supplies association table nameUser supplies association table name Read and interpret association table:Read and interpret association table: get a list of the input filesget a list of the input files open one input file to get header keywords:open one input file to get header keywords: identify calibration steps to performidentify calibration steps to perform obtain list of reference file namesobtain list of reference file names verify that all required files are presentverify that all required files are present Calibrate:Calibrate: Process associated wavecalsProcess associated wavecals determine wavecal offset from ref wavecaldetermine wavecal offset from ref wavecal calibrate each input science filecalibrate each input science file Average the individual FP-split or repeatobsAverage the individual FP-split or repeatobs Reference Files ref_flat.fits flat field image ref_flat.fits flat field image ref_geo.fits geom distort’n file ref_geo.fits geom distort’n file ref_dead.fits livetime factr table ref_dead.fits livetime factr table ref_bpix.fits dqi table ref_bpix.fits dqi table ref_brf.fits baseline (stim loc) table ref_brf.fits baseline (stim loc) table ref_pha.fits pulse height table ref_pha.fits pulse height table ref_1dx.fits spectrum loc table ref_1dx.fits spectrum loc table ref_lamp.fits template spectrm tbl ref_lamp.fits template spectrm tbl ref_phot.fits sensitivity ref table ref_phot.fits sensitivity ref table ref_dsp.fits dspn relation poly tbl ref_dsp.fits dspn relation poly tbl ref_wcp.fits wave params table ref_wcp.fits wave params table

9. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 9 Outline of Calibration Steps TIME-TAG dataTIME-TAG data Copy input data into output arrayCopy input data into output array Add pseudo-random number toAdd pseudo-random number to each X and Y position (FUV only) each X and Y position (FUV only) Thermal correction (FUV only)Thermal correction (FUV only) Geometric correction (FUV only)Geometric correction (FUV only) Flat field correctionFlat field correction Livetime correctionLivetime correction Data quality assignmentData quality assignment Filter by pulse height (FUV only)Filter by pulse height (FUV only) Filter by timeFilter by time Orbital & heliocentric Doppler correctionsOrbital & heliocentric Doppler corrections Create 2-D ACCUM imagesCreate 2-D ACCUM images Cal StepDescription BRSTCORRSearch for bursts (FUV) BADTCORRFlag bad time intervals (FUV) PHACORRFilter by pulse height (FUV) RANDCORRDither to reduce aliasing (FUV) TEMPCORRTemp-dependent distortion (FUV) GEOCORRCorrect nonlinear distortion (FUV) DQICORRIncludes DQ flag in DQ extension DOPPCORRCorrect for orbital Doppler shift HELCORRCorrect for heliocentric Doppler shift FLATCORRDivide by flat field DEADCORRCorrect for deadtime WAVECORRCorrect for OSM uncertainty X1DCORRExtract 1-D spectrum BACKCORRSubtract background from spectrum FLUXCORRConvert count rate to flux units

10. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 10 Outline of Calibration Steps (cont.) ACCUM dataACCUM data Verify good pulse-height distribution (FUV only)Verify good pulse-height distribution (FUV only) Find stim pulses and compute thermal distortion coefficients (FUV only)Find stim pulses and compute thermal distortion coefficients (FUV only) Convert ACCUM to temporary coordinate list (FUV only)Convert ACCUM to temporary coordinate list (FUV only) Add pseudo-random number to each X and Y position (FUV only)Add pseudo-random number to each X and Y position (FUV only) Thermal correction (FUV only)Thermal correction (FUV only) Geometric correction (FUV only)Geometric correction (FUV only) Data quality assignmentData quality assignment Compute orbital and heliocentric Doppler shiftsCompute orbital and heliocentric Doppler shifts Create 2-D temporary count rate ACCUM imageCreate 2-D temporary count rate ACCUM image Convolve flat field with Doppler shift and divide into count rate imageConvolve flat field with Doppler shift and divide into count rate image Livetime correctionLivetime correction Create 2-D effective count rate imageCreate 2-D effective count rate image

11. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 11 Outline of Calibration Steps (cont.) Steps common to TIME-TAG and ACCUMSteps common to TIME-TAG and ACCUM Extract 1-D spectrum:Extract 1-D spectrum: Compute net count rateCompute net count rate Compute error estimateCompute error estimate Compute maximum and average data qualityCompute maximum and average data quality Compute wavelengthsCompute wavelengths Compute flux from count rateCompute flux from count rate Combine FP-split or repeatobsCombine FP-split or repeatobs

12. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 12 Running CALCOS: An Example Input Running CALCOS (2 options) Run calcos.py from the UNIX command line Run from Pyraf (Python w/ IRAF imported) pyraf import calcos stsdas calcos.calcos (“file.fits”) FUV spectrum: - source: external Pt/Ne lamp - exposure sequence: 3 FP-POS positions - tagflash exposures through WCA using internal Pt/Ne lamp Science Exposure: External Pt/Ne lamp (PSA aperture) Tagflash Wavecal Exposure: Internal Pt/Ne lamp (WCA aperture)

13. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 13 Running CALCOS

14. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 14 The Details: Thermal Correction Electronic stim pulse signals are used to correct the FUV XDL data for thermal drift (The x & y positions of detected photon events are obtained from analog electronics --> susceptible to thermal changes.) Reference Stim positions used to correct the data to the reference frame Must be done at the outset before any reference files are applied NUV MAMA has physical pixels. No stim pulse capability exists or required. 2 stim pulses for each detector Segment, located at the lower left & upper right Thermal correction done after screening for bursts (outliers in bkg_counts) mm mm

15. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 15 The Details: Geometric Correction Corrects for optical distortions at the detector image FUV geometric correction reference file has been delivered by IDT dx is multiplied by 10 in the geometric distortion map below for segments A and B to illustrate the distortion characteristics NUV geometric correction need under evaluation by IDT - effect is small

16. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 16 The Details: Flat Field Correction Top panel: flat field taken pre-TV no grid wires Middle panel: 2003 TV flat field note grid wires Lower panel: same as middle panel but with zoomed y-axis note blemish Corrects for pixel-pixel variations FUV flat field under evaluation - in-flight validation planned Grid wires: A negatively charged, primarily vertical grid that increases detector efficiency by inhibiting electrons from scattering out of the MCP pores.

17. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 17 The Details: Flat Field Correction Full flat field Corrects for pixel-pixel variations NUV flat field delivered by IDT zoomed region - note hex structure

18. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 18 The Details: Data Quality Assignment Note location & flux level depression for: grid wire shadows dead spots blemishes Identifies detector regions of less than optimal performance Detector dead-zones, hot spots, grid wire shadows, blemishes Data drop-outs

19. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 19 The Result FUV spectrum: source: external Pt/Ne lamp exposure sequence: 3 FP-POS positions (PSA aperture) tagflash exposures through WCA aperture using internal Pt/Ne lamp

20. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 20 Lifetime Charge Extraction Detector lifetime is a function of the charge extracted. A cumulative signal image of the FUV & NUV detectors will be maintained to monitor the extracted charge. FUV detector has 5 default positions to mitigate effects of charge extraction (gain sag). (non-concurrent) FUSE cumulative signal image Left: Lyman Beta region Right: Background region Light stripes are grid wire shadows

21. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 21 FUV Pulse Height Maps The Pulse Height of the electron cloud can be used to discriminate between science and spurious sources monitor the lifetime of the microchannel plate (MCP) - identify regions with large charge extractions (gain sag) Red histogram: Gain sag of the FUSE detector is shown for the Lyman Beta region - which has had large charge extraction due to both airglow and science targets. Black histrogram: Background region with minimal gain sag due to charge extraction.

22. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 22 Data File Naming Conventions Product Suffix Type Contents --------------------------------------------------------------------------------------------------------------- Uncalibrated _rawimage image Raw ACCUM counts image _rawtag table Raw TIME-TAG event list _asn table Association file _trl table Trailer file _jit table _jif image _pha table Pulse-height (FUV ACCUM only) --------------------------------------------------------------------------------------------------------------- Calibrated _corrtag table Calibrated TIME-TAG event list _flt image Flat-fielded, corrected counts _fltsum image Summed flat-fielded corrected-counts _x1d table 1-D extracted spectra _x1dsum table Combined 1-D extracted spectrum _trl table Trailer file (output) _spt image Support file (header) ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Wavecal expos produce same calibrated files as science, except no flux calibration applied to the _x1* files. Dark, flat, & image exposures produce no _x1* files. Acquisition exposures produce no calibrated products.

23. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 23 Data Processing Steps Exposure type raw tag rawunage corrtag pha counts flt fltsum x1d x1dsum asn trl spt TIME-TAG No FP FUV NUVXXXXXXXXXXXXXXXX TIME-TAG FP FUV NUVXXXXXXXXXXXXXXXX ACCUM No FP FUV NUVXXXXXXXXXXXXXXX ACCUM repeatobs No FP FUV NUVXXXXXXXXXXXXXXXXXXX ACCUM FP FUV NUVXXXXXXXXXXXXXXXXX IMAGE TIME-TAG XXXXXXX IMAGE ACCUM XXXXXX IMAGE ACCUM repeatobs XXXXXXX ACQ and PEAKUPs XXX WAVECALXXXXXXXX FLAT and DARK XXXXXX

24. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 24 COS Typical File Sizes Exposure type POD(Mb)rawtagrawimage(Mb)corrtag(Mb)flt(Mb) count s (Mb)OPUS(Mb)intoHDA(Mb) TIME-TAGFUV18  = 45  = 115 160x2160x281863 TIME-TAGNUV1840110101018858 ACCUMFUV16160x2---160x2160x2976336 ACCUMNUV210---10103212 WAVECAL < 18  < 45  < 115 160x2160x2 640- 818 < 63

25. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 25 COS Configuration Keywords Keyword: some possible valuesKeyword: some possible values –OBSTYPE: SPECTROSCOPIC, IMAGING –OBSMODE: ACCUM, TIME-TAG –EXPTYPE: ACQ, PEAKUP/XDISP, PEAKUP/DISP, SCI, WAVE, FLAT, DARK, PHA –DETECTOR: FUV, NUV –SEGMENT: FUVA, FUVB –OPT_ELEM: G130M, G160M, G185M, G225M, G285M, G140L, G230L, MIRRORA, MIRRORB –CENWAVE: 1298, 1309, 1320, etc. –APERTURE : PSA, BOA, WCA, FCA (Indicates a COS unique keyword or value)

26. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 26 Keyword specification –keyword name –default value –possible values –units –datatype –short comment for header –long description –header position –DADS table –keyword source The following information is being generated by STScI Science Instrument team and DST for all COS specific keywords.

27. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 27 Association Table Column names MEMNAME, MEMTYPE, MEMPRSNTColumn names MEMNAME, MEMTYPE, MEMPRSNT MEMNAME is the rootname for a file or set of filesMEMNAME is the rootname for a file or set of files MEMTYPE distinguishes science from wavecal, and exposure from productMEMTYPE distinguishes science from wavecal, and exposure from product MEMPRSNT = true implies that input files with the specified rootname existMEMPRSNT = true implies that input files with the specified rootname exist

28. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 28 COS Associations Only associations required for data processing will be constructed by TRANSOnly associations required for data processing will be constructed by TRANS There will be 1 product, and only 1 product, for each association.There will be 1 product, and only 1 product, for each association. COS associations will consist ofCOS associations will consist of –FP-split (always 4) or repeat obs. exposures –Wavecals Wavecals executed after each grating, central wavelength, mirror, or aperture change; and at least once each orbit.Wavecals executed after each grating, central wavelength, mirror, or aperture change; and at least once each orbit. Wavecals can be shared between associationsWavecals can be shared between associations An association table will contain all information about associated datasetAn association table will contain all information about associated dataset Observations will complete Generic Conversion as individual exposures, be held by a Data Collector until all member exposures are present, and the product will be created by CALCOS.Observations will complete Generic Conversion as individual exposures, be held by a Data Collector until all member exposures are present, and the product will be created by CALCOS.

29. 28 February 2008 COS Training Series IV (kaiser@pha.jhu.edu) 29 NUV Science Data Format CBA CBA PtNeWavecalExternalScienceNUVMAMA