1. PyRAF/Pipeline Removal of SAA Persistence from NICMOS Data Elizabeth A. Barker Vicki Laidler Eddie Bergeron Anton Koekemoer Elizabeth A. Barker Vicki Laidler Eddie Bergeron Anton Koekemoer TIPS Meeting 18 October 2007

2. South Atlantic Anomaly  Part of inner Van Allen radiation belt  Higher density of charged particles  8-9 times per day  About half of all orbits  Part of inner Van Allen radiation belt  Higher density of charged particles  8-9 times per day  About half of all orbits

3. Cosmic Ray Persistence  Electrons trapped during SAA passage  Exponential decay for release of electrons  Lower SNR in short exposures means persistence has smaller contribution to total noise  Persistence becomes a more significant noise source in long exposures  Fixed noise pattern per SAA passage (different from pure random noise)  Electrons trapped during SAA passage  Exponential decay for release of electrons  Lower SNR in short exposures means persistence has smaller contribution to total noise  Persistence becomes a more significant noise source in long exposures  Fixed noise pattern per SAA passage (different from pure random noise)

4. Persistence Removal  “Post-SAA Darks” [Bergeron & Najita, 1998]  Different from standard calibration dark images  2 images taken back-to-back  First exposure starts 174 sec after SAA exit  Each 256 sec exposure  BLANK filter  Implemented as automatically scheduled exposures since Cycle 11  “Post-SAA Darks” [Bergeron & Najita, 1998]  Different from standard calibration dark images  2 images taken back-to-back  First exposure starts 174 sec after SAA exit  Each 256 sec exposure  BLANK filter  Implemented as automatically scheduled exposures since Cycle 11

5. Post-SAA Darks Persistence ModelPost-SAA Dark

6. SAA-impacted Images Calibrated ImageSAAcleaned Image NIC1, F090M, 448 sec, 918 sec since SAA exit

7. SAA-impacted Images Calibrated, Pedsub ImageSAAcleaned, Pedsub Image NIC2, F215N, 1400 sec, 917 sec since SAA exit

8. Persistence Removal Algorithm  Recommend pedestal correcting calibrated image  Use previously determined scale factor of decay between post-SAA darks (0.54-0.56)  [Bergeron & Dickinson, NICMOS ISR 2003-010]  Create Image of Persistence  Weighted combination of post-SAA darks  CR rejection  Remove pedestal  Iteratively scale and subtract persistence image from science image  Fitting to the minimum total noise in image  Repeat pedestal correction on final output image  Recommend pedestal correcting calibrated image  Use previously determined scale factor of decay between post-SAA darks (0.54-0.56)  [Bergeron & Dickinson, NICMOS ISR 2003-010]  Create Image of Persistence  Weighted combination of post-SAA darks  CR rejection  Remove pedestal  Iteratively scale and subtract persistence image from science image  Fitting to the minimum total noise in image  Repeat pedestal correction on final output image

9. High and Low Signal Populations  Long SAA passage duration  Widely varying CR energies  Earliest CR hits have decayed most  Latest CR hits have higher persistence signal  Long SAA passage duration  Widely varying CR energies  Earliest CR hits have decayed most  Latest CR hits have higher persistence signal

10. Iteratively Fitting Persistence Model [Bergeron & Dickenson ISR 2003-010]  Persistence level depends on:  Length/depth of SAA passage  Time since SAA exit  Science exposure time  Iteratively subtract scaled persistence model  Step through different multiplicative factors  Measure RMS width of pixel histogram for each factor  Minimum RMS width indicates best factor  Subtract persistence model from science exposure, after scaling by best factor  Persistence level depends on:  Length/depth of SAA passage  Time since SAA exit  Science exposure time  Iteratively subtract scaled persistence model  Step through different multiplicative factors  Measure RMS width of pixel histogram for each factor  Minimum RMS width indicates best factor  Subtract persistence model from science exposure, after scaling by best factor

11. Best Persistence Fit Figure 12: Bergeron & Dickinson ISR 2003-010 ~ (129-46)/129

12. Testing PyRAF SAAclean  Conversion from IDL [Bergeron] to Python by Vicki Laidler  Data Included:  All cameras  SAA-impacted  Flat-fielded  Pedestal-corrected (pedsub)  Data Excluded:  Grism data (not flat-fielded)  Polarizers  Conversion from IDL [Bergeron] to Python by Vicki Laidler  Data Included:  All cameras  SAA-impacted  Flat-fielded  Pedestal-corrected (pedsub)  Data Excluded:  Grism data (not flat-fielded)  Polarizers

13. Testing SAAclean - Results Maximum Error Correction Median Camera # of Images High Signal Domain Low Signal Domain High Signal Domain Low Signal Domain 141549.08%29.43%1.818%0.132% 2159559.93%53.94%1.319%0.122% 3213331.63%27.41%3.171%0.174%

14. PyRAF SAAclean - Input calcimage = Input (usually ped) file to calculate correction targimage = Input (ped or cal) file to correct output = Output cleaned datafile (targimage with correction applied) clobber = no Overwrite output files if they already exist? (readsaaper = no) Read SAA persistence image from file (If no, construct it.) saaperfile = saaper.fits Filename for SAA persistence image (writesaaper = yes) Write SAA persistence image? (flatsaaper = yes) Flat-field SAA persistence image before analysis (darkpath = saaref$) Path to dark reference files (scale = 0.54) Scale factor for constructing persistence image (wf1 = 0.7) Weight for first SAA exposure (wf2 = 0.3) Weight for second SAA exposure (crthresh = 0.3) Threshold for CR rejection (noisethresh = 1.0) Noise reduction threshold (percent) (binsigfrac = 0.3) Stddev fraction for excluding narrow bins (stepsize = 0.008) Increment multiplier for SAA scale factor fitting (fitthresh = yes) Solve for threshold value? (If no, uses value of thresh) (thresh = ) Threshold to separate high/low signal domain in SAA persistence image (histbinwidth = 0.001)Bin width for histogram in threshold fitting (nclip = 3) Number of clipping iterations for threshold fitting (hirange = 0.4) Maximum multiplier for high signal domain (lorange = 0.25) Maximum multiplier for low signal domain (fitmult = yes) Fit to determine multiplier for minimum noise? (applied = ) Cleaning applied to which domains? (hi_nr = ) Noise reduction in high signal domain (lo_nr = ) Noise reduction in low signal domain alldiags = no Write out all possible diagnostic files? (diagroot = diag) Root filename for diagnostic files

15. PyRAF SAAclean - Output saaclean version 0.99dev Input files: n8xw12n8q_subisr.fits n8xw12n8q_subisr.fits sci image : using DQ extension for badpix postsaa dark #1 : using DQ extension for badpix failing over to /data/cdbs5/nref//m9c1047pn_msk.fits postsaa dark #2 : using DQ extension for badpix failing over to /data/cdbs5/nref//m9c1047pn_msk.fits Using scale factor of 0.54 to construct persistence image flatfile : using DQ extension for badpix median used in flatfielding: 0.0379231178101 Coefficients for gauss-poly fit to persistence model histogram: Gaussian (low signal component) terms: Amplitude, Mean, Sigma: 1026.073392 17.503406 9.216971 Polynomial terms: Constant, Linear, Quadratic:47.687849 0.422538 -0.003921 Threshold for hi/lo: 0.0995250821636 Npixels hi/lo: 9596 55940 Results summary for high domain: chi2 for parabola fit = 0.000420518773132 min-noise (best) scale factor is: 0.311645043762 effective noise at this factor (electrons at gain 5.400000): 49.903919 noise reduction (percent) : 38.5484443442 Results summary for low domain: chi2 for parabola fit = 4.05079716407e-05 min-noise (best) scale factor is: 0.189995821621 effective noise at this factor (electrons at gain 5.400000): 36.192606 noise reduction (percent) : 7.21719319318 Applying noise reduction in both domains

16. PyRAF SAAclean - Products Files Output: SAA cleaned imageSAA cleaned image Persistence image (if requested)Persistence image (if requested) KeywordDescription SAAPERS SAA persistence image SCNPSCL Scale factor used to construct persistence image SCNPMDN Median used in flatfielding persistence image SCNTHRSHThreshold dividing high and low signal domains SCNHNPIX Number of pixels in high signal domain (HSD) SCNLNPIX Number of pixels in low signal domain (LSD) SCNHCHI2 HSD chi squared for parabola fit SCNHSCL HSD scale factor for min noise SCNHEFFN HSD effective noise at SCNGAIN SCNHNREDHSD noise reduction (percent) SCNLCHI2 LSD chi squared for parabola fit SCNLSCL LSD scale factor for min noise SCNLEFFN LSD effective noise at SCNGAIN SCNLNRED LSD noise reduction (percent) SCNAPPLD To which domain(s) was SAA cleaning applied SAACNTAB Reference table (with task parameters) SAACPDGR Pedigree of reference table SAADFILE SAA dark reference image file SAADPDGRPedigree of reference image SAACORR Correct for SAA signature SAADONE Status of SAA signature correction

17. PyRAF SAAclean in Pipeline  To be implemented in OPUS 2008.1  After calnica  *_cal.fits  Pedsub *cal.fits file  SAAclean  SAAclean correction applied to *_cal.fits file (without pedestal correction)  Outputs modified *_cal.fits file  To be implemented in OPUS 2008.1  After calnica  *_cal.fits  Pedsub *cal.fits file  SAAclean  SAAclean correction applied to *_cal.fits file (without pedestal correction)  Outputs modified *_cal.fits file