1 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Name of Meeting Location Date - Change in Slide Master Coaddition: Creation of Coadded Images Yusra AlSayyad University of Washington September 19, 2013 CDP FINAL DESIGN REVIEW September , 2013

2 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Charge Questions − (10) Are the plans for software to generate coadded images in place, properly understood, and consistent with the data product requirements?

3 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Charge Questions − (10) Are the plans for software to generate coadded images in place, properly understood, and consistent with the data product requirements?

4 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Level 2 Data Processing Data Products Definitions Document Section 5.2

5 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Types of coadds planned (DPDD 5.4.3) Coadd Type Subset of imagesDriverBandsPre- served DeepAll suitable images Detect objects on static sky. Maintaining low surface brightness. ugrizy + M ✔

6 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Types of coadds planned (DPDD 5.4.3) Coadd Type Subset of imagesDriverBandsPre- served DeepAll suitable images Detect objects on static sky. Maintaining low surface brightness. ugrizy + M ✔ Short Period Binned by year Detect long-timescale or moving objects that would be washed out in full-depth coadds ugrizy + M ✗

7 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Types of coadds planned (DPDD 5.4.3) Coadd Type Subset of imagesDriverBandsPre- served DeepAll suitable images Detect objects on static sky. Maintaining low surface brightness. ugrizy + M ✔ Short Period Binned by year Detect long-timescale or moving objects that would be washed out in full-depth coadds ugrizy + M ✗ Best- Seeing Only best seeing images (e.g. top quartile) Assist deblending ugrizy ✗ The inputs to these three types are the same. Because the expensive step of coaddition is resampling/warping the images, 2 of the 3 types of coadds come very cheaply because we are just stacking different subsets of these inputs. See RHL’s talk

8 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Types of coadds planned (DPDD 5.4.3) Coadd Type Subset of imagesDriverBandsPre- served DeepAll suitable images Detect objects on static sky. Maintaining low surface brightness. ugrizy + M ✔ Short Period Binned by year Detect long-timescale or moving objects that would be washed out in full-depth coadds ugrizy + M ✗ Best- Seeing Only best seeing images (e.g. top quartile) Assist deblending ugrizy ✗ PSF- matched All suitable images Measure well-defined colors of galaxies ugrizy ✗ Binned by seeing and airmass Image Differencing Templates ugrizy ✔ How should we generate these? See Andy Becker’s talk

9 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Software prototyped and planned − At this point in development, we have an end to end working prototype that builds coadds − We have coadded stripe 82 data with the software stack and released the resulting stacked images and catalogs as part of our semi-annual data challenges. − Through this iteration, we have refined our plan.

10 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Outline: Steps to make a deep coadd − Define the coadd projection and geometry For each visit: 1.Resample to defined output geometry 2.Scale to coadd zeropoint 3.Match sky-levels/backgrounds For ensemble of resampled & scaled visits: 1.Stack coadd

11 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Defining the Coadd Geometry − Projection Trade-offs − Pixel-scale trade-offs Image quality vs. Cost Can optimize for depth, quality of difference imaging templates, etc − Image geometry (tessellation of the sky) Equal-area (MOL) (conserves flux) Stereographic (STG) (conformal, no local shape distortions) vs. Image credit: Worlfram Equal-area (MOL) (conserves surface brightness) For more details see:

12 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Defining the Coadd Geometry − For LSST, want large coadds Single projection and continuous sky- background − Tradeoffs Too big -> distortion at edges Too small -> cost of overlap increases (an overlap of one focal plane (3.5 deg) − Baseline uses: A dodecahedron. Stereographic projection 0.14 arcsec/pixel − Modular implementation makes it easy to swap projections and geometries at runtime Tract Patch

13 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, Resampling − Find all images that overlap a patch Coadds will be generated patch by patch − Stitch all images per visit − Warp/resample all visits to tract geometry − Most expensive step − Contingency: look into GPUs to reduce expense 3 visits that overlap patch, warped to coadd projection

14 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, Scale all visits to coadd zeropoint − Zeropoint varies spatially over 3.5 degree field of view − Photometric self- calibration will be available before coadds are generated − Use this 2D model of zeropoint over the focal plane as multiplicative scale factor. Figure: Realization of a 0.5 mag cloud based on a structure function of observed clouds and some realistic assumptions about cloud-size and velocity. (image credit: Lynne Jones)

15 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Background-subtracted coaddSingle-epoch visit 5 arcmin

16 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Background-subtracted coaddBackground-matched coadd 5 arcmin

17 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, Background Matching − Goal: Estimate the difference in sky level between successive exposures Leave in common-mode (astrophysical) background -Wings of galaxies, diffuse nebulosity Remove time-dependent backgrounds -Atmospheric: changes in sky, airglow, scattered moonlight − One realization of the sky survives, but we can subtract this at higher S/N

18 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, Background Matching − Start with a reference image the size of a tract. Locks the sky-level of a coadd patch to that of it’s neighbors Ensures the coadd will be seamless at patch boundaries On FOV scale: a reference image can be one single-epoch visit. Reference Image for example patch

19 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, Background Matching For each input Image: 1)Take the difference Image 2)Mask out detections and bad pixels in difference image Reference Input Difference Masked Difference

20 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, : Background Matching 3)Fit a 2D spatial model to the masked difference image to generate an offset image. 4)Add the offset image to the input image, matching the background level to that of the reference image. 5)Check quality of match: (RMS of residuals, MSE/Variance. Leave out if matching failed Masked Difference

21 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Background-subtracted coaddBackground-matched coadd 5 arcmin

22 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, Stacking − The prototype weights whole image by inverse variance and stacks with sigma-clipped mean for eliminating: moving objects, transients Small-scale instrumental: glints − Prototyped multi-color as χ 2 coadd (Szalay, Connolly, Gyula 1999) − We plan to examine: What constitutes “significantly degraded seeing” Seeing percentile cut-off for “best-seeing coadds” What image weighting maximizes depth Final coadd with background of reference

23 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 − Roadmap for full functionality in 5- years − Improve working prototypes to level we want for LSST

24 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Addressing the Charge − Are the plans for software to generate coadded images in place, properly understood, and consistent with the data product requirements? Data product requirements include: image differencing templates, catalogs of deep detections, detections of slowing moving/variable sources, deblended sources, and galaxy colors. We have identified four types of coadds that meet these requirements. Plans for creating these coadds are in place. Prototype works start to finish. Tested on Stripe82 data. Extremely modular code: can swap in and out different components. We’ve identified: -Parameters to test using simulated images (e.g. seeing cut-offs) -New modules needed in order to coadd LSST data (e.g. Building large scale reference images with discrete visits.) Five-year roadmap

25 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Final Result Large Scale Coadds Figure: 2x2 deg. background- matched coadd of Stripe 82. See Winter 2013 data challenge:

26 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Appendix

27 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, Background Matching − For N input images, find least sq. solution of all N(N-1)/2 difference images (e.g. Huff et al. 2011) (Presented as “NN2”- algorithm by Barris et al. 2009) Offers better relative offsets in sky-levels, but does not provide an absolute sky-level NN2 does not eliminate need for a reference image. -If you choose the mean you get a patchwork quilt. -If you choose zero, you lose all astrophysical background on scales greater than a patch We tested both NN2 and simple ref-image. Preliminary conclusion is that the gains are not significant. We will re-examine on the ecliptic where the reference image may be heavily masked due to asteroids

28 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Do we want to maximize depth? − Open Issues (DPDD Section 5.6): What is the primary use case for retained coadds? Which coadds do we keep and serve to the public? Current plan is to make available the deep coadds and the CoaddPsf. Final coadd with background of reference