1 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 Name of Meeting Location Date - Change in Slide Master LSST Reprocessed Co-adds of SDSS Stripe 82 Yusra AlSayyad University of Washington MULTI-WAVELENGTH, MULTI- EPOCH HERITAGE OF STRIPE82 MARCH 17, 2014 Andrew Becker, Andrew Connolly, Željko Ivezić, Simon Krughoff, Russell Owen, (U. Washington) Jim Bosch, Robert Lupton, Paul Price, (Princeton), Mario Jurić and the LSST Data Management Team

2 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 LSST DM pipeline tested on Stripe 82 − LSST data management processes data 2 ways: 1.Support science cases requiring rapid identification and follow-up (transients, fast-moving NEOs) 2.Support deep static-sky science and time domain science that is not time sensitive. − Testing prototype code on simulated images and real data 3 runs of same region of sky

3 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Outline − How do we aggregate information from multi-epoch surveys, with the goal of extracting highest S/N possible. Detect on co-add Measure on single-epoch exposures (forced photometry) − Summer 2013 co-add construction details Co-addition process Datasets produced Preliminary quality assessment

4 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Three ways combine info from multi-epoch surveys 1. Catalog co-add: Detect and measure on single-epoch exposures and take a weighted mean or median of the measurements.

5 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 Light-curves derived from detecting on single- epoch images misses information Time Flux 5σ detection limit Measured Mean Flux

6 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Three ways combine info from multi-epoch surveys 1. Catalog Co-add: Detect and measure on single-epoch exposures and take a weighted mean or median of the measurements. 2. Image Co-add: Detect and measure on co- added image Lose measurements under detection threshold

7 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Measurement on Co-adds misses information 1.Poor-seeing and bright-sky images do not increase your co-add SNR. 2.Co-add PSFs are hard to characterize. 3.Impossible to distinguish between galaxies and high proper motion stars. PS F 3 PSF2PSF2 Psf 1 Image 2Image 1Image 3 COADD Lang+ 09 Individual exposures: objects are undetected or marginally detected Moving point-source and galaxy models are indistinguishable on the coadd Solved by PSF-Matching: (Kaiser 04) (Huff+11)

8 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Three ways combine info from multi-epoch surveys 1. Catalog Co-add: Detect and measure on single-epoch images and take a weighted mean or median of the measurements. 2. Image Co-add: Detect and measure on co- added image 3. Detect on co-added image. Measure on single-epoch exposures. Misses measurements under detection threshold Requires PSF- matching. Can’t characterize objects.

9 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Summer 2013 Stripe 82 Co-addition

10 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Summer 2013 Stripe 82 Co-add Process u coadd g coadd r coadd i coadd z coadd z coadd Catalog catalog 40M rows catalog Detect and Measure Detect and Measure with: StackFit/CoaddPSF Deblender 3σ detection threshold Measure flux at positions in i-band co-added catalog Details next Forced Photometry Catalog 15 B rows ~80 epochs per band Force photometry at positions

11 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 Steps to make a deep co-add − Define the co-add projection and geometry For each run: 1.Resample to defined output geometry 2.Scale to co-add zeropoint

12 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, Scale to a co-add zeropoint − Zeropoint varies spatially and temporally − Photometric self- calibration calculated before co-adds are generated − Used 1-D model of zeropoint (per camcol) as multiplicative scale factor. Figure: Realization of a 0.5 mag cloud based on a structure function of observed clouds and some realistic assumptions about cloud-size and velocity. (image credit: Lynne Jones) RA & time Dec

13 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 Steps to make a deep co-add − Define the co-add projection and geometry For each run: 1.Resample to defined output geometry 2.Scale to co-add zeropoint 3.Match sky-levels/backgrounds

14 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Background-subtracted coaddSingle-epoch visit 5 arcmin

15 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Background-subtracted coaddBackground-matched coadd 5 arcmin

16 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, Match Backgrounds − Goal: Estimate the difference in sky level between successive runs Leave in common-mode (astrophysical) background -Wings of galaxies, diffuse nebulosity Remove time-dependent backgrounds -Atmospheric: changes in sky, airglow, scattered moonlight − One realization of the sky survives, but we can subtract this at higher S/N − (see Huff+ 11)

17 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, Match Backgrounds − Start with a reference image. Locks the sky-level of a co-add patch to that of its neighbors Ensures the co-add will be seamless at patch boundaries Use a run with low background variation as a reference image Reference Image for example patch

18 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, Match Backgrounds For each input Image: 1)Take the difference Image 2)Mask out detections and bad pixels in difference image Reference Input Difference Masked Difference

19 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, : Background Matching 3)Fit a 2D spatial model to the masked difference image to generate an offset image. 4)Add the offset image to the input image, matching the background level to that of the reference image. 3)Check quality of match: (RMS of residuals, MSE/Variance. Leave out if matching failed) Masked Difference

20 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 Steps to make a deep co-add − Define the coadd projection and geometry For each run: 1.Resample to defined output geometry 2.Scale to co-add zeropoint 3.Match sky-levels/backgrounds For ensemble of resampled & scaled visits: 1.Stack co-add -Weight whole image by inverse variance and stack with 5σ-clipped mean for eliminating: moving objects, transients, satellite trails Small-scale instrumental: glints

Figure: 5 sq. deg. background-matched coadd composite (g,r,i) ~55 epochs Region: Aqr Galactic lat = -35.0

Figure: 5 sq. deg. background-matched coadd composite (g,r,i) ~55 epochs Region: Aqr Galactic lat = -35.0

23 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Summer 2013 Stripe 82 Co-add Process u coadd g coadd r coadd i coadd z coadd z coadd Catalog catalog 40M rows catalog Detect and Measure Detect and Measure with: StackFit/CoaddPSF Deblender 3σ detection threshold Measure flux at positions in i-band co-added catalog Details next Forced Photometry Catalog 15 B rows ~80 epochs per band Force photometry at positions

24 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Preliminary Quality Assessment

25 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Median zeropoint is stable W X Zeropoints calibrated using the Ivezic+ 07 star catalog. (See talk on Wednesday) Ivezic+04 Point sources

26 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 RMS approaches intrinsic width of w ~ 0.01 r mag Median PSF flux in epochs

27 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Forced phot. adds 0.5 mag to SNR=5 depth

28 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013

29 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Dataset is publicly available. Access? − What’s available? 5 sq. degree region is indexed and ready to use today. 235 sq. degree region from previous data challenge available -g,r,i, no deblending, forced photometry seeded on r-band − How to access? LSST Science Collaboration members can request a mysql account and explore the databases hosted at NCSA Send me a list of positions (< 10M rows) and I’ll send you an ascii or fits file with forced photometry and aggregates − Code is open source. − Caveat (prototype code): No forced galaxy photometry. Deblender not dependable in blends containing stars < 17mag.

30 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Summary − Stripe 82 has proved to be a great dataset for testing LSST software. − Result of Summer 2013 production run is deeper than best publicly-available optical Stripe82 catalog. − Forced Photometry: Complete light curves for even marginally detected objects Disentangles extended and moving sources Includes all epochs (not just the best seeing, lowest background)

31 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014

32 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Appendix

33 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Weighted Mean

34 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 i-band Completeness

35 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Completeness HSTSDSS

36 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Star galaxy separation

37 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 False galaxies are unresolved blends

38 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 False stars are compact

39 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 Winter 2013 comparison to previous coadd We go deeper than Annis et al. (2012) by 0.75 magnitudes at same equivalent S/N. SNR

40 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 Quasar Luminosity Function at z ~ 4 40

41 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014

42 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 Level 2 Data Processing Data Products Definitions Document Section 5.2

43 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, Background Matching − For N input images, find least sq. solution of all N(N-1)/2 difference images (e.g. Huff et al. 2011) (Presented as “NN2”- algorithm by Barris et al. 2009) Offers better relative offsets in sky-levels, but does not provide an absolute sky-level NN2 does not eliminate need for a reference image. -If you choose the mean you get a patchwork quilt. -If you choose zero, you lose all astrophysical background on scales greater than a patch We tested both NN2 and simple ref-image. Preliminary conclusion is that the gains are not significant. We will re-examine on the ecliptic where the reference image may be heavily masked due to asteroids

44 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 Level 2 Processing DRP begins with processing, detection, and measurement on single frames, generating Sources. These are used to photometrically and astrometrically calibrate the survey. A series of coadds is built next, where Objects are detected. Detections on co-adds are deblended and associated to form a master object list. The objects are simultaneously characterized in all observed epochs (MultiFit). Time variability is characterized by independent measurement of Forced Sources in individual epochs.

45 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 Types of coadds planned in LSST Coadd Type Subset of imagesDriverBandsPre- served DeepAll suitable images Detect objects on static sky. Maintaining low surface brightness. ugrizy + M ✔ Short Period Binned by year Detect long-timescale or moving objects that would be washed out in full-depth coadds ugrizy + M ✗ Best- Seeing Only best seeing images (e.g. top quartile) Assist deblending ugrizy ✗ PSF- matched All suitable images Measure well-defined colors of galaxies ugrizy ✗ Binned by seeing and airmass Image Differencing Templates ugrizy ✔

46 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Use stellar locus to check calibration W X Ivezic et al W

47 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 Median zeropoint is stable W X

48 MULTI-WAVELENGTH, MULTI-EPOCH HERITAGE OF STRIPE 82 | PRINCETON, NJ | MARCH 17, 2014 Median forced photometry is better than coadd photometry