1. Overview of Science Verification Plan Keith Bechtol and Zeljko Ivezic LSST Commissioning Plan Review January 24-26, 2017

2. Day 2 breakout on Science Verification: 1) Science Verification Plan (Ivezic) Scope of Science Verification Guiding principles and basic plan On-sky Data/mini-surveys for Science Verification First on-sky data: 3 months w/ ComCam, 2 months with LSSTCam S#1: ~1600 deg2 x 15 visits x 6 filters x 2 phases (~30,000 visits, ~40 nights) S#2: ~300 deg2 x 825 visits in 6 filters (~25,000 visits, ~30 nights) Operational Readiness Review 2) Data Quality Analysis and Revalidating the Pipelines (Lupton) ”We commissioned the pipelines; we’ll repeat the process using real LSST data.” 3) Science Software Evolution During Commissioning (Lim) ”How shall we deal with things that go wrong in DM?”

3. The Goals of Science Verification
Determining whether the specifications defined in the SRD and LSR can be met with the full survey
Characterizing other system performance metrics in the context of the four primary science drivers
Studying environmental dependencies and technical optimization that inform early operations
Documenting system performance and verifying mechanisms to monitor system performance during operations
Validating data delivery, derived data products, and data access tools that will be used by the science community

4. Terminology
Early Science Verification: data quality assessments made during initial testing of ComCam and LSSTCam to provide early feedback on system performance (prior to the designated Science Verification phase) (3 months of on-sky data with ComCam and 2 months with LSSTCam)
The Science Verification phase of the Commissioning Plan refers specifically to the 5-month period that includes observations and analysis for 2 operational readiness mini-surveys (3 months of mini surveys with LSSTCam)
Final Science Verification with Mini-Surveys: is part of the 5-month Science Verification phase of the Commissioning Plan

5. Science Verification: The Final Commissioning Phase

6. Science Verification: The Final Commissioning Phase
3 Periods of Sustained On-sky Observations with ComCam and LSSTCam

7. Science Verification: The Final Commissioning Phase
Science Verification Phase

8. Terminology
Early Science Verification: data quality assessments made during initial testing of ComCam and LSSTCam to provide early feedback on system performance (prior to the designated Science Verification phase) (3 months of on-sky data with ComCam and 2 months with LSSTCam)
The Science Verification phase of the Commissioning Plan refers specifically to the 5-month period that includes observations and analysis for 2 operational readiness mini-surveys (3 months of mini surveys with LSSTCam)
Final Science Verification with Mini-Surveys: is part of the 5-month Science Verification phase of the Commissioning Plan

9. The Goals of Science Verification
Determining whether the specifications defined in the SRD and LSR can be met with the full survey
Characterizing other system performance metrics in the context of the four primary science drivers
Studying environmental dependencies and technical optimization that inform early operations
Documenting system performance and verifying mechanisms to monitor system performance during operations
Validating data delivery, derived data products, and data access tools that will be used by the science community
Aim to quantify the range of demonstrated performance by using a combination of on-sky data, informed simulations, and external datasets

10. Starting assumptions and assertions
Camera and telescope are commissioned and ready to take data
DM pipelines and infrastructure will have been commissioned using extant data (e.g. HSC, ZTF) and simulations (including data product distribution services and QC/QA tools)
Meeting the SRD specifications (Key Performance Metrics) is a necessary but not sufficient condition to undertake the full survey
The key to successful science verification is the analysis of on-sky LSST data and data products produced with DM system
The most critical element of science verification is the team, which must have adequate expertise, be agile and fully dedicated to the commissioning and science verification efforts

11. Flowdown of Science Goals to System Requirements

12. Flowdown of System Requirements
The SRD lists a minimal set of the most challenging requirements for LSST data products, motivated by the main science themes, that are believed to fully exercise the technical capabilities of the system
The SRD requirements flow down to more extended and detailed requirements for the system and individual subsystems

13. DM Key Performance Metrics
The highest level requirements for the DM system come from the SRD and Observatory System Specifications (OSS, LSE-30) and have been flowed down to a set of DM Key Performance Metrics, representing tests of the DM pipelines for calibration, astrometry, PSF determination, as well as completeness and purity for moving, variable, and transient sources.
The KPMs will be routinely monitored and archived throughout commissioning and operations.

14. Verifying Requirements and More Comprehensive Characterization
Example: Single-visit imaging depth
The distribution of the 5σ (SNR=5) detection depth for point sources for all the exposures in the r band will have a median not brighter than D1 mag, and no more than DF1 % of images will have a 5σ depth brighter than Z1 mag.
Remarks:
Although this is a single-visit performance specification, the requirement is stated in terms of a median and outlier fraction from an ensemble of visits.
The requirement is stated for photometric dark nights and pointings close to zenith. How does depth vary with observing conditions? What distribution of single-visit depth can be expected for the full survey?
In many cases, analysis of on-sky commissioning data will need to be combined with simulations and/or external datasets to understand whether the system requirements can be met with the full survey

15. System Characterization Beyond the SRD
Between high-level science analyses and the basic data quality metrics specified in the SRD (image quality, depth, photometry, astrometry, etc.), there exists a set of intermediate data characteristics that represent important benchmarks of scientific capabilities:
Object detection completeness
Star-galaxy separation
DIASource classification, spurious rates
Photometric redshifts
Low surface brightness features
Weak-lensing null tests
Crowded fields / deblending
....
Pursuing a selection of such analyses during the Science Verification period (1) may reveal more subtle issues that require hardware/software adjustments and/or inform operations, and
(2) would provide valuable documentation to the scientific community

16. System Characterization Beyond the SRD
Example: Level-2 Data Products
The Data Products Definition Document (DPDD, LSE-163) specifies that the Level-2 data products include:
Adaptive second moments of source intensity for each source and for the PSF at each source location
Bulge-disk model (~200 samples from likelihood function)
Photo-z (~100 parameters describing likelihood distribution)
Morphological extendedness parameter
Statistical variability metrics
Optimization of the algorithms that generate these quantities is beyond the scope of the Commissioning Team. However, baseline characterization of these quantities is a goal of Science Verification.

17. Planning Methodology

18. Methodology for Planning On-sky Observations
Verify with on-sky data as early as possible
Gradual transition from engineering activities to sustained operations
Engineering focus during AI&T with ComCam and LSSTCam
Approach early operations level during mini-surveys
Allow 8 weeks of pre-ORR engineering after mini-surveys
Tests of increasing sophistication: single-visit performance → image stack performance → beyond SRD characterization
Direct test if possible; validate with simulations otherwise

19. Sustained Observing Periods during Commissioning

20. Early Science Verification with ComCam
Science images with ComCam provide a first opportunity to test DM software with LSST images
Gradual transition from engineering focus to periods of sustained observing

21. Early Science Verification with LSSTCam
Repeat sequence of early science verification observations and analysis from ComCam with LSSTCam, making use of experience and analysis tools gained with ComCam
Focus on range of delivered performance over larger FOV

22. Final Science Verification with Mini-Surveys
5-month period demonstrating operational readiness
Two 6-week continuous scheduler-driven mini-surveys exercising the Level-1 and Level-2 data production systems, respectively
Comprehensive characterization of bulk data acquired under nominal observing conditions
Identifying corner cases with the aid of a larger statistical sample of observations

23. Mini-Survey 1: Wide-Area Alert Production
Objectives:
Template building with DRP pipeline
Level-1 processing, real-time alert generation
Testing survey progress over wide area, validating observation simulations
Observations:
~1600 deg2 x 15 visits x 6 filters x 2 phases (~30,000 visits, ~40 nights)
In each phase, cover ~10% of footprint with exposure equivalent to 1 year of Wide-Fast-Deep survey
Long equatorial stripe covering range of source densities
Phase 1: Observations for template generation (3 weeks)
Phase 2: Re-observing area for alert production (3 weeks)
Phases separated by 6 weeks to allow for astrophysical evolution and template processing (mini-survey 2 scheduled in-between phases)
1 visit = 9.6 deg2, 825 visits per field in 10 years (~1 night per field)
~2,000 fields in the main deep-wide fast survey, 2.5 million total

24. Mini-Survey 2: 10-year Depth Survey
Objectives:
Focus on Level-2 data products at full survey depth
Data quality characterization beyond the SRD through analyses that are closely tied to main science themes
e.g., source detection completeness, star-galaxy separation, photo-z, weak-lensing null tests, cluster weak lensing
Template generation and real-time alert production; these high-cadence observations will form an exceptional dataset for detailed characterization of transient, variable, and moving object alerts
Observations:
~30 fields x 825 visits in 6 filters (~25,000 visits, ~30 nights)
~300 deg2 to full depth of Wide-Fast-Deep survey (~1% of footprint)
Select fields to overlap with external reference datasets
Scheduler used to optimize data quality across fields

25. QA Environment and QC Pipelines
To the extent possible, Science Verification analyses by the Commissioning team will make use of Quality Assessment (QA) and Quality Control (QC) tools
Quality Assessment pipelines are identical to other DM pipelines; built from set of python tasks which can be reused in other contexts
The task of ensuring that KPMs are calculated and tracking their distributions as the pipelines evolve and encounter new data is Quality Control

26. Data Delivery Services for Users
As part of the commissioning effort, DM will serve data to the Commissioning Team using the same data delivery services that will eventually be used by the LSST data rights community. The access and analysis of Science Verification data by the Commissioning Team will therefore serve a dual purpose to further test those services for representative use cases.

27. Data Release Policy for Commissioning Data
Data distribution beyond the Commissioning Team is outside the scope of the Commissioning Plan
The Commissioning Team will consider assembling a small team of experts from the community to facilitate additional specific data quality assessments during commissioning. These community experts will be held accountable for deliverables.

28. Preparations for Operational Readiness Review
Following the two mini-surveys, the observatory will enter a 8-week shutdown period to prepare for the Operational Readiness Review (ORR)
Primary activities:
Data processing and analysis for mini-surveys, compiling results of requirements testing
Preparing documentation for the as-built hardware and software, system operation, and delivered system performance
Pre-ORR engineering
M1M3 mirror recoating
Camera maintenance and servicing

29. Operations Readiness Review
An Operations Readiness Review (ORR) defines the end of the Science Verification phase, when the project will present results of the commissioning effort to an external panel. The ORR will signify the end of construction and the conclusion of the NSF MREFC fund project and DOE early operations (DOE-EO).
The expert panel will be appointed by the NSF and DOE, in consultation with the LSST project team. The expert panel will make a recommendation to the Operations Facility, which will then make a recommendation to the agencies on whether to declare the LSST project ready to enter the “steady state” Operations Phase.

30. Science Verification Plan Summary
Science goals and system technical parameters are connected through, and communicate via, data properties
Science Verification activities are designed to demonstrate compliance of data products with SRD requirements
Start of a more comprehensive characterization of system capabilities
Three periods of sustained on-sky observations and bulk data processing
Data quality assessments begin as soon as possible during AI&T of ComCam and LSSTCam to allow multiple opportunities for feedback between data quality and hardware / software
Two operational readiness mini-surveys exercising the Level-1 and Level-2 data production systems comprise the Final Science Verification phase
Science Verification activities, the Commissioning Plan, and the construction project conclude with Operational Readiness Review