1. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.41 Gamma-ray Large Area Space Telescope GLAST Large Area Telescope: Instrument Science Operations Center CDR Section 3.4 Science Data Products and Science Analysis Software Richard Dubois SAS System Manager

2. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.42 Outline Introduction to SAS Scope and Requirements Overall Test Plan Data Challenges DC1 Summary

3. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.43 Science Analysis Software Overview Data Pipelines –Prompt processing of Level 0 data through to Level 1 event quantities –Providing near real time monitoring information to the ISOC –Transient sources –Monitoring and updating instrument calibrations –Reprocessing of instrument data Performing bulk production of Monte Carlo simulations Higher Level Analysis –Creating high level science products from Level 1 Point source catalogue –Providing access to event and photon data for higher level data analysis Interfacing with other sites (sharing data and analysis tool development) –Mirror PI team site(s) –SSC Supporting Engineering Model and Calibration tests Supporting the collaboration for the use of the tools

4. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.44 Level III Requirements Summary Ref: LAT-SS-00020

5. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.45 RD LAT sim/recon System Architect Simulation ACD CAL TKR GEANT4 Recon ACD CAL TKR Event Interpretation System Tests Release Manager Flight Int Support I/O ScienceTools ArchitectsAnalysis Tools Likelihood Pulsars GRBs Obs Sim User Interface Release Manager I/O Infrastructure Code Distribution Release Manager System TestsData Server Issues TrackerDocumentation SLAC Linux environment SLAC Windows Environment Sundry UtilitiesCaliibrations Event Display Code Management Tool There is a name for each box, except Infrastructure lead. Asking for SLAC to fill this position. Making do for now.

6. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.46 SAS in the Ground System Pipeline (Gino) is robotic backbone of ISOC process handling – Performs L1 & L2 processing Keep everything on disk and archive to tape Gino server and database can handle multiple arbitrary sequences of tasks: L1 pipeline; reprocessing; MC; ….

7. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.47 Pipeline Spec Function –The Pipeline has five major functions automatically process Level 0 data through reconstruction (Level 1) provide near real-time feedback to IOC facilitate the verification and generation of new calibration constants produce bulk Monte Carlo simulations backup all data that passes through Must be able to perform these functions in parallel Fully configurable, parallel task chains allow great flexibility for use online as well as offline –Will test the online capabilities during Flight Integration The pipeline database and server, and diagnostics database have been specified (will need revision after prototype experience!) –database: LAT-TD-00553LAT-TD-00553 –server: LAT-TD-00773LAT-TD-00773 –diagnostics: LAT-TD-00876LAT-TD-00876

8. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.48 Pipeline in Pictures State machine + complete processing record Configurable linked list of applications to run Expandable and configurable set of processing nodes

9. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.49 Overall Test Plan Combination of Engineering Model tests, Data Challenges and LAT Integration Support EM tests –EM1 demonstrated ability to simulate/reconstruct real data from single (non-standard) tower All within standard code framework/tools Data analyzed with SAS tools Data Challenges –End to end tests of sky simulation through astro analysis –Exercise pipeline LAT Flight Integration –Combine tools from EM & DC applications –Sim/recon/analysis & pipeline processing and record keeping

10. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.410 Processing Flow Data Pipeline Sim Raw Data Level 0 Recon Level 1Science Tools Level 2 Recon: interpret LAT readout and estimate directions and energies; flag background Sim: full modeling of e/  /p interactions and readout in the LAT

11. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.411 Level 1 Sim/Recon Chain Real Data 3 GeV  Level 1

12. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.412 Sim/Recon Toolkit PackageDescriptionProviderStatus ACD, CAL, TKR Recon Data reconstruction LAT90% done In use ACD, CAL, TKR Sim Instrument simLAT95% done In use GEANT4Particle transport sim G4 worldwide collaboration In use xmlParametersWorld standardIn use RootC++ object I/OHEP standardIn use GaudiCode skeletonCERN standardIn use doxygenCode doc toolWorld standardIn use Visual C++/gnuDevelopment envsWorld standardsIn use CMTCode mgmt toolHEP standardIn use cvswebcvs web viewerWorld standardIn use cvsFile version mgmtWorld standardIn use

13. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.413 Software Development Approach Enable distributed development via cvs repository Extensive use of electronic communications –Web conferencing (VRVS), Instant Messaging (icq), Wikis CMT tool permits equal development on Windows and Linux –Coupled with cvs gives machinery enables config control –Superior development environment on Windows; compute cycles on linux documentation and coding reviews enforce coding rules JIRA issue tracking system “Continuous integration” –Eliminate surprises for incoming code releases –Build code every night; alert owners to failures in build or running of unit tests. Results tracked in database. –Developing comprehensive system tests in multiple source configurations. Track results in database; web viewable.

14. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.414 Using Jira Issue Tracker

15. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.415 Nightly Builds Past release Release in progress Future release Display created from database query Performing builds for Science Tools now also Build status Unit test status

16. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.416 System Tests Comparison of current to previous release.

17. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.417 Main Science Tools PackageDescriptionFirst Use LikelihoodWorkhorse model fitting for detection & characterization of cosmic gamma-ray sources DC1 Level 1 database access Extracts desired event data DC1 Exposure calculationUses IRFs, pointing, livetime etc. for deriving calibrated source fluxes DC1 Source identificationIdentifies gamma-ray sources with cataloged counterparts at other wavelengths DC2 GRB analysisTemporal and spectral analyses of burst profiles DC1 Pulsar analysisPhase folding & period searching of gamma-ray pulsars and candidates DC2 Observation simulator High level simulation of observations of the gamma-ray sky with the LAT DC1

18. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.418 Science Tools Toolkit PackageDescriptionProviderStatus PIL, PIL++IRAF parameter access HEASARCIn use cfitsio, CCFitsFITS file manipulation HEASARCIn use XSPEC, SherpaFor GRB spectral modeling HEA standardsUnder consideration Rootgui etcHEP standardUnder consideration pythonScriptingWorld standardUnder consideration doxygenCode doc toolWorld standardIn use Visual C++/gnuDevelopment envsWorld standardsIn use CMTCode mgmt toolHEP standardIn use cvswebCvs web viewerWorld standardIn use cvsFile version mgmtWorld standardIn use

19. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.419 Data Products DC1 DC2 under discussion DC2/3 ?

20. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.420 Purposes of the Data Challenges “End-to-end” testing of analysis software. Familiarize team with data content, formats, tools and realistic details of analysis issues (both instrumental and astrophysical). If needed, develop additional methods for analyzing LAT data, encouraging alternatives that fit within the existing framework. Provide feedback to the SAS group on what works and what is missing from the data formats and tools. Uncover systematic effects in reconstruction and analysis. Support readiness by launch time to do all first-year science. S.Ritz

21. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.421 SAS Checklist Detailed Simulation Event Reconstruction Event Classification Instrument Calibration ACD CAL TKR Processing Pipeline MC I&T Re-processing High Level Analysis GRBs, AGN, Pulsars Catalogue, Diffuse User Support Code distribution Data Distribution Institutional Mirrors SSC LAT Mirrors High Level Instr Diags Quicklook DC1 DC2 DC3 I&T EM I&T Flight

22. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.422 Data Challenge Planning Approach Walk before running: design a progression of studies. DC1. Modest goals. Contains most essential features of a data challenge. Original plan: 1 simulated day all-sky survey simulation, including backgrounds find flaring AGN, a GRB recognize simple hardware problem(s) a few physics surprises exercise: –exposure, orbit/attitude handling, data processing pipeline components, analysis tools DC2, start end of CY04. More ambitious goals. Encourage further development, based on lessons from DC1. One simulated month. DC3. Support for flight science production. S.Ritz

23. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.423 Data Challenge 1 Closeout 12-13 Feb 2004 http://www-glast.slac.stanford.edu/software/Workshops/Feb04DC1CloseOut/coverpage.htm

24. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.424 DC1 Components Focal point for many threads –Orbit, rocking, celestial coordinates, pointing history –Plausible model of the sky –Background rejection and event selection –Instrument Response Functions –Data formats for input to high level tools –First look at major science tools – Likelihood, Observation Simulator –Generation of datasets –Populate and exercise data servers at SSC & LAT –Code distribution on windows and Involve new users Teamwork!

25. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.425 Science Tools in DC1 User Interface aspects of the standard analysis environment, such as Image/plot display (UI2), Command line interface & scripting (UI4), and GUI & Web access (UI5) are not shown explicitly. 1 This tool also performs periodicity tests and the results can be used to refine ephemerides 2 These tools can also take as input binned data from other instruments, e.g., GBM; the corresponding DRMs must also be available. 14 Sept 2002 Pulsar ephem. (D4) Level 1 (D1) LAT Point source catalog (D5) Interstellar em. model (U5) Pointing/livetime history (D2) Astron. catalogs (D6) Level 0.5 IRFs (D3) Alternative source for testing high- level analysis Alternative for making additional cuts on already- retrieved event data Pt.ing/livetime simulator (O1) Observation simulator (O2) Pt.ing/livetime extractor (U3) Data sub- selection (U2) Data extract (U1) Exposure calc. (U4) Likelihood (A1) Map gen (U6) Src. ID (A2) Event display (UI1) Pulsar profiles (A3) 1 Catalog Access (U9) Pt.ing/livetime extractor (U3) Pulsar phase assign (U12) Pulsar period search (A4) GRB spectral-temporal modeling (A10) Source model def. tool (U7) Arrival time correction (U10) GRB temporal analysis (A7) 2 GRB LAT DRM gen. (U14) GRB spectral analysis (A8) 2 GRB event binning (A5) GRB unbinned spectral analysis (A9) GRB visual- ization (U13) IRF visual- ization (U8) Ephemeris extract (U11) GRB rebinning (A6) 2 The big picture: Details are changing, but still basically right DC3 Standard Analysis Environment S.Digel and P.Nolan

26. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.426 Science Tools in DC1 Level 1 (D1) Interstellar em. model (U5) Pointing/livetime history (D2) IRFs (D3) Alternative source for testing high- level analysis Alternative for making additional cuts on already- retrieved event data Pt.ing/livetime simulator (O1) Observation simulator (O2) Pt.ing/livetime extractor (U3) Data sub- selection (U2) Data extract (U1) Exposure calc. (U4) Likelihood (A1) Map gen (U6) Pt.ing/livetime extractor (U3) Source model def. tool (U7) GRB LAT DRM gen. (U14) GRB spectral analysis (A8) 2 GRB event binning (A5) All components are still prototypes The DC1 functionality is Data extraction Limited visualization Model definition Model fitting Observation simulation S.Digel and P.Nolan

27. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.427 The data on to individual components! T.Burnett

28. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.428 http://www-glast.slac.stanford.edu/software/DataChallenges/DC1/DC1_StatusAndInfo.htm Documentation User’s Guide Data Description Likelihood Tutorial DC-1 Discussion List Analysis Code download sites Wiki page for sharing results!

29. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.429 Lessons Learned Analysis Issues –Astrophysical data analysis –Software usage and reliability –Documentation –Data access and data server usage –UI stuff –Software installation and release –Software infrastructure & framework –Communication and Time frame Infrastructure Issues –SciTools did not run on windows at the last minute –We discovered problems with sources and ACD ribbons late –Manual handling of the processing –No checking of file integrity –Large failure rate in batch jobs (~10%) –Tools are not checking inputs much –Code distribution scripts were written manually Issues have been addressed

30. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.430 DC2, based on lessons from DC1 –1 simulated month of all-sky survey gammas (backgrounds: see next slide) –key sky addition: source variability AGN variability, including bright flares, quiescent periods expand burst variety (and include GBM? see later slides) pulsars, including Gemingas, w/ orbit position effects. –more realistic all-sky attitude profile –background rate varies with orbit position –more physics surprises, and add nominal hardware problems (and misalignments?), add deadtime effects and corrections –Analysis Goals: produce toy 1-month catalog and transient releasesproduce toy 1-month catalog and transient releases detailed point source sensitivity and localization studies first systematic pulsar searches (timing!); detailed diffuse analyses recognize simple hardware problems (connect with ISOC/SOG) –benchmark: processing times, data volume, data transfers. Strawperson Updated Plan for DC2 S.Ritz

31. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.431 SAS Support of Flight Integration Expected Deliverables –Automated processing pipeline with bookkeeping Gino: in test now Revised draft pipeline spec at http://confluence.slac.stanford.edu/display/pipe/Functional+Spec+for+Processing+Pipeline –Stable Simulation/Reconstruction package Frozen as of 8/4 (more versions will be necessary before actual use) Special system tests for surface muon data –Calibration algorithms for TKR, CAL – due by 1 Sept TKR hot/dead strips –Later: ToT gains, inter-tower alignment CAL peds, gains, light tapers –Calibration infrastructure for input to recon - rdbGui Next step is allowing for binary format of parameters (800k ToT gains!) –Event display – FRED Ready –Will inaugurate CCB process for code used in Pipeline –Plan to extend System Tests scheme to deliver the High Level Diagnostics

32. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.432 Pipeline View

33. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.433 Example of FRED

34. GLAST LAT Project ISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 3.434 Summary We believe that EMs, DCs and Flight Integration will leave us ready for flight EM1 worked with our tools DC1 worked well, showing very good capabilities from sky modeling through astronomical analysis –Plenty of work still to so, but reasonably understood –Will be demonstrated in DC2, 3 and LAT Integration, 16- tower cosmic ray tests and the beam test prior to launch LAT Flight Integration prep in full swing now DC2 within a year (being negotiated)