1. Computing and Software – Calibration Flow Overview
Kay Graf - ECAP
Timecalibration Workshop
Nikhef, May 2017

2. Calibration and Data Processing

3. Calibration and Data Processing
Data processing and analysis
Calibration application
Calibration determination
Interplay of calibration and analysis:
Determination of calibration
Application of calibration (to data)
(Offline) Data processing: simulation, reconstruction, analysis
Example from time calibration: analysis drives calibration
Timecalibration WS - Comp&Soft — K. Graf — May 2017

4. Data Processing: Simulation Chain
Simulation Data Format
Event Generation
Event Simulation and Cherenkov Light Generation
Online Data Format
Background Light Generation
Detector Response
Data Filtering
Offline Data Format
Event Classification
Event Reconstruction
Summary Data Fromat
Analyses
antcc-format (.evt)
Geometry, Configuration, Monitoring and Calibration Data (File, DataBase, ascii, .detx)
Jpp format (.root)
Jpp root format/aanet (.root)
aanet offline format (.root)
Processing chains for simulation and data should be as equal as possible, i.e. use of the same algorithms/programs as much as possible
Comp&Soft WG — K. Graf — KM3NeT Collaboration Meeting — February 2015

5. Data Processing: Data Chain
detector
offshore
CLB – recording (Electronics WG)
shore station
DataQueue – receiving, sorting / DataFilter – filtering/DataWriter – writing
online monitoring
quasi-online
Postprocessing/data quality –add info
quasi-online processing – quick reco, high-level monitoring
offline processing
calibration / data quality
reconstruction
analysis
at institutes (user)
develop analysis software package
UDP – CLB data format
Jpp root format/controlhost
Geometry, Configuration, Monitoring and Calibration Data (File, DataBase, ascii, .detx)
Jpp root format
offline root format (aanet)
Processing chains for simulation and data should be as equal as possible, i.e. use of the same algorithms/programs as much as possible
Comp&Soft WG — K. Graf — KM3NeT Collaboration Meeting — February 2015

6. Interface Calibration⬄ Simulation/Analysis
Some thoughts:
To allow for consistent calibration information parsing and efficient resource management (developing, testing and processing) calibration tasks/processing should have clear interace definitions
i.e.: the determination of calibration can be done in analysis software but in a separate programs/algorithms
That does not mean that development/improvement/testing of calibration data in analysis is discouraged, on the contrary – we calibration with drive analysis and vice-versa for the next years.
Necessities:
Clear interfaces of calibration data are needed → database
Clear definitions for calibration streams/handling in the analysis software
“intelligence” in the handling of calibration data to improve on calibration
Timecalibration WS - Comp&Soft — K. Graf — May 2017

7. Combined fit ( line shape) → future Other calibrations
General Scheme
Detector
Acoustic system
Orientation system
Database
Position Calibration
Positions
Orientation
Combined fit ( line shape) → future
Other calibrations
Offline Analysis/Reco
(could extent to quasi-online)
Timecalibration WS - Comp&Soft — K. Graf — May 2017

8. What we currently use in offline calibration
Position/orientation in detx – derived from standard detector geometry
hard coded values
no orientation at all
T0 in detx – derived from in situ measurements + dark room (see time calibration procedure)
Detx format:
Human readable
Large overhead (4MB for a buidling block)– redundant information on DOM geometry)
only contains position, orientation and T0
Other calibration parameters (especially from K40 analysis)
As ASCII files
Timecalibration WS - Comp&Soft — K. Graf — May 2017

9. global_det_id format_version\n UTC_validity_from UTC_validity_to \n
Dataformats
See
Detx (v2)
global_det_id format_version\n UTC_validity_from UTC_validity_to \n
UTM_ref_grid UTM_ref_easting UTM_ref_northing UTM_ref_z \n
ndoms \n
dom_id line_id floor_id npmts \n pmt_id_global x y z dx dy dz t0 \n pmt_id_global x y z dx dy dz t0 \n
...
pmt_id_global x y z dx dy dz t0 \n
#repeat for each dom
Timecalibration WS - Comp&Soft — K. Graf — May 2017

10. What we need
Possibility to stream all calibration data from the database to analysis codes
Stream should be efficient (small, only necessary information)
Unique identification of the “best” available calibration on an event basis NB: ”best” could change in time
possibility to use an intermediate data format for caching
For now: position/orientation data, however: scheme should be extendable also to other calibration data, e.g. from K40 analysis
Timecalibration WS - Comp&Soft — K. Graf — May 2017

11. Calibration Data for Offline Analysis
Discussion between comp&soft, calibration and DAQ group
Improvement necessary in data bookkeeping and data transfer
Many parameters handled via ASCII files (MonitorK40 output, PMT parameters, etc).
Better quality control needed
Challenge: find the appropriate, best calibration for each individual event; pull the necessary information from the database
Strategy defined:
the detx file contains all general detector information for a given run – as is the case now;
Additional (light-weight) calibration stream to provide data changing during run
Position/orientation, (PMT information)
Hosted at separate tables in the DB that mark the interface from calibration to analyses
Giorgio to finalise the proposal
Comp&Soft WG — K. Graf — PSC Meeting, May 2017

12. Proposal Store data in dedicated DB table(s) Time info (UTC time)
Detector ID
Calibration tag (define e.g. online, offline v1, v2, v3 calibration sets)
(RUN)
DOM ID
Position (x,y,z)
Orientation (yaw, pitch, roll)
... Repeat 1-7 for all DOMs in detector or: repeat 5-7 for all DOMs in detector?
Detx provides the general setting (detector setup; DOM ID to position in detector; DOM geometry; T0s)
Position/orientation is streamed as ntuple
⇒ result: detector object in the individual analysis/reco code
Timecalibration WS - Comp&Soft — K. Graf — May 2017

13. How to retrieve from and store to the database calibration-related information
Timecalibration WS - Comp&Soft — K. Graf — May 2017

14. Database Interaction
Via dedicated pages, see
E.g. detx: see for all available detx-files
Stream Data Service: ascii file streams, see
XML and JSON Data Service: xml, json data file up- and download, see
E.g. tests/calibration upload: meters
Special file parsers (for Excel files):
Registration and Integration data upload
Timecalibration WS - Comp&Soft — K. Graf — May 2017

15. How to define tests/calibrations
Define parameters:
Define test type:
Link test type with parameters:
Upload tests via XML service
Timecalibration WS - Comp&Soft — K. Graf — May 2017

16. Handle Calibrations via Tags
we can handle calibration sets – by combining calibrations ⇒ add to run generation → easy to generate (xml file upload possible)
Timecalibration WS - Comp&Soft — K. Graf — May 2017