1. European Organization For Nuclear Research CERN Accelerator Logging Service Overview Focus on Data Extraction for Offline Analysis Ronny Billen & Chris Roderick 15 th March 2010

2. 15-Mar-2010Forum on Interfacing to the Logging Database for Data Analysis2 Outline  Logging History  Today’s CERN Accelerator Logging Service  Architecture and Current Status  Data Extraction  Summary

3. Logging History  The LHC Logging System Sub-project launched by LHC Controls Project in Sep-2001  Original mandate Analysis, design, procurement of Logging Facilities for the future LHC Controls System  Original goals Information management for LHC performance improvement Meet INB requirements for recording beam history Make available long term statistics for management Avoid duplicate logging efforts  Scope creep over the years Cover complete CERN accelerator complex LHC Hardware Commissioning 15-Mar-2010Forum on Interfacing to the Logging Database for Data Analysis3

4. 15-Mar-2010Forum on Interfacing to the Logging Database for Data Analysis4 Today’s CERN Accelerator Logging Service  Persistence of logged time-series data for the lifetime of LHC  Based on the Oracle database management system  On-line availability of all data  Unique architecture: parallel reads/writes of same data  Data extraction user interface with common functionality Selection of variables, time range Fast statistics on query result Graphical visualization of data set Data extraction to different file formats  Data extraction requirements have evolved significantly First order data manipulation (interpolation, averaging,…) Applicative interface to the data extraction methods

5. PL/SQL filtered data transfer Architecture 15-Mar-2010Forum on Interfacing to the Logging Database for Data Analysis5 LDB MDB Equipment – DAQ – PLC Equipment – DAQ – FEC ffffffffffff fff f ffff f ELEC COMMCV EAU TIM f fff f QPSPIC SU Coll CNGS Exp Cryo CIET WIC VACRad BLM BETSBIC BCTBPM FGC MSMK VAC ~20 Years filtered data 7 Days raw data Filters for data Reduction

6. PL/SQL filtered data transfer Current Status 15-Mar-2010Forum on Interfacing to the Logging Database for Data Analysis6 LDB MDB Equipment – DAQ – PLC Equipment – DAQ – FEC ffffffffffff fff f ffff f ELEC COMMCV EAU TIM f fff f QPSPIC SU Coll CNGS Exp Cryo CIET WIC VACRad BLM BETSBIC BCTBPM FGC MSMK VAC ~20 Years filtered data 7 Days raw data ~ 200’000 Signals ~ 50 data loading processes ~ 5.1 billion records per day ~ 130 GB per day  46 TB per year throughput ~ 800’000 signals ~ 300 data loading processes ~ 3.8 billion records per day ~ 105 GB per day  38 TB per year stored > 300 extraction clients 0.4  2 million extraction requests per day

7. Current Status 15-Mar-2010Forum on Interfacing to the Logging Database for Data Analysis7

8. 15-Mar-2010Forum on Interfacing to the Logging Database for Data Analysis8 Data Extraction – No SQL Access  Direct database access must be avoided Not scalable across all clients  Number of connections  Security considerations  Volatile infrastructure Not secure  Badly written queries / application logic will crash the entire service! Not performant  Most programming languages provide database access  Few languages optimized to work with Oracle in a performant manner

9. 15-Mar-2010Forum on Interfacing to the Logging Database for Data Analysis9 CERN Accelerator Logging Service Data Extraction – Java API Spring HTTP Remoting Custom Java Applications (currently > 30) 10g AS Spring HTTP Remoting metadata JDBC TS Data JDBC Metadata TIMBER LDB MDB

10. 15-Mar-2010Forum on Interfacing to the Logging Database for Data Analysis10 Data Extraction – Java API  Java has been the strategic choice for all high-level control room applications  Our Java API to the Logging Service is available since several years Well documented  http://slwww.cern.ch/~pcrops/releaseinfo/pcropsdist/dm/logging-data-extractor- client/PRO/build/docs/api/ http://slwww.cern.ch/~pcrops/releaseinfo/pcropsdist/dm/logging-data-extractor- client/PRO/build/docs/api/ Easy to use  Sample code available Heavily used (> 30 custom applications + TIMBER) Fully optimized and instrumented, essential for us to monitor and guarantee the Service Provides secure access to databases hidden on Technical Network

11. 15-Mar-2010Forum on Interfacing to the Logging Database for Data Analysis11 Data Extraction – Java API  JDBC fulfils our requirements, particularly with respect to Oracle performance, as it supports: Connection Pooling Statement Caching Bind Variables Flexible Array Fetching  3-Tier architecture has many more benefits Resource pooling (connections, statements) Database protection Database isolation, since users don’t need to care about:  Database schema  Server details and login credentials  Access to Technical Network

12. 15-Mar-2010Forum on Interfacing to the Logging Database for Data Analysis12 Summary  The Logging Service is being heavily used.  Data access must be controlled & monitored to ensure continuity of the overall service.  A well documented, easy to use Java API is available for data extraction.  Care should be taken as to how data is extracted, since misuse will impact the whole service, including data writing!