1. 1 11 1 Conditions Database Status and Plans for 2005 Andrea Valassi (CERN IT-ADC) LCG Applications Area Review 31 March 2005

2. 2 22 2 Andrea Valassi IT-ADC31-Mar-2005Conditions Database The scope of the project Conditions data: non-event data that vary with time –And may also exist in different versions Data producers from both online and offline –Geometry, readout, slow control, alignment, calibration... Data used for event processing and more –Detector experts, alignment/calibration, event reconstruction/analysis… Scope of the common project? –Common software and tools for time-varying and versioned data –Not the generic access to distributed relational data (RAL, 3D…) –Not the problems specific to one experiment or one data type –Scope has grown larger in time (from offline to online use case)

3. 3 33 3 Andrea Valassi IT-ADC31-Mar-2005Conditions Database Background and introduction Project launched in 2003 within LCG Persistency Framework –Kick-off workshop at CERN in December 2003 –Project leader: A.V. (CERN IT-ADC) –Present staff (Atlas-LHCb): ~1 (common work) + ~2 (exp. integration) –Weekly phone meetings with ~5-10 people connected on average Background of common activities in 2000-2003 –Collaboration between IT-DB, LHCb, Atlas, COMPASS, Harp –C++ API definition and Objectivity implementation –Oracle implementation of the original API (“BLOB” data payload) –MySQL implementation with API extensions (user-defined payload) Two goals in the mandate for the common project –Integrate the existing Oracle and MySQL packages into LCG AA –Coordinate new development of API, software and tools

4. 4 44 4 Andrea Valassi IT-ADC31-Mar-2005Conditions Database LCG integration of existing Oracle and MySQL packages –Apr.04 (CONDDB 0.1.0 ): two separate packages in LCG CVS and SCRAM –Jul.04 (CONDDB 0.2.0) : common functionality factored out behind common API, support for gcc3.2.3 for both implementations –Atlas test beam data taking using Lisbon MySQL package Discussion about new API, software and tools –Review limitations of API and software, collect new requirements Project faced two main problems in this phase –Lack of manpower for new developments –Divergence of the two packages Decision to start new implementation taken in October2004 –During public presentation at the Application Area Meeting First phase of the project (Dec.2003 - Oct.2004)

5. 5 55 5 Andrea Valassi IT-ADC31-Mar-2005Conditions Database Original “common API” Designed to handle data “objects” that –Can be classified into independent data items –VARY WITH TIME –May have different versions (for given time and data item) Pere Mato (Feb 2000) A “CondDBObject” A CondDBObject has –Metadata: Data item identifier Start-of-time-validity End-of-time-validity Version number –Data “payload”: Encoded as BLOB/string

6. 6 66 6 Andrea Valassi IT-ADC31-Mar-2005Conditions Database Atlas-Lisbon API extensions Support for data payload other than BLOBs –“ICondDBTable” interface for user-defined data fields –ICondDBTable “with ID” for improved data item addressing Support for “online” data not requiring versioning –Measured (temperature) rather than computed (alignment) MySQL implementation of extended API –Useful tools built above extensions (PVSS, data browser) –Used for Atlas test beams and integrated with Athena

7. 7 77 7 Andrea Valassi IT-ADC31-Mar-2005Conditions Database Why a new software implementation? CondDBOracle: poor performance and missing functionalities –Bulk insertion and retrieval missing in key points of the code: very slow! –No user defined data payload (only BLOBs) –No “online” mode (only versioned data) CondDBMySQL: improvements necessary in the API and schema –Cleaner API: whole table vs. individual row (object); schema vs. data –Unified approach to schema and code for online and versioned conditions data Minimize differences between Oracle and MySQL implementations –Use the same data model and schema to simplify data copy across backends –Address new user requirements (eg user tags, HVS) using a unified approach –Minimize duplication of effort, factor out common code whenever possible Maximize integration with and reuse of LCG software (SEAL/POOL) –AttributeList, Time, plugin loading, RAL (also grid/security), POOL integration…

8. 8 88 8 Andrea Valassi IT-ADC31-Mar-2005Conditions Database New software: overview (“COOL” – Conditions Objects for LCG) Idea: merge best of previous APIs and reimplement for Oracle/MySQL Single implementation for Oracle/MySQL using RAL –Strictly the same relational schema for both back-ends –Direct implementations (MySQL, OCCI, …) may always come later User-defined data payload modeled by POOL AttributeList –Support for simple types (int, float, bool, string) of various precisions –Support for CLOB/BLOB data types as just two of many possible data types Support both multi-version “offline” and single-version “online” data –Open-ended IOV (t1,+inf) updated as (t1,t2) on next insertion of (t2, +inf) Support “channel ID” for improved data addressing –Using the same approach for both single-version and multi-version data

9. 9 99 9 Andrea Valassi IT-ADC31-Mar-2005Conditions Database COOL software – a few technical details LCG infrastructure (CVS and viewcvs, SCRAM build system) –Platforms: rh73_gcc32(3)(_dbg), slc3_ia32_gcc323(_dbg), win32_vc71_dbg Two packages –CoolKernel (public API) Compile dependency on SealBase (exceptions, Time, int64) and POOL AttributeList –RelationalCool (implementation using POOL RAL) COOL database service is a loadable SEAL service and uses SEAL logging Compile dependency on SealServices and POOL RelationalAccess Runtime loading of POOL OracleAccess/ODBCAccess (Oracle or MySQL on demand) Separate classes for generic ‘Relational’ implementation and ‘Ral’-specific DB access (foresee the possibility to reuse the same schema for direct MySQL/OCCI access) Database implementation issues –Unit tests routinely run against Oracle/MySQL (change the connection string) –Execution plan for each operation is tested to ensure Oracle indexes are used –Bind variables and bulk operations are used whenever possible –Each user operation offered by the API is encapsulated within one transaction

10. 10 Andrea Valassi IT-ADC31-Mar-2005Conditions Database Data payload: typical use cases channelIDsince(tag)tillpressuretemperature channelIDsince(tag)tillCLOB FK clobIDCLOBchannelIDsince(tag)tillclobID FK channelIDsince(tag)tillFK1FK2PK1PK2?? channelIDsince(tag)tillPOOL string token …….……….. …….…xxxxx xxxxxxxxxxxx xxxx…………............…….. POOL file POOL Conditions Database “core” responsibility Plugin-specific responsibility (may be experiment-specific) Example: XML interpreter Payload inside the CondDB Inline attributes Inline CLOB Referenced CLOB Payload outside the CondDB POOL token Relational FK Still missing in COOL core: 1. CLOB support (next week) 2. Out-of-line LOB (later: only storage overhead, no functional difference)

11. 11 Andrea Valassi IT-ADC31-Mar-2005Conditions Database COOL software development –Dec.04: single-object read/write for single-version (online) mode –Dec.04: bulk read/write for single-version (online) mode –Feb.05: initial performance studies (very useful feedback for RAL) –Feb.05: multi-version (offline) mode with tagging –Mar.05: last changes (int64, CLOB) and fixes before production release –First production version of COOL should be released next week –Atlas plans to use COOL for the 2005 detector integration data taking Aggressive schedule in spite of relatively limited resources –Two developers (A.V. @~50% and Sven A. Schmidt @~80%) With useful contributions from Antoine Perus (performance studies), Marco Clemencic (tests and debugging) and feedback from other users –Work carried out in constant collaboration with the SPI, SEAL and POOL/RAL teams (many thanks especially to Radovan and Ioannis!) Second phase of the project (Nov.2004 – Mar.2005)

12. 12 Andrea Valassi IT-ADC31-Mar-2005Conditions Database Production release (Q1-Q2) and documentation (Q2) –A formal review of the production release should follow (Q2) Performance validation on reference workload (Q2-Q3) –Need realistic reference workload from the experiments Support COOL deployment in Atlas at Tier0 (Q2-Q3) –In collaboration with CERN Physics Database Services Data slicing API and data distribution tools (Q4) Other priorities need to be agreed with the experiments –Handling of external data payload (eg POOL tokens)? –COOL integration with FroNTier distribution model? –Core API enhancements (user tags, multichannel browsing, HVS…)? Work plan proposal until the end of 2005 (Apr.2005 – Dec.2005)

13. 13 Andrea Valassi IT-ADC31-Mar-2005Conditions Database Conclusions Project has started off slowly but is getting momentum –Mainly thanks to convergence on a single API and data model –Development manpower relatively limited but focused on common goal Milestones in 2004: –LCG integration of existing Oracle/MySQL packages with common API –Review of existing software and decision for a new implementation Milestones in 2005: –First production release of COOL expected next week Merging the best ideas from the older packages and implementing them for both Oracle and MySQL using a consistent approach Reducing the duplication of effort with other LCG projects