1. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t COOL – Status and Plans Andrea Valassi (CERN IT-PSS-DP) LCG Application Area Review, 19 September 2006

2. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 2 Outline Issues from March 2005 LCG AA review Project overview –Project organization and manpower –Software infrastructure –Release process Development status and plans –Releases (and a few highlights) since the last review –Work in progress and plans COOL deployment in LHCb and Atlas

3. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 3 Issues from March2005 review “COOL is very young and has yet to succeed” –Several production releases since then –Being deployed in Atlas and LHCb (success in Pere’s terms!) “Commitment from Atlas and LHCb to use COOL; CMS is also considering using COOL” –Continued commitment (~and manpower) from Atlas/LHCb –COOL not used by CMS afaik (but no clear statement yet) “Experiments interested in COOL should commit more manpower to ensure project survival” –Manpower was adequate (not abundant) in 2005-2006 –Present manpower situation critical again since June 2006 (both Atlas collaborators left - no more Atlas contribution)

4. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 4 COOL development team Active manpower A.V. (CERN - IT/PSS) –80% FTE since Oct. 2004 –Project coordination, core development and release mgmt Marco Clemencic (CERN - LHCb) –20% FTE since Nov. 2004 –Core development and release mgmt With useful contributions from many other people –David Front (IT/LCG since April 2005) - stress tests –Richard Hawkings and other ATLAS users, testers and DBAs –The CORAL, SEAL/ROOT, SPI and 3D teams Former collaborators Sven A. Schmidt (Mainz - ATLAS) –80% FTE (Oct. 2004 to June 2006) - back at 20% FTE in October? –Core development Uli Moosbrugger (Mainz - ATLAS) –80% FTE (Sep. 2005 to March 2006) – performance optimization Kristoffer Dahl (CERN - IT/PSS summer student 2006) –60% FTE (July-August 2006) - performance optimization

5. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 5 COOL - doc and organization Project web page –http://lcgapp.cern.ch/project/CondDBhttp://lcgapp.cern.ch/project/CondDB Weekly phone meetings – with detailed minutesminutes –5 to 10 people on average “Usual suspects”: Andrea, (Sven,) Marco, Richard, RD… Mailing lists with archives –developers (high traffic), general (low traffic)developersgeneral Savannah portal – heavily used for bugs and tasksSavannahbugstasks Code documentation and user guide via dOxygendOxygen –With links to the SEAL and CORAL pages –Additional user documentation on twikitwiki

6. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 6 COOL - build infrastructure SCRAM v0.2 for build and configuration –Initially copied from POOL/CORAL – still a large overlap Several COOL-specific hacks over POOL and SPITOOLS –Good points Granularity of package dependencies Easy to use several platforms in parallel Stable and satisfactory situation (after initial learning curve) –Bad points May be painfully slow (especially on Windows) Many independent configuration files Future options –Would not move away unless strictly necessary Any migration would require manpower! Must port/preserve COOL-specific configuration (eg WINE) –Would rather consider CMT than SCRAM v1.0 Some CMT experience in the COOL team Both Atlas and LHCb use CMT for their own software

7. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 7 COOL - 8 supported platforms Linux –32 bit platforms: all official platforms are supported 7 supported platforms: slc3_ia32_gcc323(_dbg/_gcov), slc3_ia32_gcc344(_dbg), slc4_ia32_gcc345(_dbg) –64 bit platforms (AMD64): will be supported in COOL1.4.0 See next slide for AMD64 details – data type issues in API 2 test platforms: slc4_amd64_gcc345(_dbg) Agreed with experiments to drop slc3_amd64_gcc345(_dbg) Windows –Software built and tested using WINE on Linux 1 supported platform: win32_vc71_dbg Mac –Private builds on osx103_gcc33 since 2004 till June 2006 –No port to osx104_ppc_gcc401 yet

8. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 8 COOL - AMD64 status AMD64 test builds since COOL 1.3.1 (April 2006) –Many failures in functional tests in early builds identified as external package bugs – now all fixed MySQL/SQLite types in CORAL (fixed in 1.5.1 - July) Exception handling in ROOT (fixed in 5.12.00 - July) Pending issue in API: data type precision when creating new tables – will be fixed in COOL 1.4.0 –Present API: create payload column of type ‘long’ Delegate payload specification to CORAL This creates 32/64-bit persistent data on 32/64-bit Linux (creating data on AMD64 and reading it back on Linux32 fails) –Future API: create payload column of type ‘int32’ or ‘int64’ COOL-specific payload specification

9. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 9 COOL - s/w dependencies (1) RelationalCool (Coral implementation) CoolKernel (abstract interfaces) CoolApplication (context and factories) PyCool (python bindings) SealKernel Examples (user examples) Only PyCool depends on ROOT RelationalCool, CORAL plugins and SealServices are loaded at runtime (SEAL plugin manager) COOL depends on SEAL both via CORAL and directly (not all dependencies are shown) CoralBase (AttributeList) RelationalAccess (Coral relational API) SealBase (int64, Time) SealPlatform PluginMgr SealUtil (SealTimer) boost SealServices (Application, MessageSvc…) reflex root python In addition: Utilities (C++), PyCoolUtilities (python)

10. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 10 COOL - s/w dependencies (2) Fine granularity of package dependencies –Possible thanks to the use of SCRAM (or CMT) ‘scramShowUses’ (COOL tool), ‘cmt show uses’ Please keep SEAL/CORAL substructure if migrate to CMT… –Only need to rebuild COOL if specific packages change E.g. no need to rebuild if CORAL plugins change –Only need to build relevant SEAL/CORAL packages Private SEAL/CORAL builds routinely used for debugging ROOT dependency (only) via PyCool –COOL rebuild release often needed when ROOT changes –It would be nice to clarify which ROOT packages Reflex depends on - and also to minimise these dependencies SEAL plugin manager and component model –Positive experience (in spite of some bugs - being fixed) –Stability has been a plus - concern if ROOT migration

11. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 11 COOL - backend support Oracle –From the start (replace old CondDBOracle) –Main focus of development and performance optimization MySQL –From the start (replace old CondDBMySQL – “Lisbon API”) SQLite –Since COOL 1.2.1 (July 2005) Frontier –Since COOL 1.3.2 (May 2006) –Read-only backend - cannot reuse the same test suite A lot of work in the last two months and still in progress Same relational schema for all backends –A choice (for COOL-independent copies), not a necessity Cross-backend replication (e.g. Octopus) not tested yet

12. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 12 COOL - software releases COOL development started in November 2004 –First production release 1.0.0 in April 2005 –Latest production release 1.3.3 in August 2006 Current release numbering (after some evolutions) –Minor release (e.g. 1.3.0) for important enhancements with schema changes and backward-incompatible API changes –Bug-fix release (e.g. 1.3.2) for enhancements with no schema changes and backward compatible API changes –Rebuild release (e.g. 1.3.2a) for rebuilds using the same source code and changes only in external dependencies 21 software releases so far – details later –4 minor releases, 14 bug-fix releases, 3 rebuild releases –Rebuild releases are a very recent concept (1.3.2a-b-c)

13. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 13 COOL - development process In retrospective, priority was given to: –Functional tests Debug COOL code and external packages –Configuration changes and rebuilds –Addition of new features While trying to minimise API and schema changes –Internal documentation –Internal cleanup –Performance fixes –User documentation Which may have suffered from this… Different order for the actual effort spent

14. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 14 COOL - functional tests The most important part of the release process! –This is what takes time (and often causes delays…) Checking results – but especially fixing the bugs –As many lines of code (~25k) as the implementation code Three separate test suites –RelationalCool (C++), PyCool and PyCoolUtilities (python) –All must succeed on the 8 supported platforms –Results documented on the “platform support” page“platform support” page –The same (C++ or python) user code is executed against all backends (changing only the database URL) Useful to debug COOL as well as external software –CORAL, SEAL, ROOT, Frontier client and server...

15. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 15 COOL - CORAL feedback Tight collaboration with CORAL –Excellent interaction with CORAL team –Reuse of configuration infrastructure –COOL-oriented enhancements of CORAL –Fast support and bug fixes –“Hidden” team – CORAL does a lot of the COOL job! CORAL thoroughly tested within COOL –Same code tested against all 3 (or 4) backends All available data types are routinely tested –Contributed many bug reports (and many fixes) –Mutually beneficial collaboration! Benefit from separation of RAL from POOL –No software dependency of COOL on POOL

16. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 16 COOL - conditions data model Main use case: fetch data valid for given item, time and tag –Typical query: ‘what is the ECAL calibration set (computed using the v.215 algorithm) to be used to reconstruct run #347?’ –Inverse lookup (from payload to time/version) not the main priority Pere Mato (Feb. 2000) Each conditions DB object has –Metadata: Data item identifier Interval-of-validity [since, until] Version information –Data “payload”: Actual data values (temperatures, calibration parameters…) Separate fields or encoded as LOB

17. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 17 COOL - basic functionality Conditions data insertion and retrieval –Individual object as well as bulk insertion/retrieval –Bulk retrieval returning a full result set or a server-side cursor –Single channel as well as multi-channel bulk retrieval One SQL statement for different channels in the same table Multi-channel bulk insertion still not fully implemented IOV versioning with handling of IOV overlap –At any given event time, there is never more than one object valid in the “HEAD” or in any given “tagged version” of the data Also “user” tags and “HVS” tags Data type support –Simple data types via the CORAL AttributeList

18. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 18 Initial developments –Nov-04: start of COOL development –Dec-04: single-version mode (bulk object read/write) –Feb-05: performance studies (disable scrollable cursors) –Feb-05: multi-version mode with tagging –Mar-05: int64 validities, handling of SQL types Software releases (omitting most bug fixes) –COOL 1.0.0 (Apr-05): first public release –COOL 1.0.1 (Apr-05): bug fixes –COOL 1.0.2 (May-05): listChannels, listTags, new CVS –COOL 1.1.0 (May-05): basic multi-channel bulk insertion –COOL 1.2.0 (Jun-05): IFolderSet, setDescription, performance improvements, AuthenticationService –COOL 1.2.1 (Jul-05): untag/retag, SQLite, Examples, Oracle privilege mgmt, RAL fixes (pthread lock, #open cursors) –COOL 1.2.2 (Jul-05): bug fixes in SEAL –COOL 1.2.3 (Aug-05): PyCool, multi-channel bulk retrieval, CLOB, user guide COOL – evolution 2004-2005

19. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 19 Software releases (omitting most bug fixes) –COOL 1.2.4 (Sep-05): RAL bug fixes (memory leaks) –COOL 1.2.5 (Oct-05): tools for Oracle table statistics –COOL 1.2.6 (Nov-05): bulk retrieval with server cursor, PyCoolUtilities, closeDatabase, countObjects –COOL 1.2.7 (Jan-06): RAL migration, Reflex migration, Wine –COOL 1.2.8 (Jan-06): ROOT migration, gcc344, AMD64 test –COOL 1.2.9 (Mar-06): new CORAL API –COOL 1.3.0 (Apr-06): ConnectionService, HVS, user tags, schema evolution, performance fixes, std::exception –COOL 1.3.1 (Apr-06): bug fixes –COOL 1.3.2 (Apr-06): Frontier, bug fixes –COOL 1.3.2a (May-06): rebuild –COOL 1.3.2b (Jun-06): rebuild –COOL 1.3.2c (Jul-06): AMD64 bug fixes in ROOT/CORAL –COOL 1.3.3 (Aug-06): Frontier bug fixes, SEAL bug analysis, performance report COOL – evolution 2005-2006

20. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 20 COOL from python - PyCool PyCool contains python bindings for the COOL API –It allows fast and interactive use of COOL via python –Python bindings also for a few SEAL and CORAL classes PyCool first added in COOL 1.2.3 (August 2005) –Migration from Reflection to Reflex in Jan 2006 –Migration from SEAL Reflex to ROOT Reflex in Jan 2006 –Re-implemented from scratch in COOL 1.3.0 (April 2006) Reflex-free solution was used for POOL FileCatalog –This different option may be used to pythonize CORAL too –We may consider re-implementing PyCool a third time using this solution to be consistent with CORAL This would also remove all COOL dependencies on ROOT Additional python tools in PyCoolUtilities package

21. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 21 COOL - HVS versioning HVS - Hierarchical Versioning System –In a hierarchy of nodes, where each node can exist in different versions, store persistently the association between tag of parent node and tag of child node –Version of inner nodes is defined by set of child versions Initially discussed for the Atlas detector description –Conditions data note by A.V. and V. Tsulaia (June 2004)note –Atlas note by V. Tsulaia and J. Boudreau (August 2004)note –Implemented for the Atlas detector description since 2004 Version of leaf nodes: versioning of detector structures HVS implemented in COOL 1.3.0 (April 2006) –Folder sets (containers of folders or folder sets) can also be tagged: tag simply points to the tags of the children Version of leaf nodes: versioning of IOVs in a folder

22. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 22 COOL – user tags Very old Atlas offline requirement (David Malon) Assign private “user” tag to an IOV at insertion time –Keep consistent HEAD for subset of IOVs with that user tag (ignore all IOV insertions done by other users) User tags implemented in COOL 1.3.0 (April 2006)

23. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 23 COOL - schema evolution Schema and API may change from COOL 1.X to 1.Y –Even if we try to minimise the frequency of these changes –Last change was in COOL 1.3.0 (user tags and HVS) –Next change in COOL 1.4.0 (dynamic replication) Backward compatibility –Schema evolution scripts added in COOL 1.3.0 (April 2006) –Databases created with COOL 1.2.x must be evolved to 1.3.0 before they can be opened with COOL 1.3.x Forward compatibility –Since COOL 1.3.3, databases created with 1.3.x may be opened with 1.3.y (y<x) if their schema number is the same

24. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 24 COOL - performance (1) Studied both query optimization and use case validation –Focus is Oracle only – not MySQL or SQLite (no manpower…) Scripts to compute table statistics since COOL 1.2.5 (Oct 2005) –Understood tricky features of Oracle execution plans Atlas prompt reconstruction validation studies (October 2005) –Achieved 20 MB/s and 20k rows/s sustained read-back rates –See CHEP2006 poster for detailsCHEP2006 poster Performance report since COOL 1.3.3 (Aug 2006) –Example: report for COOL 1.3.3 linked to COOL web pagereport for COOL 1.3.3 –Includes plots for several pre-defined use cases (e.g. single/multi- version insertion/retrieval from single/multi-channel folders) Most of the issues shown in the COOL 1.3.3 report are pending Some instead are fixed in CVS HEAD or in private test code

25. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 25 COOL - performance (2) Example: query time for single-version single-channel retrieval of all IOVs between two time points t1 and t1+  t –Query time increases with increasing values of t1 –Fix prototyped in private COOL code, but not yet in CVS –Effect previously observed (now fixed) to fetch one IOV at time t1 COOL 1.3.3Fixed (not yet in CVS HEAD) K. Dahl

26. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 26 COOL - my own code review (1) Cleanup needed in many areas –API and especially internal implementation –Many functionalities have been added over the past 18 months under stringent time constraints There was not much time to clean up on the way… –Cleanup needed before adding new features Especially if someone new should join COOL Proposed areas of improvement –Consistent transaction implementation –Simplify query implementation –Schema review (indices and FK constraints) –Minimize CORAL-specific classes and code

27. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 27 COOL - my own code review (2) Main concern for the future: scalability –Manageability of large databases –Performance with large databases Proposed development priorities –Progress with SQL performance optimization –Schema improvements and simplification Single table for several folders with the same schema –Partitioned and federated COOL database Introduce folder “partition” by time (and tag?) Database partitioning also by folder sets Attach/detach partitions/folders to existing database Data spread across several physical databases Technical ideas are there, not the manpower…

28. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 28 COOL - development plans Work in progress (summer-fall 2006) –Bulk insertion into different ‘channels’ at the same time –Channel metadata (improved data item selection) –ATLAS ‘dynamic’ replication (keep replicas in sync) –AMD64 port and storage precision in API (int32 vs. int64) –Functional and performance tests for Frontier Further developments (end 2006 and later) –Will depend on expt. priorities and available manpower –Code and schema cleanup –Oracle query optimizations –Merge tables for different data items but same schema –Large distributed databases (partitioning, external payload)

29. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 29 COOL - deployment Development is kept tightly linked to deployment –Close collaboration with DBA team within IT-PSS-DP Oracle account deployment model defined for Atlas –Separation of owner, writer, tagger and reader accounts –Tools to grant privileges added in COOL 1.2.1 (July 2005) Many deployment issues are addressed by CORAL –But query performance must be addressed within COOL

30. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 30 COOL – connection service Integration with CORAL connection service –Connection pool and retrial, db lookup… (see CORAL talk) –Moved to connection service in COOL 1.3.0 (April 2006) A few issues are still open – mostly waiting for CORAL –Database lookup on the Grid (choose closest physical database replica of given logical database) Could use different dblookup.xml (or local LFC catalogs) at different sites – should be addressed by each experiment –Secure authentication to the database server No Oracle proxy certificates, presently using user/password pair – waiting for LFC-based CORAL authentication –Role-dependent authentication - will be in CORAL 1.6.0 No integration yet with CORAL monitoring service –Not a high priority for the experiments, no manpower

31. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 31 COOL - data distribution Replication at the database backend level –Oracle Streams technology (tested and deployed within the 3D project) - see slides about Atlas and LHCb –Cross-technology replication is possible in principle (same schema for all backends), not really considered/tested yet Oracle remote access via Frontier –Intermediate Squid web caches –Work in progress (functional/performance tests with Squid, cache optimization for specific use cases…) – Atlas tests Replication using tools based on the COOL API –Data slicing/selection is also possible –Cross-technology replication (e.g. to SQLite files) –Work in progress (dynamic replication) - COOL 1.4.0

32. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 32 COOL - deployment in LHCb Computing model –Reconstruction at Tier0/1 –Only MC prod at Tier2 COOL stores only conditions data for event reconstruction –Oracle at PIT, Tier0, Tier1 with replication via Streams –Geometry and conditions for MC sent to Tier2 as SQLite file Online db master at PIT –Replicated forward to Tier0 and Tier1 via Streams –Data from PVSS processes Offline db master at Tier0 –Replicated back to PIT and forward to Tier1 via Streams –Data computed in offline calibration/alignment jobs (Marco Clemencic, COOL meeting 3 July 2006) COOL

33. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 33 COOL - deployment in Atlas Online OracleDB Offline master CondDB Tier-0 recon replication Tier-1 replica Tier-1 replica Online / PVSS / HLT farm Tier-0 farm Dedicated 10Gbit link ATLAS pitComputer centreOutside world CERN public network Calibration updates Streams replication ATLAS pit network (ATCN) gateway (Sasha Vaniachine and Richard Hawkings, 3D workshop 14 Sep 2006) Replication to Tier2’s –COOL ‘dynamic replication’, e.g. to MySQL – under development –Evaluating COOL Frontier backend (performance, cache consistency…)

34. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 34 COOL Summary A lot was achieved by COOL since the last review –Several production releases with several new features –In spite of relatively little development manpower New features and optimizations are still needed –Including many deployment-oriented developments The COOL project is so far a success… –Atlas and LHCb will use COOL for their conditions database …but manpower is now critically low again –Present FTE contribution: 0.8 from IT, 0.2 from LHCb –This is already slowing down the development schedule

35. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 35 Reserve slides

36. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 36 Conditions database choices ALICE –Alice-specific software for time/version handling –ROOT files (via AliROOT) CMS –CMS-specific software for time/version handling –Oracle with Frontier cache (via POOL-ORA API) ATLAS and LHCb –COOL common software for time/version handling –Oracle, MySQL, SQLite, Frontier (via COOL API)

37. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 37 COOL - software scope NOT the problems specific to one experiment or one data type (handled by each experiment) NOT the generic C++ access to relational data (handled by CORAL) Software for time-varying and versioned data: a common component with a well-defined task (RDBMS implementation of technology-neutral API ) NOT the generic deployment of relational services and distribution of relational data (handled by 3D: at CERN by IT-PSS-DP) (COOL) (CORAL) (3D)

38. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 38 COOL - relational implementation Modeling of condition data “objects” –System-managed common “metadata” Data items: many tables, each with many “channels” Interval of validity - IOV: since, until Versioning information with handling of interval overlaps –User-defined schema for “data payload” Support for simple C++ types as CORAL “AttributeList” Inline payload (in alternative: references to externally stored payload data)

39. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 39 Conditions data versioning Two types of conditions data containers (“folders”) –Single-version: ~online, measured data (e.g. slow-control) Main example: data uploaded from PVSS into COOL –Multi-version: ~offline, (re-)computed data (e.g. calibration) Versioning of IOVs for a data item (“within a folder”) –Payload cannot be updated: insert new version instead! –Assign “tags” to most recently inserted IOVs (HEAD) –Assign one “user tag” to IOVs as they are inserted –E.g. Calib_Ecal_prod in /calib/Ecal: A in [0,10], B in [10,20]… “Hierarchical” versioning of “folders” in “folder sets” –E.g. Calib_prod in /calib: Calib_Ecal_prod in /calib/Ecal… –Same mechanism used for Atlas detector description

40. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 40 COOL133 - Frontier bug fixes Frontier tests were failing before COOL133 due to many problems in CORAL, Frontier client and server – all fixed now Bug fixes in FrontierAccess in CORAL153 (thanks to Rado) –Bug #18275 in using column metadata – cause of COOL bug #18270 for MAX(IOV_SINCE) and bug #16995 for DISTINCT(CHANNEL_ID) queriesBug #18275bug #18270bug #16995 Bug fixes in frontier client 2.5.1 (thanks to D.Dykstra and Luis) –Bug #18359 in data compression on AMD64Bug #18359 Bug fixes and upgrades in frontier3d server (thanks to Luis) –Upgrade 3.0 server to 3.1 to enable compression Compression is used by the 2.4.5 client by default –Fix over 3.2 server for bug #18191 in providing column metadatabug #18191

41. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 41 COOL - pending Frontier bugs New bugs have been recently identified in COOL133: –COOL (CORAL?) bug #19753 with table aliases (Richard)bug #19753 High priority bug for Atlas (COOL multi-version data with user tags) –CORAL bug #19758: segfault in QueryDefinition::processbug #19758 –CORAL (SEAL?) bug #19762: abort in Session::isConnectedbug #19762 Lower priority bugs pending from previous releases: –COOL (CORAL?) bug #18146 for unsigned char data typesbug #18146

42. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t A. Valassi - 19 September 2006 LCG AA Review: COOL - 42 COOL - multithreading? SEAL issues recently discovered (August 2006) –Present implementation of seal Handle using intrusive boost pointers was not thread safe – it is being fixed –Also triggered the appearance of other bugs in the SEAL component model – most are understood and fixed –All these problems were triggered by the use of a second thread in the CORAL ConnectionService Issues also in OCI (CORAL OracleAccess) –OCI_THREADED must be disabled, otherwise OCI hangs –This may indicate thread safety problems in OCI too No actual test of multiple COOL threads yet –But issues in SEAL (and OCI?) should be fixed first