1. Status Report of the RD45 Project z Based on LCB Review, November 1999

2. Overview zInitial Goals of the Project zSummary of April 1998 LCB Review zWork on Milestones zEvolution of O(R)DBMS market zRisk Analysis: Summary and Conclusions zFuture Activities zSummary

3. RD45: Introduction zProposed in 1994; approved early 1995 zGoal: provide persistency for LHC data Objects: event, calibrations, histograms, etc. zAssumed: O-O environment, C++ (initially) Now also Java, including interoperability issues zEmphasis: standards (OMG, ODMG,...), potential use of commercial solutions zAnticipated 3 project phases: Requirements gathering, limited prototyping Detailed prototyping and evaluation Implementation

4. Guiding Principles  “In particular, the data should be presented in as consistent a way as possible. The data themselves may be stored in a variety of formats but this should be hidden from the user…” z"The ODMG... binding is based on one fundamental principle: the programmer should perceive the binding as a single language for expressing both database and programming operations, not two separate languages with arbitrary boundaries between them.“ zCapability of scaling to LHC data volumes & rates zCapable of satisfying wide variety of HEP needs yDAQ, SIM, REC, Analysis,... zUse of “standard”, widely-used solutions if applicable CMS

5. Phase 1 - Milestones & Conclusions  Requirements specification  Evaluation of ODMG’s ODL for HEP data models  Prototype based upon commercial ODBMS and above data model zRapid & successful focus on standards & commercial ODBMSs zIdentification of federated databases as a valid solution for high volume event data zVery promising tests of event data retrieval from a commercial ODBMS Supplement to 1st milestone: zProduce a “Statement of Probable Capabilities” for a HEP persistent object manager based on commercial ODBMSs and large-market mass storage systems

6. Phase 2 - Milestones  Impact of using an ODBMS  Object model, physical data organisation, use of CASE tools, 3rd party class libraries, C++ application code  Evaluation of ODBMS features & suitability for HEP  Schema Evolution, Object Versioning, Data Replication  Performance comparisons with existing solutions  PAW + Ntuples  Use of ODBMS for typical simulation, reconstruction and analysis scenarios with data volumes of up to 1TB.  Impact of ODBMS on end-user physicist. (Including private schema & collections for simulation, reconstruction and analysis.)  Demonstrate the feasibility of using an ODBMS and MSS at data rates sufficient for ATLAS and CMS 1997 test-beam requirements. 19961996 19971997

7. LCB Review, April 1998 z“The project has achieved the initial R&D goal of investigating and identifying potential solutions to the problem of persistent data storage for LHC experiments.” z“The proposed solution: ODBMS (Objectivity/DB) is now adopted for data persistency not only by all the LHC experiments but by many others (BaBar, NA45, COMPASS, RHIC) ready to take data in 1-2 years.” No longer valid (except ALICE)

8. Milestones (April 98) ¶Provide, together with the IT/PDP group, production data management services based on Objectivity/DB and HPSS with sufficient capacity to solve the requirements of ATLAS and CMS test beam and simulation needs, COMPASS and NA45 tests for their '99 data taking runs. ·Develop and provide appropriate database administration tools, (meta-)data browsers and data import/export facilities, as required for  ¸Develop and provide production versions of the HepODBMS class libraries, including reference and end-user guides. ¹Continue R&D, based on input and use cases from the LHC collaborations to produce results in time for the next versions of the collaborations' Computing Technical Proposals (end 1999).

9. ¶Production Servers setup; used for CDR and other activities. Milestones: ATLAS: 1TB, CMS: 100MB/s. COMPASS, CHORUS, NA45, others,... ·Federated DB Backup tool developed (based on multiple FDs); numerous DB browsers (CERN DRO_Tool, SLAC BDB, Micram Hudson, …), DB Import/Export based on SLAC model ¸New release of HepODBMS including scalable event collections + import of BaBar “conditions DB”. Revised user doc (XML) + ref. manual (DOC++), CSC tutorials + examples ¹R&D activities: yDatabase usage over a wide area network yClustering and re-clustering strategies yMulti-user, multi-federation issues yDatabase integration with MSS Milestones - Results MONARC Examples follow...

11. Multi-user, Multi-FD Issues zMultiple FDs used mainly to workaround limitations in Objectivity/DB, e.g. ylock contention on global resources (catalogue) xe.g. online / offline systems (BaBar, CMS, …) ylack of private schema/catalogue xe.g. user schema / data ylack of security xsee Objy V5.2 ylack of support of partial backups xdescribed above One of the main issues for with Objy meeting in late Feb

12. Multi-FD Example: CMS DB1DB2DB3DBnDB1DB2DB3DBn RunDB LogDB BConfDB Prod ProdFD Prod Boot Offline - cmsc01 Clone FD Online Prod Boot Prod ProdFD LogDB BConfDB RunDB

13. Multiple FDs & User Data zProduction FD cloned by users zUsers can add private data / schema zCan share data zScalability? Approach also used by other large Objy users, e.g. COMPASS Space telescope

14. Federation Backup Procedure zIs production FD in a consistent state? zCopy all relevant DB files zInstall on backup FD zCheck consistency zCopy to tape Partial backup procedure high on wish-list of Objy customers (ETF)

15. Database Production Service - What is missing? zTransparent non-blocking interface with MSS zUser capability to: yexport, extract, replicate data and schema ymanipulate data and schema outside production database and while accessing data and schema from production database zFully functional, reliable high-quality database system including yVLDB support (>>1PB) ymanagement tools From L. Silvestris: Review of application software services for the LHC era, FOCUS 07/10/99 Objy V5.2 Objy V6? BaBarBaBar

16. O(R)DBMS Evolution From CMS Computing Technical Proposal: z“If the ODBMS industry flourishes it is very likely that by 2005 CMS will be able to obtain products, embodying thousands of man-years of work, that are well matched to its worldwide data management and access needs. The cost of such products to CMS will be equivalent to at most a few man-years. We believe that the ODBMS industry and the corresponding market are likely to flourish. However, if this is not the case, a decision will have to be made in approximately the year 2000 to devote some tens of man- years of effort to the development of a less satisfactory data management system for the LHC experiments.”

17. z RDBMS + “object extensions” yCan store ADTs y“Methods” on server z Complex Data with Queries z $8B in 1996 z Likely to become dominant DBMS technology zComplex Data zPerformance, scalability zTight Language Binding yOQL - SQL3 query subset zGrowth similar to RDBMS in ’80s z~$1B market by 2001 ODBMS / RDBMS / ORDBMS ~$100M?

18. Risk Analysis: Issues zChoice of Technology yODBMS, ORDBMS, RDBMS, light-weight POM, files + meta-data etc. zChoice of Vendor y#1 Objectivity, #2 Versant zThe Home-Grown approach yEstimate resources required  Implies proof-of-concept prototype Versant

19. Risk Analysis: Summary of Options zEvaluate C++ binding to e.g. ORACLE zAdd ESCROW clause to Objectivity contract zPursue possibility of source license zVisit key Objectivity customers zProduce new requirements list zEstimate manpower to support Objy in house zEstimate manpower for “clean-sheet” solution zContinue to monitor alternatives The LCB agrees with the other suggested steps to mitigate risk, with the addition of trying to insure that user code in reconstruction and analysis programs is kept as standards compliant as possible.

20. Risk Analysis: Conclusions zA solution is certainly possible!  How much should we align ourselves with industry trends / standards? zODBMS unlikely to dominate DBMS market yLikely to survive foreseeable future - market! zNeed to complete current prototype to make meaningful manpower estimates yTarget: end-1999; present at this workshop!

21. Future Activities Production Services zConsidered essential by several experiments zTools, documentation, regular releases, … ygeneral production level support zPush for VLDB and other enhancements “2001” milestone z Revise requirements z Visit other HEP labs (BNL, FNAL, SLAC, …) z Provide ODBMS- independent s/w layer z Estimate man-power for alternative POM z Evaluate ORDBMS technology Feb. meeting at Objy

22. Summary (+) zWe have a good understanding of ODBMS technology & Objectivity/DB in particular zSystem has been demonstrated to work in production up to level of today’s (BaBar) experiments zMany enhancements have been delivered, others in pipeline zProduction experience will be invaluable for LHC (product enhancements, tools, etc.)

23. Summary (-) zThe ODBMS market has not taken off as was previously predicted zWe need to assure ourselves that there is sufficient non-HEP demand (and $$$) zWe need to (in any case) understand how an eventual migration could be handled zWe need to develop at least one realistic fallback scenario

24. Conclusions zR&D phase of RD45 has now led to production ODBMS services zRisks of current strategy well understood - risk management must continue zWe are well placed to prepare for “2001 milestone” zFuture focus: yProduction yRoad-map to 2001 and beyond