Data Management Overview David M. Malon Argonne U.S. ATLAS Physics and Computing Project Advisory Panel Meeting Berkeley, CA November 2002

14 November 2002 David M. Malon, ANL PCAP Review 2 Outline  Technology transition  Architecture and design  Support for data challenges  LHC-wide common projects  Database support for detector description  Other collaborative efforts—conditions databases  Some challenges on the horizon  Summary

14 November 2002 David M. Malon, ANL PCAP Review 3 Technology transition  Objectivity/DB has been the ATLAS baseline, and the persistence technology for Data Challenge 0, but will be phased out this year  Currently retained as a reference implementation  ATLAS Software Release (January 2003) and developer releases leading up to it will not depend upon Objectivity/DB  Migration for data in Objectivity/DB databases based upon earlier releases planned for 2003  Technology strategy is to adopt LHC-wide common persistence infrastructure (hybrid relational and ROOT-based streaming layer) as soon as this is feasible  A U.S.-developed ROOT-based conversion service provides the persistence technology for at least Phase I of Data Challenge I  This, too, will be phased out when common project software is sufficiently capable

14 November 2002 David M. Malon, ANL PCAP Review 4 Technology transition (2)  ATLAS architectural separation of transient and persistent representations has meant that the transition has been relatively painless for physicists and physics software developers  Not so painless for the database group, partly because of the need to support multiple technologies simultaneously with limited personpower  But AthenaROOT conversion services provide valuable prototyping for LHC common project work  Short-term problem in any case  Complicated by need to support data access inside and outside Athena  Remember that Geant3 simulations are still in FORTRAN  When ADL and related common project dictionary infrastructure (see David Quarrie’s talk) are more mature, such transitions should be substantially easier

14 November 2002 David M. Malon, ANL PCAP Review 5 Architecture and design  U.S.-led effort produced an event store architecture document last fall  Since last review, a U.S.-led effort produced a hybrid (relational/streaming) event store design document, using the architecture document as a starting point  Represents the most detailed thinking among any of the LHC experiments about how to build a hybrid store  Circulated to other LHC software architects, and the principal subject of an April database workshop in Orsay  CERN IT/DB and ROOT team experts attended as well  Not all of the ideas will survive an LHC-wide common project, but many will, and they provide a non-trivial starting point for LHC-wide discussions in any case

14 November 2002 David M. Malon, ANL PCAP Review 6 Support for data challenges  Data Challenge 0 was finally completed (Summer ’02), when Data Challenge 1 was well underway(!)  It’s a good thing, though, that ATLAS did not simply declare success without serious continuity tests—these should be the true legacy of DC0  Database group was therefore supporting two different persistence technologies (Objectivity/DB and AthenaROOT) for these data challenges  Also supporting event generation for both data challenges, to different extents  Seizing the opportunity to introduce grid project technologies into ATLAS data challenges  Magda from PPDG  Virtual data ideas from GriPhyN in event generation and simulation recipes—even in advance of the release of GriPhyN VDL toolkit

14 November 2002 David M. Malon, ANL PCAP Review 7 Support for data challenges (2)  U.S. database group has been trying to avoid getting its developers dragged into still more data challenge responsibilities—not so easy  Not being involved in DC production management, though, makes it harder to push for prototyping and adoption of tools developed by U.S. grid projects (e.g., virtual data infrastructure from GriPhyN)  CERN-based database effort (Goossens, Smirnov), though, has been largely lost to the data challenges  ATLAS Data Challenge Coordinator (Poulard) is also the CERN group leader  Update: Smirnov is leaving CERN to join the Chicago iVDGL team  Note that there has been a significant increase in DC involvement by U.S. ATLAS grid testbed participants

14 November 2002 David M. Malon, ANL PCAP Review 8 LHC-wide common projects  First RTAG (Requirements Technical Assessment Group) commissioned by SC2 was to try to find sufficient common ground for an LHC-wide project to deliver a shared persistence infrastructure  RTAG membership: Brun (ROOT), Duellmann (IT/DB), Innocente (CMS), Malon (ATLAS; convenor), Mato (LHCb), Rademakers (ALICE)  Succeeded in producing a document and achieving consensus sufficient to launch a common project  Final report delivered 5 April 2002  Proposes ROOT-based streaming layer plus a relational database layer  Persistence project launch workshop held 5-6 June 2002 at CERN  Quarrie and Malon also represent ATLAS in the LCG Architects Forum (and there are other RTAGs)

14 November 2002 David M. Malon, ANL PCAP Review 9 LHC common persistence infrastructure (POOL)  Workshop produced agreement to attempt to meet a rather aggressive schedule—a September 2002 release with non-trivial functionality, and a Spring 2003 release sufficient to support serious data challenges  ATLAS database group is fully committed to contributing to this effort and to adopting this technology  To be clear: the common project infrastructure that POOL will provide IS our baseline event store technology  All event-store-related ATLAS database development is planned to be a contribution to or an extension of POOL, or an integration of POOL into the ATLAS/Athena environment  U.S. is contributing approximately 2 FTEs directly to common project development; this should increase as Objectivity responsibilities wane  Orsay has expressed an intention to contribute ~1 FTE

14 November 2002 David M. Malon, ANL PCAP Review 10 U.S. contributions to POOL  We are attempting to avoid mission creep in the common project as well, by participation in selected clearly defined work packages  Common project event collections and collection management (ANL)  Contributions to persistence for non-ROOT objects (BNL)  Craig Tull (LBNL) is contributing to dictionary effort (strongly related to ADL work)  We have also volunteered for management of external MySQL-related packages (ensures coherence of ATLAS and LHC-wide versions)  Both ANL and BNL (Malon, Adams) will continue to contribute to overall common project architecture and design  We have established “local” liaisons with Fermilab-based CMS contributors to the common project (Joshi, Tanenbaum)

14 November 2002 David M. Malon, ANL PCAP Review 11 Integration of POOL into ATLAS  First really testable release of POOL is due in December  U.S. ATLAS plans to deliver an Athena conversion service based on POOL in Release (January ’03) if POOL release is on schedule  Likely to be a prototype, for use by database developers  First “production” release of POOL is due in Spring ’03  First ATLAS “production” conversion service based upon POOL will follow (Release 7.0.0, tentatively)

14 November 2002 David M. Malon, ANL PCAP Review 12 Database support for detector description  Database group was asked in April by detector description working group to provide access to “primary numbers”—numbers that parameterize ATLAS geometry—for Athena applications  U.S. database group in August delivered access to “primary numbers”—numbers that parameterize ATLAS geometry description— via a conversion service that respects the Gaudi/Athena architecture  Numbers are resident in a MySQL database  Approach strongly leverages NOVA work, funded at BNL as an LDRD project

14 November 2002 David M. Malon, ANL PCAP Review 13 Other collaborative efforts—conditions databases  Strategy has been to use IT-provided conditions database if possible, rather than writing such a service ourselves  IT implementations, though, are in Objectivity/DB and Oracle9i  Lisbon ATLAS group has delivered a MySQL implementation for TDAQ community; we have enlisted their help, and encouraged them to contribute this to LCG repository (done!)  Concurrently, TDAQ implementation has been integrated into ATLAS offline software releases  U.S. database group has refrained from work in this area in an effort to avoid overcommitment, but real work is needed soon to connect conditions infrastructure to Athena  Planned for Release 6.0.0

14 November 2002 David M. Malon, ANL PCAP Review 14 Some challenges on the horizon  Development of an architecture based upon POOL  POOL project is currently delivering building blocks upon which an architecture might be based  Will be helped by Blueprint RTAG work  U.S. ATLAS is in a position to take a leadership role in articulation of a common project persistence architecture, thanks to prior work on ATLAS database architecture and hybrid event store design  Development of a coherent approach to relational database services  Currently many, many MySQL applications in ATLAS; ad hoc approach to access and services  Should be addressed on an LHC-wide level  ATLAS database group is currently formulating strawman requirements, near-term deployment plans, and proposed longer-term strategies as starting points for LHC-wide discussions (presentation next week)

14 November 2002 David M. Malon, ANL PCAP Review 15 Summary  We are in the midst of a major technology transition while supporting data challenges, with no reserve personpower  No replacement for Ed Frank (Chicago), and U.S. database group is funded thus far well below agency guidance  Have lost an additional 0.5 FTE at Argonne; expect restoration in ‘03  Committed to ensuring the success of the LCG persistence project, and to using the resulting infrastructure as the principal ATLAS persistence technology  Relying upon joint projects and leveraging other projects wherever possible (LCG (POOL), CERN IT and ATLAS/TDAQ (conditions), PPDG (Magda), GriPhyN (virtual data), LDRD (NOVA for primary numbers), HENP Grand Challenge (POOL event collections and iterators), …)