1. CHEP 2003 March 22-28, 2003 POOL Data Storage, Cache and Conversion Mechanism Motivation Data access Generic model Experience & Conclusions D.Düllmann, M. Frank, G. Govi, I. Papadoupolos, S. Roiser

2. M.Frank CERN/LHCb 2CHEP 2003 March 22-28, 2003 Motivation  Physics software should be independent of the underlying data storage technology  Data of different nature has to be accessed  Event data, detector data, statistical data, …  The data sizes: O(10 6 ) to O(10 13 ) Bytes/experiment/year  The access patterns differ  It is unclear how these data will be stored  Locking into one technology may be a disadvantage Need for a technology free data storage and data access mechanism

3. M.Frank CERN/LHCb 3CHEP 2003 March 22-28, 2003 Strategy  Hide any technology details from the clients  Clients deal with objects or object references  Hide all cache/persistency specific details  No compromise on transient data representation due to technology details  Each technology can be handled transparently  Transient representation sufficient for persistency  Ensure independence of experiment framework  Run-time binding of transient data to the underlying technology  Need for object description: “dictionary”

4. M.Frank CERN/LHCb 4CHEP 2003 March 22-28, 2003 Object Cache Data Service Manages object cache Client Data Access ClientClient ClientClient ClientClient Ref<T>Ref<T> Object Cache Data Service Ref<T>Ref<T> Object Cache Data Service Ref<T>Ref<T> Client access data through References Different context  Event data  Detector data  other Different context  Event data  Detector data  other

5. M.Frank CERN/LHCb 5CHEP 2003 March 22-28, 2003 Cache Access Through References  References know about the Data Cache  2 operation modes: - Clear at checkpoint - Auto-clear with reference count Object Reference in Cache Manager Reference to Cache Manager Ref<T>Ref<T> Pointer to object Dereference  References are implemented as smart pointers  Use cache manager for “load-on-demand”  Use the object key of the cache manager

6. M.Frank CERN/LHCb 6CHEP 2003 March 22-28, 2003 Data Service object cache … TokenObject Cache Access by Smart Pointer Persistency Service Object type Storage type Persistent Reference T o k e n Cache Ref Data Service Pointer Ref File Catalog

7. M.Frank CERN/LHCb 7CHEP 2003 March 22-28, 2003 Generic Persistent Model Objects & pointers Objects, object IDs, collections & DBs Transient Persistent C++ pointer >> object ID

8. M.Frank CERN/LHCb 8CHEP 2003 March 22-28, 2003 Access to the Data (5) Register -Object -References (2) Look-up Data Cache Will be unsuccessful, requested object is not present (3) Load request Persistency Service Technology dispatcher Data Service ClientClient (1) read(…) Ref<T>Ref<T> Try to access an object data Storage Service Storage Service Common Handling Conversion Service Conversion Service

9. M.Frank CERN/LHCb 9CHEP 2003 March 22-28, 2003 Map objects and write Storage Service Storage Service Storing objects Ref. mark for write Start Transaction Commit Transaction Persistency Service Technology dispatcher Object Cache ClientClient Data Service Common handling Conversion Service Conversion Service cache.startTransaction(...) Ref.mark_write(placement)... Ref.mark_write(placement) cache.endTransaction(...,COMMIT)

10. M.Frank CERN/LHCb 10CHEP 2003 March 22-28, 2003 The Storage Mechanism  The underlying model assumptions  How they map to “known” technologies  Migrating objects to/from the persistent medium  Object mapping  Reference handling  References are objects, not primitives  Need setup: Reference to data cache  ROOT: Callback for base class (Streamer)

11. M.Frank CERN/LHCb 11CHEP 2003 March 22-28, 2003 The Generic Model StorageSvc database Database Disk Storage Container Objects Data Cache Object type (class name) Optional data transform Storage type DB name Cont.name Item ID

12. M.Frank CERN/LHCb 12CHEP 2003 March 22-28, 2003 Database Technologies  Identify commonalties and differences between technologies Necessary knowledge when reading/writing  Model adapts to any technology with direct record access  Need to know record identifier in advance  RDBMS: More or less traditional  Primary key must be uniquely determined before writing  Probably two round-trips

13. M.Frank CERN/LHCb 13CHEP 2003 March 22-28, 2003 Object Mapping  Objects must maintain personality when persistent  Allow for queries, selections and independent element access  If technology supports objects…  Want to make use of such features  These technologies must be instructed how to do it  Need object dictionary  If technologies support only primitives  Split objects into primitives [until reasonable level]  Need full access to object member data[member offset, type]  Constructor and Destructor with defined signature  Need object dictionary

14. M.Frank CERN/LHCb 14CHEP 2003 March 22-28, 2003 Technology dependent.h ROOTCINT CINT dictionary code Dictionary Generation CINT dictionary I/O Data I/O Dictionary: Population/Conversion GCC-XML LCG dictionary code.xml Code Generator LCG dictionary Gateway Reflection Other Clients

15. M.Frank CERN/LHCb 15CHEP 2003 March 22-28, 2003 Follow Object Associations Link ID Link Info DB/Cont.name,...... Local lookup table in each file (1) TokenObject (2)(3) (4) Entry ID Link ID

16. M.Frank CERN/LHCb 16CHEP 2003 March 22-28, 2003 The Link Table  Contains all information to resurrect an object  Storage type  Database name  Container name  Object type (class name)  Cache hints  E.g. other possible transient conversions  Size: O(Associations in class model)  Local to every database  Size is limited

17. M.Frank CERN/LHCb 17CHEP 2003 March 22-28, 2003 Experience & Conclusions  We adopted a mechanism to write physics data without knowledge of the underlying store technology  Our approach can adopt any technology based on database files, collections and objects within collections  ROOT (implemented) and RDBMS (work ongoing)  We are able to choose technologies according to needs  As can save any objects described by the dictionary  We can offer a uniform interface to persistency clients http://http://lcgapp.cern.ch/project/persist