1. More Interfaces Workplan
Dirk Düllmann, IT-DB
LCG Persistency Workshop, June 5th 2002

2. Conventions for next slides
All interface
assume usual set of basic fixed length types pool::u32, pool::u64, ….
but drop namespace prefix for readability
On following slides, I will use
open(string &name, …) instead of
open(const std::string &name, …)
proper use of const, &, virtual is left to the reader
Prefix interface by suggested namespace

3. IFileCatalog Assumptions:
“logical file name” will be replaced by a unique file identifier for the scope of the persistency framework
This logical file IDs needs to be agreed on at least within LCG if not within a larger scope!
I’ll make a proposal after consulting AF people
lfid resolveRef(Ref aRef)
extract lfid from ref (technology dependent)
Proposal: I would move this functionality into the Ref/Token responsibility: eg each Storage Manager token must provide a method to obtain the unique logical file identifier it’s referring to.
string pfn findReplica(lfid)
perform lfid to pfn lookup
assuming EDG RLS will handle lfid rather than lfn
lfid createFile(string pfn)
allocate unique file ID for a new physical file
eg using GUID or from pre-allocated local range or..
removeFile(lfid)

4. poolImpl::StorageMgr
provides pool::ITransactionHandler
ISmToken write(ISmToken pl, byte *buf, u32 len);
writes an opaque byte sequence into store
byte *read(ISmToken obj, u32 &len)
reads opaque byte sequence from store
ISmToken open(string &psxName, pool::openMode m);
opens or creates a file according to given mode
return token is usable as placement hint for write
close(ISmToken t);
closes a file which has been opened with above method

5. poolImpl::ISmToken Not sure how abstract we can be… class ISmToken {
// provides access to file id, eg for catalog lookup/catalog level re-mapping
pool::fileUID getFileUID();
// provides access to object id (unique in a file), may be used for object level re-mapping
pool::objID getObjectID();
// extenalise/internalise
void internalise(const string &exForm);
string externalise();
// equality
bool operator==(ISmToken rhs);
// order / hash
int operator>(ISmToken rhs); }
Not sure how abstract we can be…

6. Collections
See David Malon’s proposal

7. Dictionary Interface see Pere’s, Stefan’s and Victor’s slides
Propose to accept Gaudi interface as starting point
start a quick comparison with
cint (C++) and Java reflection interfaces
possibly extending it
Need agreement about
who provides the master dictionary
I tend to agree with Pere’s arguments
model and prototype for to feeding slaves
by September

8. pool::Ref<T> Proposal: ODMG like interface
Ref<T> and RefAny from Espresso
CacheMgr interface see next slide
ptr(); // return pointer to open object in cache
close(); // invalidate real memory pointer

9. pool::ICacheMgr and pool::Ref<T>
addRef(Ref r) // bi-directional link between ref/cm
removeRef(Ref r)
openRef(Ref r) // state change to data in memory // ref real memory pointer set
closeRef(Ref r) // state change to data on disk // ref contains only PmToken
On cache manager cleanup it calls
closeRef for_each subscribed ref

10. Proposed Work Packages
Refs and Object Lookup
Definition of IStorageManager and ISMToken interface
Implementation of a RootI/O storage manager
File Catalog and Grid Integration
MySQL, XML and EDG based implementations of IFileCatalog
FileUID generation, local catalog management tools

11. Proposed Work Packages
Collections and Iterators
Explicit and implicit collection implementations
for RootI/O and RDBMS
Conversion and Dictionary
Transient and persistent IDictionary implementation
Cross population between cint/RootI/O dictionary and
Dictionary import/export

12. Proposed Work Packages
Meta Data and Query
IAttributeList implementation for RootI/O and RDBMS
Query result iterator
Integration/Common services/Packaging?
RDBMS interface
PersistencyManager
Project specific development infrastructure

13. Main question: How to stage required component decoupling?
My Proposal :
focus first on working Navigation (You all asked for it!)
First Stage: IO and streaming still coupled
Use RootI/O as storage manager which delivers directly C++ objects instead of byte streams
Still using normal Root streaming
Decoupled Dictionaries and StreamerSrv components not required yet
Can already exercise Refs and navigation with real objects
Second Stage: IO and streaming components decoupled
(after September release)
RootI/O StorageManager now delivers opaque bytes
IDictionary and IStreamerSrv moved in place between PersistencyManager and StorageManager

14. Workplan Proposal
Sequence of functional releases which provide an increasing set of interfaces to the user
Start integration with most critical functionality
Proposed priority order
Navigation
Collections
Meta Data
Dictionary Interfaces
“September” release
(decoupled) Streaming …
Does not mean that component WP could/should not start working on implementation, but not yet integration

15. Workplan Basic Navigation – July IStorageManager, ISmToken
provided for RootI/O
Ref<T>
Basic navigation works (physical OID, not into embedded collections)
IFileCatalog
basic lookup by fileUID – no property based retrieval yet
Two back ends (LCG MySQL + LCG XML)
ITransactionHandler - in place for all components
working at least for MySQL FileCatalog
prototype RootI/O checkpointing -> decision about availability in September release
Assume persistency for objects not deriving from TObject as presented by Rene is sufficient
Agreed but not yet integrated
Interface for persistent and transient dictionaries

16. Work Plan II Collections - August Collection<T> Ref
basic - no property based retrieval
Explicit & implicit collection implementation for eg RDBMS
Iteration interface & population interface
Ref
Extension: navigation into embedded collections
New IFileCatalog implementation integrated
EDG/Globus
Proof of concept:
Checkpointing of root files
all other components as before

17. Work Plan III Meta Data & Query – September
Meta Interface eg IAttributeList in place
For file and collection catalogs
Same population and query interface for both
Registry for descriptions and support for attribute based retrieval (query)
IFileProperties + ICollectionProperties
Root I/O Checkpointing works
all other components as before

18. Work Plan IV Encapsulated Dictionary – December
IDictionary + Meta Class system - Support for dictionary reading via the agreed interface
Proof of concept implementation for population of the dictionary from C++ (cint) and at least one external source through new interface
all other components as before

19. Towards the mid’03 release
Production LCGRef -> single collection element
Production Schema import / export
Full population of LCG dictionary from external (non-C++) sources as ATLAS ADL/ LHCb XML
Sufficient to use ROOT I/O default converters
Proof of concept – Meta Data import/export
How to extract all required meta data for a given event collection/run
Production checkpoint recovery implementation

20. Resources / Schedule Very aggressive schedule for “September release”
Still large uncertainties about required and available effort
how many contributors can be at CERN at least for some time
Need to get started to see
Expectation
Release will aim to be usable for first production tests…
… but it’s for sure it won’t pass those tests yet!
It will tell us what and how much needs still to be fixed.
No way, that a 3 month old implementation can be production quality by then!