LHCb Software week November, 1999 M.Frank LHCb/CERN N-Tuples within Gaudi ã Definition ã Usage ä Run through simplified example ã The persistent back-end

M.Frank LHCb/CERNGaudi N-Tuples PersistencySvc PObj Converter TObj Obj2 TObj TObjContainer ObjContainer EventDataSvc Transient Event Store TObj Obj1 N-tuple Service EventSelectorMessageSvc JobOptionsSvcAnotherPercySvc PObj What are we talking about? AppManager TObj1 Obj1PObject PDetElem DetDataSrv TDetElem1 T Detector Store T Histogram Store HistogramSvc Hist1 Algorithm1 DetPerstySvc Converter Alg Properties AlgFactory uses creates navigability HistPerstySvcPHist Converter PHist PNTup Converter ConversionSvc T N-Tuple Store NTupleSvc NTup NTup1

M.Frank LHCb/CERNGaudi N-Tuples What was an N-Tuple in CERNLIB ã Collection of identifiable n-dim. primitive items ä LOGICAL*4, INTEGER*4, REAL*4, REAL*8 ã See HBOOK ä Row wise N-Tuples ä REAL*4 array(SOME_DIMENSION) ä Items are single numbers identified by tags ä Column wise N-Tuples ä Structure mapped to a common block ä Items (number, array, …) are identified by tags ä Allow for variable size arrays

M.Frank LHCb/CERNGaudi N-Tuples What is an N-Tuple in Gaudi ã Collection of N-tuple Items ä Items represent primitives: bool, long, float, double ã N-tuples reside in a data store ä Can be accessed like any other DataObject ã Data management (e.g. writing records) is done by the N-tuple service Things did not change dramatically

M.Frank LHCb/CERNGaudi N-Tuples What’s the job: Wishes ã Stick to architecture ä We want to independent from the “back-end” storage mechanism (HBOOK, ROOT, OBJY...) Cernlib will die ä Encapsulate implementation: Interfaces or ABC ã Easy to use Items must behave like ordinary numbers Must support basic arithmetic operations and type conversions ã N-tuples should not go all into one single file possibly, but not always Connect several streams to the data store

M.Frank LHCb/CERNGaudi N-Tuples The N-Tuple Data Store ã Top Entry point /NTUPLES ä Logical file names User defined First useable directory /MC /REC /USER ä Directory structure ä N-tuples 1,2,3,4

M.Frank LHCb/CERNGaudi N-Tuples The Example ã Simplified version of GaudiExample/Ntuples ã Explain how to write and read a column wise N-tuple ã Differences to row wise N-tuples are minor ä Only record structure differs ã N-tuples should be easy to use ä Tell me if you think stuff is overcomplicated ä I do not use them daily

M.Frank LHCb/CERNGaudi N-Tuples Easy to Use… (Hopefully) ã Define data items e.g. as members of filling Algorithm ä Items are based on primitives bool, long, float, double // Items for n-tuple // _________________ // // 0 dimensional item (=number) NTuple::Item m_ntrk; // 1 dimensional items (Arrays) NTuple::Array m_px, m_py, m_pz; // 2 dimensional items (Matrices) NTuple::Matrix m_hits;

M.Frank LHCb/CERNGaudi N-Tuples NTuplePtr nt(ntupleService(),"/NTUPLES/MC/1"); if ( !nt ) { nt = ntupleService()->book("/NTUPLES/MC/1”, CLID_ColumnWiseTuple, "Hello World"); } NTuplePtr nt(ntupleService(),"/NTUPLES/MC/1"); if ( !nt ) { nt = ntupleService()->book("/NTUPLES/MC/1”, CLID_ColumnWiseTuple, "Hello World"); if ( nt ) { status = nt->addItem ("Ntrack", m_ntrk, 0, 5000 ); status = nt->addItem ("px", m_ntrk, m_px); // m_hits[0:m_ntrk][0:5]; numbers within [0,8] status = nt->addItem ("hit", m_ntrk, m_hits, 5, 0, 8 ); } Easy to Use… (Hopefully) ã Book and add the items you later want to fill Book the N-tuple Add items

M.Frank LHCb/CERNGaudi N-Tuples for(m_ntrk=0; m_ntrk < numTracks; m_ntrk++) { if ( m_ntrk >= m_ntrk->range().distance() ) break; } for(m_ntrk=0; m_ntrk < numTracks; m_ntrk++) { if ( m_ntrk >= m_ntrk->range().distance() ) break; m_px[m_ntrk] = track[m_ntrk]->px(); m_hits[m_ntrk][0] = track[m_ntrk]->adcValue(0);... m_hits[m_ntrk][4] = track[m_ntrk]->adcValue(4); } for(m_ntrk=0; m_ntrk < numTracks; m_ntrk++) { if ( m_ntrk >= m_ntrk->range().distance() ) break; m_px[m_ntrk] = track[m_ntrk]->px(); m_hits[m_ntrk][0] = track[m_ntrk]->adcValue(0);... m_hits[m_ntrk][4] = track[m_ntrk]->adcValue(4); } // Commit record (here of a column wise N-tuple) status = ntupleService()->writeRecord("/NTUPLES/MC/1"); Easy to Use… (Hopefully) ã Fill all items and commit record ä Simple calculations can be done directly using the items Fill items Commit record Do not overrun dimensions Reset values to defaults: Is this good?

M.Frank LHCb/CERNGaudi N-Tuples Easy to Use… (Hopefully) ã At the end of the job ä All N-tuples will automatically be closed ã The directory structure of the data store is reflected by RZ directories ä Access N-tuple for reading in the same way as for writing ä Don’t forget to “cd // / ” in PAW

M.Frank LHCb/CERNGaudi N-Tuples NTuple::Item ntrk; NTuple::Array px, py, pz; NTupleDirPtr nt(ntupleService(), "/NTUPLES/MC/1"); if ( nt ) { } NTuple::Item ntrk; NTuple::Array px, py, pz; NTupleDirPtr nt(ntupleService(), "/NTUPLES/MC/1"); if ( nt ) { status = nt1->item ("Ntrack", ntrk ); status = nt1->item ("px", px);... } NTuple::Item ntrk; NTuple::Array px, py, pz; NTupleDirPtr nt(ntupleService(), "/NTUPLES/MC/1"); if ( nt ) { status = nt1->item ("Ntrack", ntrk ); status = nt1->item ("px", px);... do { float sumPx = 0; if(ntupleService()->readRecord(nt.ptr()).isSuccess()){ for ( long j = 0; j < ntrk; j++ ) sumPx += px[j]; } } while ( status.isSuccess() ); } Easy to Use… (Hopefully) ã Reading back (…or with PAW) Will open the corresponding logical file Search for the directory Load the N-tuple Assign the entries to the N tuple Loop over records

M.Frank LHCb/CERNGaudi N-Tuples Easy to Use… (Hopefully) ã Configuration (using job options file) ä Reading N-tuples NTupleSvc.Input = { ”MC#tuple1.hbook", ”USER#tuple2.hbook" }; // Persistency type of the N-tuple service: 6=HBOOK NTupleSvc.Type = 6; NTupleSvc.Output = { ”MC#tuple1.hbook", ”USER#tuple2.hbook" }; // Persistency type of the N-tuple service: 6=HBOOK NTupleSvc.Type = 6; ä Writing N-tuples

M.Frank LHCb/CERNGaudi N-Tuples Back-End: HBOOK Limitations ã Limited type information ä Cannot define all “C” data types ä Therefor only: bool, long, unsigned long, float and double ä NOT: char, short, int (+unsigned) ä Does not affect disk space: HBOOK compresses internally ã N-tuple name is an “integer” ä “integer” translates later to HBOOK ID ä Must be constant independent of booking time for kumacs ã Do not mix up N-tuple IDs and Histogram Ids ä There is only one PAWC common block!

M.Frank LHCb/CERNGaudi N-Tuples Back-End: HBOOK Limitations ã Only one index variable per item ã RZ directory names are CHARACTER*8 ä … and upper case ONLY ã double/REAL*8 data ä PAW has trouble: better don’t use ä REAL*8 must be quadword aligned (64 bits) ä Padding must be done by user ä Not more than 1000 REAL*8 entries per N tuple ã Row wise N-tuples ä Type information is stored as first “event” ä Start reading at the second...

M.Frank LHCb/CERNGaudi N-Tuples Other Back-Ends ã The Present: HBOOK ä PAW is the most widely used viewer ä N-tuples don’t help if they can’t be analyzed interactively ã The Future (?): ROOT I/O ä Will give ROOT a try as back-end ä Implementation with trees/branches is straight forward ä Would solve probably all the limitations of HBOOK

M.Frank LHCb/CERNGaudi N-Tuples Conclusions ã It is possible to define dynamic structures (N-tuples) with Gaudi ã If you think the usage is too complicated, please say so! ä Otherwise it will never change ã Depending on the back-end, limitations invade transient side

M.Frank LHCb/CERNGaudi N-Tuples The Back-End: HBOOK (At least for the time beeing) ã Items must be contigous in memory ä Array of reals ä Common block ä Must be exported in detail where ever a variable has to be filled PARAMETER (MAXTRK = 100, MAXHIT = 300) COMMON /CMTRK/ NTRACK, PX(MAXTRK), PY(MAXTRK), + PZ(MAXTRK), HITS(MAXHIT,MAXTRK) CALL HBNAME(ID,'VARBLOK2',NTRACK, + 'NTRACK[0,100], '// + 'PX(NTRACK), PY(NTRACK), PZ(NTRACK), HITS(300,NTRACK)’)

M.Frank LHCb/CERNGaudi N-Tuples Expected Functionality ? ã Like a library ä Store objects for later use by clients ä Retrieve objects when needed ã Traverse the content (using “Agents”) ä Retrieve collections Make only selected objects persistent ã Ownership ä Responsibility for cleanup

M.Frank LHCb/CERNGaudi N-Tuples Structure of the data store ã Tree - similar to file system MCEvent DataObject ã Store item= directory + attributes ã Browse capability ã Identification by logical addresses: ”/Event/Mc/MCEcalHits”