1. Cathode Strip Chamber (CSC) Raw Data Unpacking and Packing using bit field data classes Alex Tumanov, Rick Wilkinson September 2007

2. Alex Tumanov Rice University September 2007 2 CMS Muon Endcap n 468 Cathode Strip Chambers send data to Front End Drives (FEDs) n 8 FEDs with the capacity of 200 MByte/s each n A typical CSC event size is 10-100Kb n The L1 event rate is therefore 10-100KHz

3. Alex Tumanov Rice University September 2007 3 CMS HLT trigger basics n Every 25ns 20 pp collisions occur n 800 million pp collisions per second n L1 event rate of 100KHz n The CMS HLT trigger which is software based and its output is 100Hz

4. Alex Tumanov Rice University September 2007 4 CSC DAQ schematic n CSC Raw data unpacking is a part of the HLT: è Runs online è Must be very fast in order to handle 100KHz->100Hz n CSC unpacking is also used in offline Reconstruction and local “spy” data acquisition FED Global Event Builder 100KHz CFEB ALCT HLT Farm (runs unpacking) CFEB ALCT CSCs with on-chamber electronics CFEB ALCT CFEB ALCT CFEB ALCT 100Hz

5. Alex Tumanov Rice University September 2007 5 CSC Event Data Format DCC Data Event # 1 36 DDUs DDU Data... … 468 CSC chambers... … Chamber Data... … Contains data from different boards corresponding to each chamber (ALCT, DMB, CFEB, TMB) DCC Data 8 DCC slinks

6. Alex Tumanov Rice University September 2007 6 CSCCFEBData CSCEventData CSCDDUEventData Unpacking classes DCC bytestream CSCDCCEventData CSCDDUEventData up to10 DDUs CSCEventData up to15 CSCs CSCDMBHeaderCSCALCTHeaderCSCAnodeData CSCTMBData CSCDMBTrailer CSCCFEBData CSCTMBHeader CSCCLCTData CSCTMBScope CSCRPCData CSCTMBTrailer CSCALCTDigi CSCCLCTDigi CSCCorrelatedLCTDigi CSCComparatorDigi CSCWireDigi CSCRPCDigi CSCStripDigi CSCStatusDigi

7. Alex Tumanov Rice University September 2007 7 Taking a closer look at one class CSCCFEBData CSCEventData CSCDDUEventData DCC bytestream CSCDCCEventData CSCDDUEventData up to10 DDUs CSCEventData up to15 CSCs CSCDMBHeaderCSCALCTHeaderCSCAnodeData CSCTMBData CSCDMBTrailer CSCCFEBData CSCTMBHeader CSCCLCTData CSCTMBScope CSCRPCData CSCTMBTrailer CSCALCTDigi CSCCLCTDigi CSCCorrelatedLCTDigi CSCComparatorDigi CSCWireDigi CSCRPCDigi CSCStripDigi CSCStatusDigi CSCDMBHeader n The standard approach for extracting meaningful variables out of sea of binary data are bitshifts and masks n This is what was used in early versions of the CSC unpacking code: L1A Number DMB Id CSC Id … … … // the L1A number int L1A = (theData[i] >> 12) & 0xF; // CSC Id int CSCId = (theData[i] >>6) & 0x3F; //DMB Id int DMBId = (theData[i+2] >> 8) & 3F; … … …

8. Alex Tumanov Rice University September 2007 8 Binary data formats can be complicated: These are our simpler data objects We have 5 (and counting) different versions of the data formats There are 29 variables in this class

9. Alex Tumanov Rice University September 2007 9 Better way to unpack data n We use bit field classes: class CSCDMBHeader { public: … … … unsigned L1ANumber() const {return L1A_;} unsigned CSCId() const {return cscId_;} unsigned DMBId() const {return dmbId_;} … … … private: … …... unsigned L1A_ : 4; unsigned cfebId_ : 6; unsigned dmbId_ : 6; … … … }; The example of the data class CSCDMBHeader(unsigned short * theData) { memcpy(this, buf, sizeInWords()*2); } This is how it gets unpacked

10. Alex Tumanov Rice University September 2007 10 001011 DMB Id Old way of unpacking binary data L1A Number = CSCId = DMBId = Binary Data Fill variables one by one 1011 L1A 101110 CSC Id

11. Alex Tumanov Rice University September 2007 11 001011 DMB Id Bit Field based unpacking L1A Number = CSCId = DMBId = Binary Data Fill variables at once!!! 1011 L1A 101110 CSC Id

12. Alex Tumanov Rice University September 2007 12 Why bit field unpacking is faster n CSCDMBHeader class contains 29 variables that are filled from binary data n Using bitshifts & masks that would result in 29 lines of code like this: int L1A = (theData[i] >> 12) & 0xF; n Using bit field based classes all 29 variables are filled during just one line: CSCDMBHeader(unsigned short * theData) {memcpy(this, theData, sizeInWords()*2); } n Because of this we are able to achieve an order of magnitude of improvement in the average unpacking time (currently around 3ms/event) n Total number of bit field variables in our package – 355

13. Alex Tumanov Rice University September 2007 13 Understanding memcpy() n Memcpy implementation in assember loads two addresses and the size into registers and does the copying in a single hardware instruction n Assignment operator in L1A = (theData[i] >> 12) & 0xF; is at least one copy operation // C++ code #include const int MAXSIZE = 1000; int main() { int *a, *b, i; a = new int[MAXSIZE]; b = new int[MAXSIZE]; memcpy(b, a, sizeof(int) * MAXSIZE); return 0; } // Assembly code // the address of a is in eax // and the address of b in edx mov ecx, 1000 mov esi, eax mov edi, edx rep movsd

14. Alex Tumanov Rice University September 2007 14 Memcpy() timing benchmarks #include const int MAXSIZE = 1000; int main() { int *a, *b, i; a = new int[MAXSIZE]; b = new int[MAXSIZE]; for (long j = 0; j < MAXSIZE; ++j) { *a++ = *b++; } return 0; } #include const int MAXSIZE = 1000; int main() { int *a, *b, i; a = new int[MAXSIZE]; b = new int[MAXSIZE]; memcpy(b, a, sizeof(int) * MAXSIZE); return 0; } CPU Time, seconds per 1mln repetitions MAXSIZE X1000

15. Alex Tumanov Rice University September 2007 15 A word about raw data packing n Data stored in persistent objects (digis) can be converted into raw binary bite stream (Digi->Raw packing) n Because each class stores data as bit fields all that is needed to create raw bit stream is this pointer and size() method that returns the size of the object: n We use boost::dynamic_bitset<> class to handle bit streams boost::dynamic_bitset<> rawDMBbits = UshortToBitset(theDMBHeader.size(), theDMBHeader.data()); unsigned short * data() {return (unsigned short *) this;}

16. Alex Tumanov Rice University September 2007 16 Conclusions n An order of magnitude in data unpacking speed can be gained by using bit fields instead of conventional bit shifts and masks n The bit field data classes are usually easier to read and write – a huge advantage to help avoid probably the biggest source of unpacking problems – errors/typos/bugs in bit definitions n Particular attention should be paid to questions related to memory and 32bit vs 64bit CPU architecture and endian-ness issues: è Recently CSC unpacking code was successfully moved from 32bit Scientific Linux 3 (SLC3) machines to 64bit SLC4 è Extensive testing by Valgrind shows no memory problems

17. Alex Tumanov Rice University September 2007 17 Appendix

18. Alex Tumanov Rice University September 2007 18 Pouring wine (data) slowly Inefficient way: pour one at a time

19. Alex Tumanov Rice University September 2007 19 Pouring wine (data) fast! Efficient way: prearrange glasses and pour into all at once!