1. Debugging of #100019 P. Hristov 04/03/2013

2. Introduction Difficult problem – The behavior is “random” and depends on the “history” – The debugger doesn’t show what actually happens – The standard tools (i.e. valgrind) do not detect it – All the tricks I tried did not help Important problem – Many jobs crash with the same message

3. Binary search for problem identification Needs fast multiprocessor machine for compilation (thanks to DAQ!) One has to repeat several (N) times the same test to be sure that given revision works or crashes (I used N=5) In this way I identified rev. 59755: the fix for “label==0” problem You remember that this also caused #99670 Increased virtual memory after the "Label 0 fix"

4. Static initialization order fiasco See the details in – http://www.parashift.com/c++-faq-lite/static-init-order.html http://www.parashift.com/c++-faq-lite/static-init-order.html – http://www.parashift.com/c++-faq-lite/static-init-order-on-intrinsics.html http://www.parashift.com/c++-faq-lite/static-init-order-on-intrinsics.html Short description: in the implementation file #include “MyClass.h” … const int fkLookForTrouble = AnotherClass::GetValue(); //Methods … void MyClass::DoMess() { // Use fkLookForTrouble … } We had similar problem in 2007 with pointers initialized from a factory method: easier since the crash is “less random” The fix is to use AnotherClass::GetValue() directly

5. Test after the fix Run 195566 – one of the worst runs – DONE 2,686 from 4,648 (57.8%) – ERROR_V 1227 (26.5%) – bug – EXPIRED 724 (15.6%) – memory After the fix – DONE 4,089 from 4,648 (88.0%) – ERROR_V 415 (8.9%) – bug – EXPIRED 144 (3.1%) – memory

6. To Do Check all similar places in AliRoot and provide a fix Fix some memory leaks found by Insure++ (an evaluation license was provided by Parasoft) Run again the test on run 195566

7. Old slides

8. Introduction More than 15% of the jobs crash in one of the two streamers: AliTRDtrackV1::Streamer or AliTRDcluster::Streamer The problem is reproducible only on SLC5 – The same Root/AliRoot with the same raw files work on Ubuntu or MacOS – If the code is compiled without optimization (-O0) it works also on SLC5 – If you start directly from the event that crashed, the job is OK => the crash depends on the “history” – Sometimes the reconstruction doesn’t crash, and the probability that it is OK again in the next run is higher Hypothesis – memory corruption – problem in IO

9. Localization of problem Replace AliEn OCDB with local one Replace the AliEn raw file with local one => xrootd is not involved in the crash Reduce the list of detectors: – “minimal configuration” to reproduce the crash: ITS, TPC, TRD, PHOS, EMCAL, HLT Debug printout (gDebug): large and useless

10. “Simple” debugging with gdb Find the exact place of crash Investigate the content – Corrupted structure in CINT – Try watchpoint on the address with wrong content: this doesn’t work because the corrupted address changes Compile without optimization only the affected class – The problem is reproducible, but almost no additional information came out

11. Debugging with test function The test function examines the content of the global list, where the corruption occurs Possibility to “bracket” the place of corruption (closer to the actual place, less reproducible) Localized to the reading of PHOS raw data Possibility to set watchpoint (worked once/twice out of many attempts) Full calling chain: involves TBufferFile, TStreamerInfoReadBuffer, TStreamerInfoActions, TBranchElement, TBranch, TBranchRef, TRefTable, TRef, AliRawEquipmentV2

12. Changes in Root/AliRoot Inspection of all modifications in the affected classes – Nothing suspicious Test with old Root tag: works, but probably by chance PHOS raw data format/consistency: tested by the PHOS experts, no changes since 2011 RAW data framework: no changes since 2011

13. Runs with Valgrind Memcheck – it detects the use of corrupted CINT structures, but not the moment they are corrupted – no errors when we only read RAW SGcheck – one invalid write in string operations – no errors when we only read RAW Latest version of Valgrind (3.8.1): no new problems detected

14. Other tools Free – electric fence: put 2 words for each allocated word, too “memory hungry” => cannot be easily used – duma: clone of electric fence – libcwd: not tried Commercial – Insure++: problem with the license server, no reply from Sdt.Support@cern.ch Sdt.Support@cern.ch Parasoft contacted for evaluation license – Purify: the same problem + no experience – TotalView: no version is available Coverity: check carefully the remaining defects

15. Status on 21/02/13 Hypothesis: “second order” corruption – Corruption in the IO caused by unknown code (in allocated memory since Valgrind doesn’t detect it) – Corruption in the CINT structures caused by the problem in IO Difficult to debug

16. Additional simplification rec.SetRunVertexFinderTracks(kFALSE); rec.SetRunMultFinder(kFALSE); rec.SetRunCascadeFinder(kFALSE); rec.SetFillTriggerESD(kFALSE); rec.SetWriteAlignmentData(kFALSE); rec.SetRunLocalReconstruction("ITS TPC TRD PHOS HLT"); rec.SetRunTracking("ITS TPC TRD"); rec.SetFillESD("HLT"); When HLT is out of FillESD, everything works!

17. Investigation of FillESD Set “return” at different places to localize the code that causes crash Since somehow the success of the attempt is correlated with the previous attempt, always rerun the “crashing” version and then go to the changed one Calling chain – AliHLTReconstructor::FillESD – AliHLTSystem::ProcessHLTOUT – AliHLTOUTHandler::ProcessData -> AliHLTTriggerAgent::ProcessData – AliHLTOUT::GetDataObject – AliHLTMessage::Extract – AliHLTMessage::ReadObject (calls TBufferFile::ReadObject) The object that causes problems is AliHLTGlobalTriggerDecision Everything works with default object (no IO)

18. AliHLTGlobalTriggerDecision This class has not changed since long time (607 days) The object is written using Root v5-33-02b. Contains – AliHLTDomainEntry (594 days after the last change) – AliHLTComponentDataType (in AliHLTDataTypes.h, 374 d.) – AliHLTTriggerDomain (807 days) – AliHLTTriggerDecision (861 days) – AliHLTLogging (209 days) – AliHLTCTPData (857 days) – AliHLTReadoutList (502 days) – AliHLTEventDDLV1 (in AliHLTDataTypes.h, 374 days)

19. Plans Try with realistic AliHLTGlobalTriggerDecision object without IO Investigate all changes between Root v5-33- 02b and v5-34-02 Check with different raw files