EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI AMOD report – Fernando H. Barreiro Megino CERN-IT-ES-VOS 10/22/2015 AMOD report –

EGI-InSPIRE RI Summary Technical Stop for LHC and ATLAS No shifters during the whole week Stable beam expected for tomorrow Overall very quite week. Few highlights caused by Interventions scheduled during this period Hurricane Irene HLT reprocessing problems AMOD report –

EGI-InSPIRE RI DDM summary All activities Sources Destinations

EGI-InSPIRE RI Panda summary AMOD report –

EGI-InSPIRE RI Monday CERN LFC migration Sunday set CERN offline in Panda Monday 8:30 CERN excluded from DDM Intervention as planned: ~9:20-11:30 Update of LFC information in information systems and services 12:30 re-included in DDM ~17:00 online in Panda CERN CASTORATLAS DB patch and defragmentation CERN ATONR rolling intervention BNL emergency downtime Sat. evening – Monday evening Frontier fallback mechanisms failed – US cloud offline in Panda ~19:00 back in DDM ~22:30 testjobs and cloud online in Panda Decision to postpone dCache upgrade until next technical stop ASGC in downtime for CASTOR upgrade to ASGC affected by broken 10Gbps link between Chicago and Amsterdam 10/22/2015 AMOD report –

EGI-InSPIRE RI Tuesday Wednesday CERN myproxy.cern.ch machine migration transparent for ATLAS, not so much for the other experiments CERN ATLR, ATCR & ATLDSC rolling intervention ATLDSC migration problematic Gap of transactions created after restarting Replication of conditions data to T1 stopped Wed. between 12:30-16:30 CERN: "Invalid SRM version [] for endpoint []" error (GGUS 73918) affecting T2-CERN transfers Disappeared with the CERN FTS T2 OS upgrade ASGC network maintenance AMOD report –

EGI-InSPIRE RI Thursday Friday Victor stopped with reboot of machine on Critical service now Consequences: Victor should restart with machine reboot Monitoring proposal: Implement SLS monitoring in ADC Central Services category If it shows unavailable shifters should check Victor’s webpage and/or notify AMOD AMOD report –

EGI-InSPIRE RI Saturday Sunday Errors with T1s involved, mostly because of high load. In particular MCTAPE To TAIWAN-LCG2_MCTAPE: destination file failed on the SRM with error [SRM_ABORTED] GGUS: Felix Lee: “We found a lot of jobs are queuing in Castor transfer manager, which might cause srm aborting new transfer, we have increased slot to see if things can be improved.” To RAL-LCG2_MCTAPE: destination file failed on the SRM with error [SRM_ABORTED] GGUS: Alastair Dewhurst: “MCTAPE is a quite underpowered space token at RAL. It appears the problem is just high usage and it can't cope. The immediate solution is to reduce the number of FTS transfers allowed into it and then during the working week add some more hardware so it can cope with this kind of load.” SARA-MATRIX: Get error: dccp failed with output GGUS: Onno Zweers: “pool node crashed with a kernel panic” PIC_DATADISK: [GENERAL_FAILURE] AsyncWait GGUS: Fernando Lopez: “This problem was caused due a lot of queued transfers in our PoolManager. Transfers has been forced manually and now all seems ok” BNL-OSG2_DATADISK: First non-zero marker not received within 300 seconds GGUS: Jane: “As checked, quite some pools were in high load and shown as not seen, which should be the cause of the failure. Now, the system is in good shape and I don't see problems. As tested, the transfer of the file was also fine. The problem should be gone. AMOD report –

EGI-InSPIRE RI Sunday Problem on voatlas66: DDM Central Deletion stopped 11:30-21:30 RAL-LCG2: network problems during the whole night. All services unavailable during this period