Issues in Milan Two main problems (details in the next slides): – Site excluded from analysis due to corrupted installation of some releases (mainly ) This was the real showstopper Several, time consuming attempt to cleanup and reinstall Reinstallation apparently successful, but the release was corrupted again after an hour or so – StoRM silently stopping to process requests The underlying GPFS file system halted in an apparent deadlock, but the storage areas were correctly mounted -> no alarm was triggered Unfortunate timing of the two, occurred contemporaneously during Summer holidays (reduced manpower) – Other, non directly related problems (air conditioning of computing room, server h/w failures) required attention, further reducing the available manpower

Release installation issue (solved) In Milan, the WNs are split in two rooms, each one belonging to a different subnet, with a single NFS server providing the s/w area to both the rooms through two different network adapters The different NFS network names confused the s/w installation system, generating a race condition between installation jobs on the different WN subsets Definitively understood and solved (by including all WNs in a common subnet) only after three weeks – It wasn’t a really difficult one, but efforts were focused on the other, storage related issue

GPFS issue GPFS randomly goes in a deadlock state – A GPFS thread starts waiting for an unknown condition to occur on a remote node – Waiter threads start to pile up on one of the Network Disk Servers (NDS), waiting for the first one to complete – The reason for the hung thread is still not known. Possible candidates: Failure of the underlying storage hardware Network issues GPFS bug … – No clear sign of any of this, though – Very similar problem observed at Tier1 They are still investigating too – Ticket opened with IBM support We were asked to gather some debugging data, but since then, the problem occurred only twice, during non working hours, and the system was automatically restarted No solution found yet, only a workaround to detect the deadlock and restart the services (GPFS and StoRM) – This eased the consequences of the problem, avoiding further exclusion from DDM