Kevin Thaddeus Flood University of Wisconsin Tier-2 Workshop Kevin Thaddeus Flood University of Wisconsin 19 April 2018

Tier-2 Workshop Tier-2 workshop on Thursday 16:00-18:00 (40-2-A01) http://indico.cern.ch/conferenceDisplay.py?confId=37243 Users/Admins roundtable discussion Identify/address problems Share solutions Standardize “best practices” Several EWK users/admins shared their experiences Frank Wuerthwein Dmytro Kovalskyi Ezio Torassa Pablo Garcia Kalanand Mishra Thanks! 19 April 2018

EWK user experience is generally very positive Tier-2 User Feedback EWK user experience is generally very positive Able to successfully run/manage large numbers of jobs, merge output files, efficiently generate physics submit jobs through both CRAB and local batch manager publish output datasets to DBS T2s have good login pools, large RAM, fast (and multi-core) machines shorten compile/link times PhEDex dataset transfer tools easy to use Fast network links between user sites and T2s 19 April 2018

Tier-2 User Feedback Some issues still to be addressed Batch priorities sometimes seem out of balance, what is optimal assignment of batch slots when competing tasks collide Widely varying job failure rates across sites Job failure rate scales with number of input events, perhaps due to memory leaks with more failures where less available memory resource – more of a problem with earlier releases before 2_1_X Difficult to debug failures in writing output files for remote jobs back to user facility (FNAL user problem, so likely T2 issue) No efficient/easy method for user file transfer offsite from T2 login pool disk “copy [file] into my /store/user at UCSD, then lcg-cp it into castor, and back out onto my desktop [at CERN]” Grid tools for managing /store/user are difficult, much easier when possible to manage area interactively from login pool No standardized Linux installation at T2s favorite/useful tools missing in some cases out-of-date/insecure tools in some cases (Firefox 1.5!!) 19 April 2018

Tier-2 Admin Issues Need standard procedures to link PAG/POG users to T2 admins for routine and special communication Something more than announcement/complaint lists Publishing datasets works well for storage management, but unpublished datasets provide unregulated storage “backdoor” “30% of the used disk space in Legnaro is written with this "back door". Is difficult to follow what people [are] doing, it will be a hard work to clean DBS and files when this usage will become too large.” No good replication/backup options for dCache There are two but don’t scale well to large numbers Connecting problematic batch jobs (e.g., large wall time, small CPU cycles) to particular users can be difficult Could use more info on how to use VOMRS/siteDB 19 April 2018

Tier-2 PAG/POG Convenor Feedback Lots of spare CPU cycles available for private production but data management tools difficult for users/admins Analysis vs production queues need balance in favor of physics users rather than production in Wisconsin, production sleeps when there are pending analysis jobs Need management tools for PAG/POG convenors available/consumed storage reports generated periodically and at thresholds (80/90/95/99% full) formal accounting for the officially allocated storage space (exact specification of what counts against storage cap) 19 April 2018