Sviluppi in ambito WLCG Highlights Claudio Grandi (INFN-Bologna) Mini-workshop CCR - Legnaro 17 Gennaio 2011

Mini-workshop CCR - Legnaro Outline Information System Workload management Multi-User Pilot Jobs Multi-core CPUs Virtualization CERNVM-FS Storage Amsterdam Jamboree - Demonstrators Mini-workshop CCR - Legnaro 17 Gennaio 2011

Mini-workshop CCR - Legnaro Information System Quoting F.Donno at the February GDB: “The [Glue] attributes used by middleware, experiments, monitoring and accounting tools are mostly discovery or semi-static status information” Strategy: Improve the global reliability of the service Deployment of a well managed set of top level BDIIs Provide a more static view of the available resources Parallel deployment of “static”/semi-dynamic top level BDIIs Improve the coherence and correctness of the information used Cleaning-up/consolidating IS used attributes: experiments, middleware, monitoring and accounting tools Not clear what the impact will be on the gLite-WMS match-making process Mini-workshop CCR - Legnaro 17 Gennaio 2011

Mini-workshop CCR - Legnaro Multi User Pilot Jobs Late binding of payload to resources Intra-VO scheduling is under VO control In use by 3.5/4 LHC experiments Security concerns: sites loose control on access Sites require identity change within the pilot  glexec VOs can delegate part of the control to sites Running glexec on the WN requires an authorization server ARGUS is ready for production but not in use Next version of CREAM can use ARGUS Consistent authorization Mini-workshop CCR - Legnaro 17 Gennaio 2011

Exploitation of multi-core CPUs In order to more efficiently use multi-core CPU's, the LHC experiments have been developing multi-core-aware applications capable of exploiting more than a single core via multi-processing and/or multi-threading The request from the experiments is to be able to have access to the whole node rather than a single core This has impact on the whole job management chain Experiment software and job management frameworks Grid middleware Site configuration and accounting procedures A WLCG task force has been created First step is to run test at sites on dedicated queues that provide one job slot per node Mini-workshop CCR - Legnaro 17 Gennaio 2011

Grid access to whole nodes EGEE-III MPI WG recommendations Final version available at http://grid.ie/mpi/wiki/WorkingGroup Next steps Validate new attributes in the CREAM-CE: Using a temporary JDL attribute (CERequirements) E.g. CERequirements = “WholeNodes = \”True\””; Using direct job submission to CREAM Validation starting with some user communities Modify CREAM and BLAH so that the new attributes can be used as first-level JDL attributes: E.g. WholeNodes = True; Coming with CREAM 1.7 (EMI 1 release ~ April 2011) Support for the new attributes in WMS Automatic I.Bird, Multicore and Virtualization WS CERN – June 2010 + Updates Mini-workshop CCR - Legnaro 17 Gennaio 2011

Mini-workshop CCR - Legnaro Virtualization Virtualization may ease: the support of multiple VOs at a site the inclusion of a site in the VO infrastructure HEPiX WG approach: Policy for creation of trusted virtual images VO’s prepare images according to policies accepted by sites Contextualization Sites have hooks to customize the VO image to the site configuration Cataloguing and distribution CVMFS may be used to distribute the images WNoDeS approach: focus on the site Sites create images in response to VO requests and keep full responsibility of their maintenance Mini-workshop CCR - Legnaro 17 Gennaio 2011

Mini-workshop CCR - Legnaro CERNVM-FS Make a directory tree stored on a web server look like a local read-only file system Developed to deliver software distributions onto virtual machines is currently being tested to distribute experiment software to sites Better performance than NFS/AFS Ease software distribution process Mini-workshop CCR - Legnaro 17 Gennaio 2011

A change in perspective? MUPJ + “whole node” + virtualization = Cloud? Experiments just want a Worker Node service Suited to their needs (e.g. running job “attractors”) The WNs join the experiment infrastructure Experiments take the responsibility of managing their specific services and handling the users job flow Sites and infrastructure services just handle the resource allocation, including inter-VO accounting This looks very much like a Cloud, but with a few important differences: Nodes are allocated for a definite amount of time so that if the experiment doesn’t use them they can be used by others Nodes are allocated without root privileges so that they can benefit from being inside the site LAN Experiments still need to execute user jobs with glexec so that the site still has control on the authorization process WNoDeS!!! Mini-workshop CCR - Legnaro 17 Gennaio 2011

Storage Model Evolution Motivations: Non optimal use (waste) of disk resources for analysis Too many replicas for data rarely accessed More efficient use of network resources Network is cheaper and more available than it used to be More controlled access to MSS Strategies: Remote data access xrootd, NFS 4.1, WebDAV, AFS, ... Dynamic data placement pre-placement caching Global namespaces, evolution of catalogues (p2p, etc...) MSS-Disk split SRM interface may become redundant Mini-workshop CCR - Legnaro 17 Gennaio 2011

Amsterdam Demonstrators X WLCG Workshop London – Jul ‘10 WLCG GDB CERN – Jan ‘11 Mini-workshop CCR - Legnaro 17 Gennaio 2011

Mini-workshop CCR - Legnaro What’s happening... Focus on the remote data access xrootd, NFS 4.1, CHIRP, caching (ARC and more) Most popular solution is xrootd ALICE! Big investment by CERN-IT together with ATLAS Concrete activities in CMS Interesting activities in dynamic data placement Mainly by ATLAS, LHCb did not report recently, CMS starts looking to it In my opinion the drawbacks of remote data access are underestimated In particular we lack a proper authorization mechanism that prevents abuses We should invest more on dynamic data placement! Mini-workshop CCR - Legnaro 17 Gennaio 2011

Mini-workshop CCR - Legnaro Conclusions A few important components are being deployed on the WLCG infrastructure and need extensive testing CREAM – ARGUS – glexec on WNs WLCG attitude towards service support consolidation and simplification of low level services not necessarily through standard tools, e.g. xrootd reduction of support to high level services that are more and more considered to be a business internal to the experiments in particular the job management and any component functional to its operation May be that my personal perspective changed but I have the impression that the contributions by external (grid middleware and infrastructure) projects have less and less influence on WLCG strategies Mini-workshop CCR - Legnaro 17 Gennaio 2011