1. Sviluppi in ambito WLCG Highlights
Claudio Grandi (INFN-Bologna)
Mini-workshop CCR - Legnaro
17 Gennaio 2011

2. Mini-workshop CCR - Legnaro
Outline
Information System
Workload management
Multi-User Pilot Jobs
Multi-core CPUs
Virtualization
CERNVM-FS
Storage
Amsterdam Jamboree - Demonstrators
Mini-workshop CCR - Legnaro
17 Gennaio 2011

3. Mini-workshop CCR - Legnaro
Information System
Quoting F.Donno at the February GDB:
“The [Glue] attributes used by middleware, experiments, monitoring and accounting tools are mostly discovery or semi-static status information”
Strategy:
Improve the global reliability of the service
Deployment of a well managed set of top level BDIIs
Provide a more static view of the available resources
Parallel deployment of “static”/semi-dynamic top level BDIIs
Improve the coherence and correctness of the information used
Cleaning-up/consolidating IS used attributes: experiments, middleware, monitoring and accounting tools
Not clear what the impact will be on the gLite-WMS match-making process
Mini-workshop CCR - Legnaro
17 Gennaio 2011

4. Mini-workshop CCR - Legnaro
Multi User Pilot Jobs
Late binding of payload to resources
Intra-VO scheduling is under VO control
In use by 3.5/4 LHC experiments
Security concerns: sites loose control on access
Sites require identity change within the pilot  glexec
VOs can delegate part of the control to sites
Running glexec on the WN requires an authorization server
ARGUS is ready for production but not in use
Next version of CREAM can use ARGUS
Consistent authorization
Mini-workshop CCR - Legnaro
17 Gennaio 2011

5. Exploitation of multi-core CPUs
In order to more efficiently use multi-core CPU's, the LHC experiments have been developing multi-core-aware applications capable of exploiting more than a single core via multi-processing and/or multi-threading
The request from the experiments is to be able to have access to the whole node rather than a single core
This has impact on the whole job management chain
Experiment software and job management frameworks
Grid middleware
Site configuration and accounting procedures
A WLCG task force has been created
First step is to run test at sites on dedicated queues that provide one job slot per node
Mini-workshop CCR - Legnaro
17 Gennaio 2011

6. Grid access to whole nodes
EGEE-III MPI WG recommendations
Final version available at
Next steps
Validate new attributes in the CREAM-CE:
Using a temporary JDL attribute (CERequirements)
E.g. CERequirements = “WholeNodes = \”True\””;
Using direct job submission to CREAM
Validation starting with some user communities
Modify CREAM and BLAH so that the new attributes can be used as first-level JDL attributes:
E.g. WholeNodes = True;
Coming with CREAM 1.7 (EMI 1 release ~ April 2011)
Support for the new attributes in WMS
Automatic
I.Bird, Multicore and Virtualization WS
CERN – June Updates
Mini-workshop CCR - Legnaro
17 Gennaio 2011

7. Mini-workshop CCR - Legnaro
Virtualization
Virtualization may ease:
the support of multiple VOs at a site
the inclusion of a site in the VO infrastructure
HEPiX WG approach:
Policy for creation of trusted virtual images
VO’s prepare images according to policies accepted by sites
Contextualization
Sites have hooks to customize the VO image to the site configuration
Cataloguing and distribution
CVMFS may be used to distribute the images
WNoDeS approach: focus on the site
Sites create images in response to VO requests and keep full responsibility of their maintenance
Mini-workshop CCR - Legnaro
17 Gennaio 2011

8. Mini-workshop CCR - Legnaro
CERNVM-FS
Make a directory tree stored on a web server look like a local read-only file system
Developed to deliver software distributions onto virtual machines is currently being tested to distribute experiment software to sites
Better performance than NFS/AFS
Ease software distribution process
Mini-workshop CCR - Legnaro
17 Gennaio 2011

9. A change in perspective?
MUPJ + “whole node” + virtualization = Cloud?
Experiments just want a Worker Node service
Suited to their needs (e.g. running job “attractors”)
The WNs join the experiment infrastructure
Experiments take the responsibility of managing their specific services and handling the users job flow
Sites and infrastructure services just handle the resource allocation, including inter-VO accounting
This looks very much like a Cloud, but with a few important differences:
Nodes are allocated for a definite amount of time so that if the experiment doesn’t use them they can be used by others
Nodes are allocated without root privileges so that they can benefit from being inside the site LAN
Experiments still need to execute user jobs with glexec so that the site still has control on the authorization process
WNoDeS!!!
Mini-workshop CCR - Legnaro
17 Gennaio 2011

10. Storage Model Evolution
Motivations:
Non optimal use (waste) of disk resources for analysis
Too many replicas for data rarely accessed
More efficient use of network resources
Network is cheaper and more available than it used to be
More controlled access to MSS
Strategies:
Remote data access
xrootd, NFS 4.1, WebDAV, AFS, ...
Dynamic data placement
pre-placement
caching
Global namespaces, evolution of catalogues (p2p, etc...)
MSS-Disk split
SRM interface may become redundant
Mini-workshop CCR - Legnaro
17 Gennaio 2011

11. Amsterdam DemonstratorsX
WLCG Workshop
London – Jul ‘10
WLCG GDB
CERN – Jan ‘11
Mini-workshop CCR - Legnaro
17 Gennaio 2011

12. Mini-workshop CCR - Legnaro
What’s happening...
Focus on the remote data access
xrootd, NFS 4.1, CHIRP, caching (ARC and more)
Most popular solution is xrootd
ALICE!
Big investment by CERN-IT together with ATLAS
Concrete activities in CMS
Interesting activities in dynamic data placement
Mainly by ATLAS, LHCb did not report recently, CMS starts looking to it
In my opinion the drawbacks of remote data access are underestimated
In particular we lack a proper authorization mechanism that prevents abuses
We should invest more on dynamic data placement!
Mini-workshop CCR - Legnaro
17 Gennaio 2011

13. Mini-workshop CCR - Legnaro
Conclusions
A few important components are being deployed on the WLCG infrastructure and need extensive testing
CREAM – ARGUS – glexec on WNs
WLCG attitude towards service support
consolidation and simplification of low level services
not necessarily through standard tools, e.g. xrootd
reduction of support to high level services that are more and more considered to be a business internal to the experiments
in particular the job management and any component functional to its operation
May be that my personal perspective changed but I have the impression that the contributions by external (grid middleware and infrastructure) projects have less and less influence on WLCG strategies
Mini-workshop CCR - Legnaro
17 Gennaio 2011