1. Alessandro De Salvo Alessandro.DeSalvo@roma1.infn.it IFAE 2011 Perugia, 28-04-2011 New data processing technologies at LHC: from Grid to Cloud Computing and beyond Alessandro De Salvo Alessandro.DeSalvo@roma1.infn.it IFAE 2011 Perugia, 28-04-2011 A. De Salvo – Apr 28 2011

2. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 The LHC Computing Challenge Signal/Noise: 10 -13 (10 -9 offline) Data volume High rate * large number of channels * 4 experiments 15 PetaBytes of new data each year Compute power Event complexity * Nb. events * thousands users 200k of (today's) fastest CPUs 45 PB of disk storage Worldwide analysis & funding Computing funding locally in major regions & countries CERN can provide up to 20-30% of the resources 70-80% are provided by WLCG partners Efficient analysis everywhere GRID technology

3. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 The LHC computing model Centralised or individualised? Centralised data reduction or full scale analysis in “analysis trains”. Easy to deal with from an execution point of view Takes time to organise Potential waiting time for physicists Individual physicists submit jobs in “chaotic” manner Many, often inexperienced, users attempt large scale data processing Physicists in charge of when and how to perform analysis Risk of lower efficiency All the LHC experiments use a distributed analysis model Data available for analysis will be located at remote sites across the globe Grid tools are used for performing analysis Only a single “sign-on” with Grid certificate No remote logins

4. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 How does the Grid work? Middleware makes multiple computer and data centers look like a single system to the user Security Information system Data management Job management Monitoring Accounting ot easy!

5. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 World LHC Computing Grid (WLCG) > 140 computing centers 35 countries Hierarchical and regional organization 12 large centers for primary data management CERN = Tier-0 11 Tier-1 centers 10 countries Fast network links 38 federations of smaller Tier-2 centers Tier-0 CERN Tier-1 centers Tier-2 sites Taiwan ASGC USA BNL France CCIN2P3 Germany FZK USA FNAL Italy CNAF Nordic countries NDGF Spain PIC UK RAL NL SARA- NIKHEF Canada TRIUMF Cloud

6. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Sergio Bertolucci, CERN6 Fibre cut during test STEP’09: Redundancy meant no interruption Fibre cut during test STEP’09: Redundancy meant no interruption LHC Network

7. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Condition data access Condition data are generally kept either in databases (ORACLE) or as plain (ROOT) files, kept at CERN The data is replicated in the Tier-1 sites and accessed by the analysis jobs Direct access to ORACLE from the jobs is a killer application for the DBs in the Tier-1s Too many connections to the ORACLE servers High load Risk of inefficiency To cope with this limitation the preferred access to the condition data is via an intermediate caching proxy FroNTier servers Extensions of a standard squid proxy server, may also act as standard squid Caching and failover

8. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011  FroNTier deployed to enable distributed access to the conditions DB  Working toward making it more transparent to the end users  Same model used by CMS Condition data access example: ATLAS

9. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Database technologies: noSQL databases [1] Current RDBMS have limitations Hard to scale De-normalization for better performance Complicated setup/configuration with apache Bad for large, random and I/Os intensive applications Exploring the noSQL solutions to overcome the current limitations ‘Scale easily’ – But different architecture ‘Get the work done’ on lower cost infrastructure Same application level reliability Google, Facebook, Amazon, Yahoo! Open source projects Bigtable, Cassandra, Simpledb, Dynamo, MongoDB, Couchdb, Hypertable, Riak, Hadoop Hbase, etc. noSQL databases are currently not a replacement for any standard RDBMS There are classes of complex queries which include time and date ranges where RDBMS typically performs rather poorly For example, storing lots of historical data on expensive transaction-oriented RDBMS does not seem optimal An option to unload significant amounts of archival-type reference data from Oracle to a highly performing, scalable system based on commodity hardware appears attractive

10. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Database technologies: noSQL databases [2] Being evaluated or in use for ATLAS Panda system ATLAS Distributed Data Management WMcore/WMAgent … The ATLAS experience with the DQ 2 Accounting service (large tables) shows that a noSQL database like MongoDB can reach almost the same performance of an Oracle RAC cluster using almost the same amount of disk space, but running on cheap hardware Oracle 0.3s to complete the test queries Oracle RAC (CERN ADCR) 38 GB of disk space used 243 Indexes, 2 Functions, 365 Partitions/Y+hints 5 weeks + DBAs to setup the facility MongoDB 0.4s to complete the test queries Single machine, 8 Cores/16G 42 GB of disk space used 4 hours to setup the facility 1 Table, 1 Index

11. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Software distribution technologies: CVMFS ATLAS and LHCb moving to a dynamic software distribution model via CVMFS Virtual software installation by means of an HTTP File System Data Store Compressed Chunks (Files) Eliminates Duplicates File Catalog Directory Structure Symlinks SHA1 of Regular Files Digitally Signed Time to Live Nested Catalogs ATLAS also plans to distribute the condition files via CVMFS Export the experiment software as read-only Mounted in the remote nodes via the fuse module Local cache for faster access Benefits of a squid hierarchy to guarantee performance, scalability and reliability Same squid type as the one used for Frontier

12. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Data access technologies: XRootd Federations Use XRoot redirector to create a unified access point for sites - “Global Redirector” Unified namespace, 1 protocol, location-neutral High-perf, low-management data access Mainly for T3s but also reduced load on T1/T2 Run jobs on remote data (w/ or w/o local cache) XRD: High-performance, scalable, flexible, plugin-based architecture Support dCache, GPFS, Hadoop, etc as storage backends What level of security is appropriate for read-only access to physics data? GSI supported via 'seclib' plugin State of VOMS support? More overhead to deploy/operate User- or host- based? (service cert?) Performance implications

13. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Workload Management: CMS WMCore/WMAgent Essentially a reworking of ProdAgent architecture to address the shorfalls in data processing CMS is currently experiencing WMSpec The language used to describe the organizational units of a workload WorkQueue A Work Queue Element is some chunk of work WMAgent has a local Work Queue JobStateMachine All job operations deal with arrays of jobs All state changes are DB operations WMBS Defines Job Entities Allow a straightforward way to control job creation rates Provides a standard framework for implementing job splitting for various tasks Model and execute the complex tangle of dependencies within workflows. WMSpec Define Work WorkQueue Buffer Work JobStateMachine Manage Work WMBS Manage Dependencies WMAgent

14. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Workload Management: GlideinWMS A glidein is just a properly configured execution node submitted as a Grid job glideinWMS is an automated tool for submitting glideins on demand The glideinWMS is composed of three logical entities, two being actual services: Glidein factories know about the Grid VO frontend know about the users and drive the factories

15. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Workload Management: Cloud Computing in ATLAS Testing the performance of the ATLAS Simulation and Reconstruction Jobs in the Amazon EC2 cloud Compare to a local machine See which way is more cost-effective Some work to be done to use clouds Data staging (Clouds as DDM target) FTS support? CE/batch system in the cloud (e.g., condor EC2) or pilot built in to VM? Using clouds is very attractive, but to make it cost-effective, we need to be our own cloud-provider!

16. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Conclusions The WLCG Collaboration prepared, deployed and is now managing the common computing infrastructure of the LHC experiments Coping reasonably well so far with the large amount of data that is distributed, stored and processed every day All the LHC experiments are successfully using the Grid distributed computing to perform the data analysis MC, real data Central productions and user’s analysis New technologies have been implemented or are under study to overcome the current limitations of the system and to better adapt to the experiments’ needs Many R&D developments and studies, more manpower needed to support this! Credits: D. Barberis, Y. Yao, C. Waldman, V. Garonne, R. Santinelli, A. Sciabà, D. Spiga

17. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 BACKUP SLIDES

18. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Analysis interfaces: ALICE AlienSh Benefits from a strong connection with the ROOT framework Included in the more general offline framework AliRoot Moving from ROOT on your laptop to a PROOF server to the Grid nearly transparent to user. ALICE has concept where user code is added in and executed in centralised analysis pass over data Reach better CPU utilisation this way Dynamic checking of user C++ code.

19. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Analysis interfaces: ALICE [2] Each analysis instantiates an AliAnalysisManager steering object 3 analysis modes Local Local CPU, data on GRID GRID Via a plugin (AliAnalysisGrid) in the manager Allows writing a jdl and submit analysis jobs Plugin mode: test (pre-start check) full (real job) terminate (close a previous session) Proof Analysis on a PROOF cluster

20. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Analysis interfaces: ALICE [3] Input data selection TAGS based (currently not used so much, will be the standard selection in the future) Local analysis AOD or ESD chain from the input collection Analysis on the GRID N jobs started by the plugin, each one processing no more than M AODs in input Aggregation is automatically done Output data Done at the end of the chain / end of the job / end of the proof session Terminate Output data merge, important for the analysis GRID mode: The output file is stored in the pre-selected Grid SE pre-selected Output – root file: replicated on the local disk Single macro for Testing the code locally Using it on small datasets with PROOF Send large-scale jobs in GRID

21. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Analysis interfaces: ATLAS Basic model: Data is pre-distributed to the sites, jobs are brokered to a site having the data Large dataset containers are distributed across clouds, so the front-ends do not restrict jobs to a cloud. i.e. DA jobs run anywhere in the world.

22. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Analysis interfaces: ATLAS DDM Files are aggregated in datasets Files are transferred only as part of a dataset Subscription service One or more dataset can be subscribed (or unsubscribed) to trigger a replica to a specific target site Physics metadata kept in a separate DB (AMI)

23. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011  PanDA is used to run all MC and Reprocessing, and ~75% of the user analysis worldwide  PanDA@CERN deployed >1 year ago and is running successfully.  The service was well prepared thanks to pre-exercises such as STEP’09 Panda load depends more on the number of resources (~70 sites), and less so with the amount of data Analysis interfaces: ATLAS PanDA

24. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Analysis interfaces: GANGA Used both in ATLAS and LHCb Gaudi /Athena aNd Grid Alliance A fully programmable interface for the processing of data Debug code locally, progress to small analysis in batch farms, run full analysis on Grid “Configure once, run everywhere” All done by a one line change of job specification Full Run Test Debug

25. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Analysis interfaces: Ganga [2] An analysis process defined through a set of building blocks forming a “job”. All building blocks provided as plugins Easy to write your own job

26. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Analysis interfaces: Ganga [3] Application (Gaudi/Athena based) MonteCarlo, Simulation, Reconstruction, Analysis Software release selection User code or pre-compiled library submission with the analysis jobs Backends LCG/PanDA NorduGrid Destination site selection Input Dataset Local file system DDM Dataset identification tools Output Dataset Local file system DDM Destination Storage Element (defaults to close SE) Splitter – Merger Subjobs definition Can join ouput ROOT files

27. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Analysis interfaces: CRAB CMS Remote Analysis Builder (CRAB) user-friendly interface to handle data analysis in a local or distributed environment, hiding the complexity of interactions with the Grid and CMS services Data discovery and location DBS Dataset Bookkeeping Service (DBS) provides the means to describe, discover and use CMS event data catalogs CMS specific data definitions such as run number, the algorithms and configurations used to process the data together with the information regarding the processing parentage of the data it describes PhEDEx The CMS data placement and transfer system Technically PhEDEx is based on software agents storing their state and communicating via a central “black board” database hosted in the CERN ORACLE RAC

28. A. De Salvo – – IFAE 2011 – 28-04-2011 A. De Salvo – New data processing technologies at LHC: from Grid to Cloud Computing and beyond – IFAE 2011 – 28-04-2011 Analysis interfaces: CRAB [2] Job preparation Pack local user code and the environment to be sent to remote sites Job splitting Job submission to Grid sites hosting the required data Job monitoring Monitor the status of the submitted jobs by querying Grid services. Handling Output data Copy the produced output to a remote site or return it to the user for small files (few MB) Publish the produced data with their description and provenance into a local DBS