DIRAC Project A.Tsaregorodtsev (CPPM) on behalf of the LHCb DIRAC team A Community Grid Solution The DIRAC (Distributed Infrastructure with Remote Agent Control) project began as a solution for the LHCb experiment at CERN to carry out massive Monte Carlo simulation and data processing on various distributed computing resources. Now it is evolving to a complete Grid solution for community of users such as LHCb. DIRAC forms a layer between a particular community of users and compute resources which:  aggregates and masks the heterogeneity of computing and storage systems;  provides consistent workload and data management tools;  provides means to coordinate the activity of the community;  improves the reliability of available resources by adding extra redundancy and failover mechanisms. The DIRAC (Distributed Infrastructure with Remote Agent Control) project began as a solution for the LHCb experiment at CERN to carry out massive Monte Carlo simulation and data processing on various distributed computing resources. Now it is evolving to a complete Grid solution for community of users such as LHCb. DIRAC forms a layer between a particular community of users and compute resources which:  aggregates and masks the heterogeneity of computing and storage systems;  provides consistent workload and data management tools;  provides means to coordinate the activity of the community;  improves the reliability of available resources by adding extra redundancy and failover mechanisms. The DIRAC architecture consists of numerous cooperating Distributed Services and Light Agents all built using the same framework. This allows to create secure and efficient systems suiting the needs of a particular user community. The DISET framework provides secure GSI-based authentication and versatile authorization rules for interacting DIRAC components.  See R.Graciani et al – [177], A.Casajus et al – [178] The DIRAC architecture consists of numerous cooperating Distributed Services and Light Agents all built using the same framework. This allows to create secure and efficient systems suiting the needs of a particular user community. The DISET framework provides secure GSI-based authentication and versatile authorization rules for interacting DIRAC components.  See R.Graciani et al – [177], A.Casajus et al – [178] DIRAC introduced the now widely used concept of Pilot Agents. This allows to build efficient Workload Management Systems (WMS) that are resilient to failures in the ever changing Grid environment. The main WMS characteristics are:  Ability to work with various batch systems, grids and standalone PCs with different flavors of operating systems (Linux, Windows, … );  Job prioritization in the central Task Queue allows for an effective implementation of the community policies;  Flexible workload optimization with Pilot Agents allows a single DIRAC WMS instance to manage both massive data production and user analysis activities. See S.Paterson et al – [174], [176], G.Castellani et al – [340], Y.Y.Li et al – [296] DIRAC introduced the now widely used concept of Pilot Agents. This allows to build efficient Workload Management Systems (WMS) that are resilient to failures in the ever changing Grid environment. The main WMS characteristics are:  Ability to work with various batch systems, grids and standalone PCs with different flavors of operating systems (Linux, Windows, … );  Job prioritization in the central Task Queue allows for an effective implementation of the community policies;  Flexible workload optimization with Pilot Agents allows a single DIRAC WMS instance to manage both massive data production and user analysis activities. See S.Paterson et al – [174], [176], G.Castellani et al – [340], Y.Y.Li et al – [296] The DIRAC project includes a versatile Data Management System (DMS) which unites both native and third party components. The DMS features are:  Automated data distribution along the entire data processing path from the LHCb detector to the final analysis;  Reliable data transfers at all stages of the processing due to multiple failover and retry mechanisms;  Multiple data consistency checks to ensure integrity and recover from data losses. See A.C.Smith et al – [194], [195], M.Bargiotti et al – [168] The DIRAC project includes a versatile Data Management System (DMS) which unites both native and third party components. The DMS features are:  Automated data distribution along the entire data processing path from the LHCb detector to the final analysis;  Reliable data transfers at all stages of the processing due to multiple failover and retry mechanisms;  Multiple data consistency checks to ensure integrity and recover from data losses. See A.C.Smith et al – [194], [195], M.Bargiotti et al – [168] The DIRAC Production Management System is built on top of the Workload and Data Management services. This can sustain the production load of the LHCb experiment which amounts to many thousands of jobs per day. The Production Management System comprises the following functionalities:  Definition of complex workflows of the LHCb production jobs;  Automatic data driven submission of processing jobs  Automatic software installation at production sites;  Site sanity checks with results made available to the site managers and service providers. See J.Closier et al – [232] The DIRAC Production Management System is built on top of the Workload and Data Management services. This can sustain the production load of the LHCb experiment which amounts to many thousands of jobs per day. The Production Management System comprises the following functionalities:  Definition of complex workflows of the LHCb production jobs;  Automatic data driven submission of processing jobs  Automatic software installation at production sites;  Site sanity checks with results made available to the site managers and service providers. See J.Closier et al – [232] User Interface User Interface DIRAC API Community Production Tools Community Production Tools Community Policies Community Policies Central Task Queue Users Production managers Administrator DIRAC WMS EGEE NorduGrid Windows Compute Cluster Batch Systems The DIRAC System is a complete Community Grid solution. It has all the necessary components to build Workload and Data management systems of varying complexity. During the LHCb Data Challenge 2006:  1.5 million jobs were executed.  Up to 10K jobs were running simultaneously.  More than 120 sites were involved. See R.Nandakumar et al – [150] The LHCb DIRAC distributed user analysis platform has been stable for 2 years now. The DIRAC project scales to the requirements of the LHCb experiment in exploiting Grid Computing Resources. DIRAC offers a powerful Grid solution for other user communities. The DIRAC System is a complete Community Grid solution. It has all the necessary components to build Workload and Data management systems of varying complexity. During the LHCb Data Challenge 2006:  1.5 million jobs were executed.  Up to 10K jobs were running simultaneously.  More than 120 sites were involved. See R.Nandakumar et al – [150] The LHCb DIRAC distributed user analysis platform has been stable for 2 years now. The DIRAC project scales to the requirements of the LHCb experiment in exploiting Grid Computing Resources. DIRAC offers a powerful Grid solution for other user communities. DIRAC Services and Agents Configuration SystemMonitoring SystemLogging System DISET CLI / GUIWeb site The DIRAC team: A.Tsaregorodtsev (CPPM, Marseille), M.Bargiotti (CERN), N.Brook (H. H. Wills Physics Laboratory, Bristol), A.Casajus Ramo (University of Barcelona), G.Castellani (CERN), Ph.Charpentier (CERN), C.Cioffi (Oxford University), J.Closier (CERN), R.Graciani Diaz (University of Barcelona), G.Kuznetsov (Rutherford Appleton Laboratory), Y.Y.Li (University of Cambridge), R.Nandakumar (Rutherford Appleton Laboratory), S.Paterson (CERN), R.Santinelli (CERN), A.C. Smith (CERN), M. Seco Miguelez (University of Santiago de Compostela), S.Gomez Jimenez (University Rovira i Virgili, Tarragona) Automated Data Distribution System Automated Data Distribution System Reliable Data Transfer Layer Reliable Data Transfer Layer Replica Manager Data Manager LHCb Detector SRM xxxFTP DIRAC SE LFC Proc DB DIRAC FC Storages Catalogs Production Jobs FTS VO-box Jobs Production Manager Data Manager Production definitions Production definitions Data to process Data to process Processing Database Production Workflow Editor Data Distribution System DIRAC File Catalog DIRAC WMS Jobs Production Agent Grid A Grid B User Community