Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 1 Grid Accounting Status at INFN Riccardo Brunetti INFN-TORINO

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 2 Summary Why Accounting ? CPU Accounting and Storage Accounting DGAS Features DGAS Components Security and Privacy Storage Accounting DGAS Deployment in the Italian Grid Work in Progress

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 3 Why Accounting ? In a distributed computing environment people want to know who used the resources and how many resource have been used. Users : How many resources am I using ? VO managers : how many resources is my VO using ? Site Managers : who is using my resources ? ROC Managers : How many resources have been used in my federation and who used them ? A good accounting system should be able to answer these questions taking care of all the security and privacy issues.

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 4 CPU Accounting Is the task of collecting usage metering records in terms of computing resources used and/or some other derived quantity (ex. SpecInt(SpecFloat)/t) Primary source of information : CE gatekeeper log + batch system logs Already deployed in InfnGrid (though in evolution) by mean of the DGAS software package

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 5 Storage Accounting Is the task of collecting usage metering records in terms of storage space used and/or some other derived quantity Primary source of information: transfer services logs + storage system logs Work in progress (recently started as activity) in InfnGrid to define the specifications.

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 6 What is DGAS ? DGAS is a distributed accounting system able to perform a resource usage metering and Economic Accounting (eventually as a basis for billing) in the Grid environment. It is based on a client/server infrastructure relying on a network of independent accounting servers. Developed inside the EDG/WP1 and EGEE/JRA1 project by INFN-TORINO people (A. Guarise, R. Piro, G. Patania)

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 7 DGAS Components Sensors on CEs – Build usage records from LRMS accounting files Resource (site) HLRs (Multilevel structure) – Collect usage records from one or more sites User (VO) HLRs – Collect usage records for a whole VO Query clients and visualization tools – Allow to retrieve data from HLRs

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 8 DGAS Features Granularity – Resource accounting at single job level or in aggregate form per user, per VO, per resource (site) or per infrastructure (collection of sites). Scalability – Arbitrary number of Resource/VO HLRs can be deployed. Hierarchical Design – HLRs can be interconnected, in order to have multiple levels of aggregation.

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 9 DGAS Features II Completeness – Capability to collect information both for grid and local jobs. – Accounting works with all the EGEE (gLite and LCG) RBs Customization – Possibility to choose which resource and/or type of jobs must be accounted. Designed to perform pricing (not used) – Possibility to assign “computing credits” (something similar to quotas) that can be used as a basis for billing the resource consumptions.

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 10 DGAS Workflow job Usage Record CE WN Site HLR VO HLR L2 HLR 3

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 11 Information retrieval Information are retrieved parsing the LRMS accounting log files and the CE gatekeeper log file LRMS log files : resource used (cpu-time,wall- time,lrmsid,user,group) gatekeeper log files : grid info (user DN, grid- jobid,VO) Some other config files are used in order to manage local jobs, for which the VO could not be defined through the proxy certificate.

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 12 Security and Privacy Information confidentiality is guaranteed by the use of different authorization levels to access the Usage Records. – Users (can access their own detailed records and aggregates) – Site Managers (Can access their own site detailed records and aggregates) – VO Managers (Can access detailed records and aggregates of all VO members) Full VOMS integration in query authorization is available (now on L2 HLR, on every HLR in future releases) – (e.g. /atlas/Role=vomanager/Group=NULL) Security and integrity of the data flow is guaranteed by the use of GSI and data encryption. No sensitive information sent in clear text

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 13 Interface to APEL DGAS Usage Records tables can be converted into APEL LCGRecords table structure. This instance of the table can then be sent to the GOC accounting database through R-GMA. This can be performed using the already existing APEL publisher. The user credentials are sent in encrypted form The translation tool (Dgas2Apel) is already tested and working. Possibility to choose which records must be converted and then sent (resource, grid/local jobs) Still some stability problems with R-GMA, but already running at T2 and T1 sites.

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 14 Storage Accounting Different types of Storage Elements: classic SE (gridftp + rfio) dCache SE (gridftp + gsidcap) dpm SE (gridftp + srfio) castor (gridftp + (s)rfio or gsidcap) storm (gridftp + posix-like access) Need to cross check different log files in order to collect both storage and user/VO information Some type of SE does not log all type of operations At the moment no standard specifications available for “usage record” Used space ? Time of permanence on SE ? Stored files only or transferred files too ? …..

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 15 Catania: SAGE (Storage Accounting in a Grid Environment) (F.Scibilia, C. Cherubino, D. Russo) It is a software architecture to monitor the storage space used (usage metering). It works on Disk Pool Manager (DPM) based SE No modifications to DPM requested Generates Usage Records which refer to disk usage Usage Records are build by looking to GridFTP- DPM e RFIO log files DPM internal DB maintains history of operations, certifiates, turls ecc.. It is foreseen to forward storage Usage Record to DGAS HLRs as well.

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 16 SAGE data collection Table DPM_REQ Information on user requesting operation Tables containing specific information on the type of operation ( PUT, GET o COPY)

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 17 Bari: Castor & dCache monitoring/accounting (G. Cuscela, G. Donvito) Castor: Monitoring of data transfers from gridftp log User info from “messages” log Info stored in a local db Availlable information: Type of operation Transferred files Hosts involved User Info Timing information

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 18 Bari: Castor & dCache monitoring/accounting (G. Cuscela, G. Donvito) dCache: Information mainly from “billing” files Local db used Availlable information: File “storage-class” Type of operation (put, get, local access) Hosts involved (pool node & remote host) User, Group, VO User DN Bytes transferred and protocol File deletion

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 19 DGAS Deployment in the Italian Grid DGAS deployed in 43 sites (RPMs+YAIM) L1 HLR in 1 T1 site (CNAF-T1) L1 HLR in 9 T2 Sites + Padova and Catania (2 of them registering data for small T3 sites)

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 20 DGAS Deployment Italian Grid (L2 HLR) L2 HLR in 1 Site (Torino) collecting data for T2 sites (Torino,Milano,Catania,Frascati,Pisa, Bari,Napoli) Roma1,Legnaro will be added soon INFN-ROMA

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 21 DGAS data validation Data collected by site L1 HLR have been verified by mean of a cross check with LRMS accounting log files Once a site has been validated, it is admitted to send data to II level HLR. At the moment, validation was successful for 7 T2 sites. 3 are pending (including T1) because of : a known problem of DGAS with some particular configuration of LSF batch system. a discovered problem with LSF log rotation (a patched version was just certified and is going to be deployed) Validation of the small sites (not having his own HLR) is on going. Validated sites started also to send data to GOC.

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 22 Examples Site level Information from L1 HLR Aggregate per VO Aggregate per User

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 23 Examples HLR Query Client: Aggregate (job number, hours of CPU(WALL) time, ”efficiency”) for the VO running jobs in Torino.

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 24 Examples Multi Site Information from L2 HLR Jobs per site (T1 included) Jobs per VO (T1 included)

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 25 Examples Multi Site Information from L2 HLR Jobs per day (T1 included)

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 26 Web Interface to II Level HLR (Work in progress) F. Pescarmona S. Dalpra F. Rosso G. Misurelli E. Fattibene G. Patania

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 27 Web Interface to II Level HLR Shows accounting data in aggregate form A set of predefined aggregates are built using data on II level HLR User is identified by mean of his certificate and is allowed to plot charts according to his own VO role. Capability to completely customize the queries is foreseen (need to carefully play with authorizations)

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 28 DGAS References General information about DGAS can be found at: DGAS website: DGAS User Guides: kie= &document_id=571271&version=1

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 29 Bari: Castor & dCache monitoring/accounting (G. Cuscela, G. Donvito)