HLRmon accounting portal The accounting layout A. Cristofori 1, E. Fattibene 1, L. Gaido 2, P. Veronesi 1 INFN-CNAF Bologna (Italy) 1, INFN-Torino Torino.

Slides:



Advertisements
Similar presentations
Legacy code support for commercial production Grids G.Terstyanszky, T. Kiss, T. Delaitre, S. Winter School of Informatics, University.
Advertisements

Sergey Belov, LIT JINR 15 September, NEC’2011, Varna, Bulgaria.
Consorzio COMETA - PI2S2 Project UNIONE EUROPEA SAGE – Storage Accounting for Grid Environments in gLite Fabio Scibilia Consorzio.
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
Makrand Siddhabhatti Tata Institute of Fundamental Research Mumbai 17 Aug
Accounting Update Dave Kant Grid Deployment Board Nov 2007.
Sergey Belov, Tatiana Goloskokova, Vladimir Korenkov, Nikolay Kutovskiy, Danila Oleynik, Artem Petrosyan, Roman Semenov, Alexander Uzhinskiy LIT JINR The.
LHC Experiment Dashboard Main areas covered by the Experiment Dashboard: Data processing monitoring (job monitoring) Data transfer monitoring Site/service.
Storage Accounting John Gordon, STFC GDB June 2012.
 DATABASE DATABASE  DATABASE ENVIRONMENT DATABASE ENVIRONMENT  WHY STUDY DATABASE WHY STUDY DATABASE  DBMS & ITS FUNCTIONS DBMS & ITS FUNCTIONS 
The National Grid Service User Accounting System Katie Weeks Science and Technology Facilities Council.
Summary of Accounting Discussion at the GDB in Bologna Dave Kant CCLRC, e-Science Centre.
A.Guarise – F.Rosso 1 Enabling Grids for E-sciencE INFSO-RI Comprehensive Accounting Views on large computing farms. Andrea Guarise & Felice Rosso.
Monitoring in EGEE EGEE/SEEGRID Summer School 2006, Budapest Judit Novak, CERN Piotr Nyczyk, CERN Valentin Vidic, CERN/RBI.
JSPG: User-level Accounting Data Policy David Kelsey, CCLRC/RAL, UK LCG GDB Meeting, Rome, 5 April 2006.
Database weekly reports Zbigniew Baranowski Carlos Fernando Gamboa.
Cracow Grid Workshop October 2009 Dipl.-Ing. (M.Sc.) Marcus Hilbrich Center for Information Services and High Performance.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks WMSMonitor: a tool to monitor gLite WMS/LB.
Accounting in LCG Dave Kant CCLRC, e-Science Centre.
Job and Data Accounting on the Open Science Grid Ruth Pordes, Fermilab with thanks to Brian Bockelman, Philippe Canal, Chris Green, Rob Quick.
GDB March User-Level, VOMS Groups and Roles Dave Kant CCLRC, e-Science Centre.
Overview of Privilege Project at Fermilab (compilation of multiple talks and documents written by various authors) Tanya Levshina.
US LHC OSG Technology Roadmap May 4-5th, 2005 Welcome. Thank you to Deirdre for the arrangements.
LCG Accounting John Gordon Grid Deployment Board 13 th January 2004.
INFSO-RI Enabling Grids for E-sciencE EGEE is a project funded by the European Union under contract INFSO-RI Grid Accounting.
INFSO-RI Enabling Grids for E-sciencE GridICE: Grid and Fabric Monitoring Integrated for gLite-based Sites Sergio Fantinel INFN.
HLRmon accounting portal DGAS (Distributed Grid Accounting System) sensors collect accounting information at site level. Site data are sent to site or.
EMI INFSO-RI Accounting John Gordon (STFC) APEL PT Leader.
Recent improvements in HLRmon, an accounting portal suitable for national Grids Enrico Fattibene (speaker), Andrea Cristofori, Luciano Gaido, Paolo Veronesi.
Automatic Resource & Usage Monitoring Steve Traylen/Flavia Donno CERN/IT.
Accounting Update John Gordon and Stuart Pullinger January 2014 GDB.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks APEL CPU Accounting in the EGEE/WLCG infrastructure.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Monitoring of the LHC Computing Activities Key Results from the Services.
CERN IT Department CH-1211 Genève 23 Switzerland t CERN IT Monitoring and Data Analytics Pedro Andrade (IT-GT) Openlab Workshop on Data Analytics.
LCG WLCG Accounting: Update, Issues, and Plans John Gordon RAL Management Board, 19 December 2006.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Storage Accounting for Grid Environments Fabio Scibilia INFN - Catania.
Global ADC Job Monitoring Laura Sargsyan (YerPhI).
LCG Accounting Update John Gordon, CCLRC-RAL WLCG Workshop, CERN 24/1/2007 LCG.
INFSO-RI Enabling Grids for E-sciencE Policy management and fair share in gLite Andrea Guarise HPDC 2006 Paris June 19th, 2006.
FTS monitoring work WLCG service reliability workshop November 2007 Alexander Uzhinskiy Andrey Nechaevskiy.
Accounting in LCG/EGEE Can We Gauge Grid Usage via RBs? Dave Kant CCLRC, e-Science Centre.
INFSO-RI Enabling Grids for E-sciencE DGAS, current status & plans Andrea Guarise EGEE JRA1 All Hands Meeting Plzen July 11th, 2006.
SAM architecture EGEE 07 Service Availability Monitor for the LHC experiments Simone Campana, Alessandro Di Girolamo, Nicolò Magini, Patricia Mendez Lorenzo,
INFN/IGI contributions Federated Clouds Task Force F2F meeting November 24, 2011, Amsterdam.
G. Russo, D. Del Prete, S. Pardi Kick Off Meeting - Isola d'Elba, 2011 May 29th–June 01th A proposal for distributed computing monitoring for SuperB G.
HLRmon Enrico Fattibene INFN-CNAF 1EGI-TF Lyon, France19-23 September 2011.
DGAS Distributed Grid Accounting System INFN Workshop /05/1009, Palau Giuseppe Patania Andrea Guarise 6/18/20161.
WLCG Accounting Task Force Update Julia Andreeva CERN GDB, 8 th of June,
Using HLRmon for advanced visualization of resource usage Enrico Fattibene INFN - CNAF ISCG 2010 – Taipei March 11 th, 2010.
Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 1 Grid Accounting Status at INFN Riccardo Brunetti INFN-TORINO.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI Services for Distributed e-Infrastructure Access Tiziana Ferrari on behalf.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.
Storage Accounting John Gordon STFC GDB, Lyon 6 th April2011 GDB January 2012.
Enabling Grids for E-sciencE Claudio Cherubino INFN DGAS (Distributed Grid Accounting System)
DGAS Accounting – toward national grid infrastructures HPDC workshop on Monitoring, Logging and Accounting, (MLA) in production Grids 10/06/2009, Munich.
Accounting Update Dave Kant, John Gordon RAL Javier Lopez, Pablo Rey Mayo CESGA.
WLCG Resource Reporting 2 Laurence Field Cloud Status Document A few comments received Mainly minor text changes New final version (v1) has been generated.
Accounting Review Summary and action list from the (pre)GDB Julia Andreeva CERN-IT WLCG MB 19th April
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI solution for high throughput data analysis Peter Solagna EGI.eu Operations.
How to integrate portals with EGI accounting system R.Graciani EGI TF 2012.
Job monitoring and accounting data visualization
DGAS A.Guarise April 19th, Athens
POW MND section.
Accounting at the T1/T2 Sites of the Italian Grid
Advancements in Availability and Reliability computation Introduction and current status of the Comp Reports mini project C. Kanellopoulos GRNET.
Giuseppe Patania Nov, Martina Franca (Ta)‏
Ákos Frohner EGEE'08 September 2008
WLCG dedicated accounting portal (proposal)
HLRmon accounting portal
User Accounting Integration Spreading the Net.
Presentation transcript:

HLRmon accounting portal The accounting layout A. Cristofori 1, E. Fattibene 1, L. Gaido 2, P. Veronesi 1 INFN-CNAF Bologna (Italy) 1, INFN-Torino Torino (Italy) DGAS (Distributed Grid Accounting System) sensors collect accounting information at site level. Site data are sent to site or multi-site HLRs (Home Location Register). A top level HLR receives accounting data from all the Grid HLRs. HLRmon retrieves data from the top level HLR and presents them through a web interface. Aggregated data are also sent to the central EGI Accounting Portal. HLRmon is successfully deployed as accounting portal for IGI and other Grid projects/NGIs using DGAS as accounting system. General purpose CPU reports WLCG reports Different metrics to analyze:  CPU and Wall Clock Time, expressed in days and normalized on the mean power capacity of the site  Number of executed jobs  Job efficiency in terms of (CPU Time / Wall Time) ratio Different keys of data aggregation:  Site  Virtual Organization  VOMS group and role  Certification Authority and Registration Authority  Job type (Grid or local)  Grid user DN An advanced selection menu permits to choose a desired subgroup of values for each aggregation key. A specific view with data grouped per discipline is provided: the mapping information between VOs and disciplines is currently retrieved from the EGI Operations Portal. The charts allow the analysis of CPU and storage resources usage at the WLCG Tier1 and Tier2 by the LHC experiments, over the last month and the last year. The resource usage and the amount of pledged resources are compared. Accounting data are grouped by site and by VO. Storage data are taken by a script running on the sites and sent to HLRmon via . Architecture Towards complete storage reports HLRmon provides a storage accounting section for WLCG sites and VOs. A more generalized approach, integrated in DGAS, is being developed. Non-standard schema is used to generate records (no standard schema exists). Records are retrieved at sites and stored on the HLRs. HLRmon is able to retrieve storage information directly from the desired HLR. Each record on the HLR has a timeDuration calculated as the difference between two consecutive measurements. HLRmon calculates the daily disk consumption in terms of space-per- hour as the integral of the used space over time. Data are aggregated per site, VO and Storage Area. Collectors retrieve:  aggregated CPU accounting information from a desired HLR  storage accounting data by probes running on the SRM instances at WLCG sites  values of some GLUE attributes about the computing power capacity of the WLCG sites An aggregation layer is responsible for the data manipulation to produce charts and tabular reports visualized by the graphical interface. Data aggregated per Grid user are directly retrieved from the HLR database at run- time and are just accessible by authorized users.