A.Guarise – F.Rosso 1 Enabling Grids for E-sciencE INFSO-RI-508833 Comprehensive Accounting Views on large computing farms. Andrea Guarise & Felice Rosso.

Slides:

Advertisements

Similar presentations

EGEE-II INFSO-RI Enabling Grids for E-sciencE The gLite middleware distribution OSG Consortium Meeting Seattle,

Advertisements

FP7-INFRA Enabling Grids for E-sciencE EGEE Induction Grid training for users, Institute of Physics Belgrade, Serbia Sep. 19, 2008.

E-science grid facility for Europe and Latin America A Data Access Policy based on VOMS attributes in the Secure Storage Service Diego Scardaci.

CoreGRID Workpackage 5 Virtual Institute on Grid Information and Monitoring Services Authorizing Grid Resource Access and Consumption Erik Elmroth, Michał.

Sergey Belov, LIT JINR 15 September, NEC’2011, Varna, Bulgaria.

Accounting Update Dave Kant Grid Deployment Board Nov 2007.

OSG Middleware Roadmap Rob Gardner University of Chicago OSG / EGEE Operations Workshop CERN June 19-20, 2006.

1 School of Computer, National University of Defense Technology A Profile on the Grid Data Engine (GridDaEn) Xiao Nong

Publication and Protection of Site Sensitive Information in Grids Shreyas Cholia NERSC Division, Lawrence Berkeley Lab Open Source Grid.

INFSO-RI Enabling Grids for E-sciencE Logging and Bookkeeping and Job Provenance Services Ludek Matyska (CESNET) on behalf of the.

Summary of Accounting Discussion at the GDB in Bologna Dave Kant CCLRC, e-Science Centre.

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Security and Job Management.

1 OSG Accounting Service Requirements Matteo Melani SLAC for the OSG Accounting Activity.

Grid User Management System Gabriele Carcassi HEPIX October 2004.

Tool Integration with Data and Computation Grid GWE - “Grid Wizard Enterprise”

Fabric Monitor, Accounting, Storage and Reports experience at the INFN Tier1 Felice Rosso on behalf of INFN Tier1 Workshop sul.

EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks WMSMonitor: a tool to monitor gLite WMS/LB.

Accounting in LCG Dave Kant CCLRC, e-Science Centre.

June 24-25, 2008 Regional Grid Training, University of Belgrade, Serbia Introduction to gLite gLite Basic Services Antun Balaž SCL, Institute of Physics.

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks State of Interoperability Laurence Field.

What is SAM-Grid? Job Handling Data Handling Monitoring and Information.

GDB March User-Level, VOMS Groups and Roles Dave Kant CCLRC, e-Science Centre.

Overview of Privilege Project at Fermilab (compilation of multiple talks and documents written by various authors) Tanya Levshina.

Fabric Monitoring at the INFN Tier1 Felice Rosso on behalf of INFN Tier1 Joint OSG & EGEE Operations WS, Culham (UK)

INFSO-RI Enabling Grids for E-sciencE EGEE is a project funded by the European Union under contract INFSO-RI Grid Accounting.

INFSO-RI Enabling Grids for E-sciencE G-PBox Auth meeting 13/9/2005 Presenter: Vincenzo Ciaschini.

VO Privilege Activity. The VO Privilege Project develops and implements fine-grained authorization to grid- enabled resources and services Started Spring.

INFSO-RI Enabling Grids for E-sciencE GridICE: Grid and Fabric Monitoring Integrated for gLite-based Sites Sergio Fantinel INFN.

HLRmon accounting portal DGAS (Distributed Grid Accounting System) sensors collect accounting information at site level. Site data are sent to site or.

INFSO-RI Enabling Grids for E-sciencE ARDA Experiment Dashboard Ricardo Rocha (ARDA – CERN) on behalf of the Dashboard Team.

Recent improvements in HLRmon, an accounting portal suitable for national Grids Enrico Fattibene (speaker), Andrea Cristofori, Luciano Gaido, Paolo Veronesi.

DTI Mission – 29 June LCG Security Ian Neilson LCG Security Officer Grid Deployment Group CERN.

EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Grid2Win : gLite for Microsoft Windows Roberto.

Tool Integration with Data and Computation Grid “Grid Wizard 2”

EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks APEL CPU Accounting in the EGEE/WLCG infrastructure.

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Update Authorization Service Christoph Witzig,

LCG WLCG Accounting: Update, Issues, and Plans John Gordon RAL Management Board, 19 December 2006.

EGEE-II INFSO-RI Enabling Grids for E-sciencE Storage Accounting for Grid Environments Fabio Scibilia INFN - Catania.

LCG Accounting Update John Gordon, CCLRC-RAL WLCG Workshop, CERN 24/1/2007 LCG.

INFSO-RI Enabling Grids for E-sciencE Policy management and fair share in gLite Andrea Guarise HPDC 2006 Paris June 19th, 2006.

LCG User Level Accounting John Gordon CCLRC-RAL LCG Grid Deployment Board October 2006.

Accounting in LCG/EGEE Can We Gauge Grid Usage via RBs? Dave Kant CCLRC, e-Science Centre.

EGEE is a project funded by the European Union under contract INFSO-RI Grid accounting with GridICE Sergio Fantinel, INFN LNL/PD LCG Workshop November.

Accounting in LCG Dave Kant CCLRC, e-Science Centre.

HLRmon accounting portal The accounting layout A. Cristofori 1, E. Fattibene 1, L. Gaido 2, P. Veronesi 1 INFN-CNAF Bologna (Italy) 1, INFN-Torino Torino.

INFN GRID Production Infrastructure Status and operation organization Cristina Vistoli Cnaf GDB Bologna, 11/10/2005.

Site Authorization Service Local Resource Authorization Service (VOX Project) Vijay Sekhri Tanya Levshina Fermilab.

INFSO-RI Enabling Grids for E-sciencE DGAS, current status & plans Andrea Guarise EGEE JRA1 All Hands Meeting Plzen July 11th, 2006.

EGEE-II INFSO-RI Enabling Grids for E-sciencE Overview of gLite, the EGEE middleware Mike Mineter Training Outreach Education National.

SAM architecture EGEE 07 Service Availability Monitor for the LHC experiments Simone Campana, Alessandro Di Girolamo, Nicolò Magini, Patricia Mendez Lorenzo,

INFN/IGI contributions Federated Clouds Task Force F2F meeting November 24, 2011, Amsterdam.

TIFR, Mumbai, India, Feb 13-17, GridView - A Grid Monitoring and Visualization Tool Rajesh Kalmady, Digamber Sonvane, Kislay Bhatt, Phool Chand,

HLRmon Enrico Fattibene INFN-CNAF 1EGI-TF Lyon, France19-23 September 2011.

DGAS Distributed Grid Accounting System INFN Workshop /05/1009, Palau Giuseppe Patania Andrea Guarise 6/18/20161.

Using HLRmon for advanced visualization of resource usage Enrico Fattibene INFN - CNAF ISCG 2010 – Taipei March 11 th, 2010.

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 1 Grid Accounting Status at INFN Riccardo Brunetti INFN-TORINO.

EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI Services for Distributed e-Infrastructure Access Tiziana Ferrari on behalf.

The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.

Enabling Grids for E-sciencE Claudio Cherubino INFN DGAS (Distributed Grid Accounting System)

DGAS Accounting – toward national grid infrastructures HPDC workshop on Monitoring, Logging and Accounting, (MLA) in production Grids 10/06/2009, Munich.

Accounting Update Dave Kant, John Gordon RAL Javier Lopez, Pablo Rey Mayo CESGA.

EGEE is a project funded by the European Union under contract INFSO-RI DGAS Grid accounting L.Gaido on behalf of A.Guarise LCG Workshop November.

Jean-Philippe Baud, IT-GD, CERN November 2007

A Grid Job Monitoring System

DGAS A.Guarise April 19th, Athens

Accounting at the T1/T2 Sites of the Italian Grid

October 11th, CNAF GDB Meeting

Giuseppe Patania Nov, Martina Franca (Ta)‏

Danilo Dongiovanni INFN-CNAF

DGAS Today and tomorrow

Presentation transcript:

A.Guarise – F.Rosso 1 Enabling Grids for E-sciencE INFSO-RI Comprehensive Accounting Views on large computing farms. Andrea Guarise & Felice Rosso – HEPIX, Rome, April 5th,

A.Guarise – F.Rosso 2 Enabling Grids for E-sciencE INFSO-RI A Grid accounting infrastructure A generic grid accounting infrastructure should cover the requirements of tree type of customers: Sites, VOs and their users, grid administrators. While fulfilling the requirements it must also be secure, flexible and scalable. Site accounting database Site layer Usage MeteringResource’s layer VO accounting database Users/VO layer grid accounting databasegrid layer Web reports: -Aggregate info -Sites load -Anonymised statistics -Aggregate VO info -Group/Role info -Detailed User info -Job level info -Aggregate site info -VO (with role/group) usage on the site. -Detailed Resource Usage info -Job level info

A.Guarise – F.Rosso 3 Enabling Grids for E-sciencE INFSO-RI DGAS features Gianduia (sensors deployed at site) can build accounting records for grid jobs as well as local jobs (accounted on different HLR accounts) for grid jobs: integrates local usage information with grid-related information (user DN, user FQAN (VOMS), CE ID, grid job ID). supports PBS and LSF (work on SGE and Condor) “next-to-real-time” accounting (after job completion) HLR service (accounting servers) fully distributed architecture (scalability) deployed as User (VO) HLRs and/or Resource (Site) HLRs command line tools and C++ API for queries strict authorization control (x509) for accessing private info on Resource/Site HLRs: DGAS2APEL (optional) PA service (resource pricing servers) for manually or dynamically setting prices (virtual credits) of CEs price information might be included in the RB's scheduling process in order to establish a grid resource market (may help in balancing demand and supply) Data transport: all communication between the components as well as between client tools and servers is encrypted and secure (x509-based, GSI) mostly asynchronous data transport between the components for more robustness

A.Guarise – F.Rosso 4 Enabling Grids for E-sciencE INFSO-RI REDEYE + DGAS In this t alk we illustrate how the resource’s layer sensor system ‘REDEYE’ has been integrated with DGAS to implement the italian grid T1 resource accounting system. DGAS site HLR Site layer Red eye + DGAS Resource’s layer APEL and GOC web interface grid layer Web reports: -Aggregate info -Sites load -Anonymised statistics -Aggregate VO info -Group/Role info -Detailed User info -Job level info -Aggregate site info - VO (with role/group) usage on the site. -Detailed Resource Usage info -Job level info DGAS VO HLR Users/VO layer

A.Guarise – F.Rosso 5 Enabling Grids for E-sciencE INFSO-RI How RedEye works Developed for LSF 6.x (working since April 2005) Parse LSF logfile Info: RAM, WCT, CPU, Time, WN, Submitter etc. Crossed check between logfiles and LSF cache. From local DB: KSI2K and # of LSF slots of the WN Identity local user: LDAP server Identity GRID user: DN certificate from DB-cert DB-cert: realtime upgraded from local CEs Upgrade HLR server We are ready for fireworks!

A.Guarise – F.Rosso 6 Enabling Grids for E-sciencE INFSO-RI Accessing to HLR Protocol: https Common GRID user can have info about his/her own jobs or to global statistics (privacy is safe!) Special users can have access to their whole VO info Local System Managers have access to whole HLR Table creations Global/VO/User pie chart plots Global/VO/User historical plots

A.Guarise – F.Rosso 7 Enabling Grids for E-sciencE INFSO-RI Jobs submitted at T1-INFN Accessible by everyone

A.Guarise – F.Rosso 8 Enabling Grids for E-sciencE INFSO-RI KSI2K used at T1-INFN Accessible by everyone

A.Guarise – F.Rosso 9 Enabling Grids for E-sciencE INFSO-RI KSI2K used at T1-INFN Accessible by everyone

A.Guarise – F.Rosso 10 Enabling Grids for E-sciencE INFSO-RI KSI2K used at T1-INFN Accessible by everyone

A.Guarise – F.Rosso 11 Enabling Grids for E-sciencE INFSO-RI Feb 2006 stats at T1-INFN Accessible by everyone

A.Guarise – F.Rosso 12 Enabling Grids for E-sciencE INFSO-RI KSI2K used by CMS users at T1-INFN Accessible by special CMS users and local sys managers

A.Guarise – F.Rosso 13 Enabling Grids for E-sciencE INFSO-RI What more? For now access to HLR is only by LAN-CNAF In few weeks WAN access Job monitoring (same HLR ACLs) [Already working, for now access from LAN-CNAF] Storage (SE) accounting [in progress]