Stephen Booth EPCC Stephen Booth GridSafe Overview

2 Grid-SAFE JISC funded project to build general purpose accounting/monitoring solution. – –Builds on accounting subsystem from SAFE user administration system used by HPCx/HECToR Challenges: –Need to work with wide variety of different local policies. –Need to work with both grids and local HPC resources. One solution won’t fit all potential users –Build kit of parts –Pre-built solutions for common deployment scenarios. Key aims –Modular design, individual functions can be deployed independently –Behaviour can be customised using plug-ins to implement different service policies.

Overview

4 Stand-alone accounting portal Initial development phase concentrated on –Accounting module –Reporting module Can be deployed together as stand-alone accounting portal –Consumes and stores accounting data in variety of formats –Generates on demand reports. –Target users: –Local Campus HPC resources. –Grids with centralised accounting. –Requirements –Java / apache-tomcat / mysql Current status –New D4 release of code and documentation August 24 th –Code deployed for ECDF cluster in Edinburgh –Development team can assist with installation/configuration.

5 Accounting Module Accounting data is stored in relational DB UsageRecords are treated as a collection of properties Code does not mandate formats. –Users can standardise on one format if they wish. Plug-in parser modules handle different types of input data. –OGF-UR –SGE –PBS –EGEE JobManager –Etc. Plug-in policy modules augment these allowing site local customisation –Generate additional properties such as Charge –Generate derived properties defined as expressions over other properties. –Decrement budgets –Link records to other tables (Person, Project etc.)‏ –Generate Aggregates

6 Report generation module Grid-safe uses XML templates to define reports –Can generate unified reports over multiple data tables containing different types of data –Tables/charts –Parameterised reports (e.g. to select user or project). Support reports in multiple formats –PDF HTML CSV Performance of report generation a particular issue –Utilise database effectively. –Old data can be moved to separate tables. –Support for Daily aggregates –For ECDF 51 million records -> 35 thousand aggregates

ClassificationPolicy Converts selected fields from raw accounting data into references to separate database table. –Reduces data footprint. –Augmenting information can be added to these tables. Example from ECDF SGERecord DailyAggregate Person UnixGroup Project Queue Institute School

8 OGF-UR support OGF-UR XML is supported as an interchange format –Parser plug-in to parse OGF-UR –Export module to format internal data as OGF-UR Grids may only want to use only this Format Various grid communities seem to interpret OGF-UR differently and/or make additional requirements beyond that in the schema –Required fields –Different charging models –Different global username models –OGF-UR spec allows extensions. –Specification will also evolve over time. Parser/exporter highly configurable to support variations/extensions.

Web Services RUPI –Current proposal from OGF RUS-WG –Web service for the upload of XML usage record. –Grid-SAFE has an implementation of the current upload service (RUPI). RUQI –Currently working on a proposal for a Query specification –Aims –Easy to implement in different code bases. –Provide sufficient functionality for efficient report generation. –Long term aim to provide reporting portal that can query any system that implements this interface.

Where next Need more early adopters. –Can add features/capabilities as requested Possible candidates –Instrumentation of Job Prologue/Epilogue –Useful for systems without an accounting logfile to parse. –Would also allow run-time budget checks. –Storage accounting. –There is existing code from HPCx/HECToR we could cross-port.