Accounting and Allocation Management with Gold Scott Jackson Moab·Con 2010

Allocation Manager Gold is an open source allocation system that tracks and charges for resource usage on High Performance Computers. Ensures resources are used according to mission plan Tracks resource utilization Allows for insightful capacity planning Facilitates resource sharing between organizations Used in production at dozens of sites

Gold in a nutshell: A resource billing system Gold tracks and manages resource usage on High Performance Computers. It behaves like a bank in which deposits are made into accounts and charges occur at the moment of job completion. Reservations prevent oversubscription. An allocation manager Credits can be allocated to specific projects, users and machines. Timeframes can be given for the expenditure of the credits enabling allocation cycles, capacity planning and use-it-or-lose it. An accounting system Gold maintains past state and a complete transaction history. It can track and maintain arbitrary custom accounting statistics. An information service Custom objects and attributes can be dynamically defined and manipulated.

Setting up Gold Define Users, Projects, Machines Add Users to Projects Setup Default Projects (optional) Make Deposits into Accounts

Accounts Projects Users Machines Allocations Universe of Projects Universe of Projects Account: 1234 Name:Chemistry Account: 1234 Name:Chemistry Universe of Users Universe of Users Universe of Machines Universe of Machines Account: 1234 Name:Chemistry TimePeriodAllocationCreditLimit F1Q10250,0000 F2Q10250,0000 F3Q10250,0000 F4Q10250,0000 FY10020,000

Allocations An allocation is an allotment of computational resource credits for use by specific projects or users during a specified time period. When jobs complete, projects are charged and resource usage recorded. Jobs are prevented from starting unless they have an active allocation with a sufficient balance. Limitations: Does manage short-term usage, does not impact scheduling priority. ProjectUsersMachineTime PeriodBalance BiologyAmy, BobColonyFY ,000 chemistryBob, DaveColonyFY ,000

Reservations and Dynamic Charging Accounting Manager (Gold) Deposit 0 Deposits are made in Account 1 A Job is Submitted 2 A Quote is Requested Quote 3 A Reservation is Made Scheduler (Moab) Scheduler (Moab) 4 The Job is allowed to Start Reservation 5 The Job Completes Resource Manager (Torque, LL, LSF) Resource Manager (Torque, LL, LSF) 6 The Reservation is Removed Reservation and a Charge is Issued Charge

Capacity & Workload Planning: The Brick Approach Controlling project usage is key to capacity and workload planning. The use of regularly expiring allocations allows you to establish a project cycle and use the brick approach to allocation management. Now - 1 Qtr+ 3 Qtr+ 2 Qtr+ 1 Qtr- 3 Qtr- 2 Qtr+ 4 Qtr 100 % Capacity

Year-end Resource Exhaustion It is common to see projects get off to a slow start and then have a surge of activity at the end when results are due. Without careful management of expectations, this can be a cause of considerable anxiety. 100% Machine Capacity Project As Allotment Project As Demand Project Period

Expiring Allocations

ChargeRates Consumable Resource Charge Rates Consumable Resource Charge Rates define how much it costs per unit of time to use a consumable resource like processors, memory, etc. Usage Charge Rates Usage Charge Rates define how much to charge for metrics of total resource usage such as CpuTime, Licenses used, Power consumed (in Watts), etc. Name-Based Multiplier Rates Name-Based Multiplier Rates are quality based charge rates which apply a multiplicative charge factor based on a quality of the job such as QOS, nodetype, queue, user, time of day, etc. Value-Based Multiplier Rates Value-Based Multiplier are scaled multipliers which apply a multiplicative charge factor based on a numeric scaling factor. TypeNameRateDescription ResourceProcessors1Processor Charge ResourceMemory.001Memory in MB UsagePower.05Watts UsageCpuTime1Actual CPU QOSPremium2Hi Prio, Preemptor QOSNormal1Regular Prio QOSBottomfeeder.5Lo Prio, Preemptee

Other Gold Features Flexible Charging Credit and Debit Models Guaranteed Quotes Powerful Querying

Other Gold Features Strong Security Historical JournalingTransparency Features Dynamically Customizable

Web-based GUI

Nested Accounts University Biology Actinides Chem 201 Physics Chem 101 Chemistry Workshop

End of Life Spending Spree Offline banks cannot strictly enforce allocations Without reservations, more jobs can get started than allocation balance can support. Job Debit Accounts Job Debit Accounts

Sneak Peek -- ChargeRates Type Name Instance Rate Description Resource Processors 10 Value Based Resource (VBR) Service License Abacus 10 Name Based Resource (NBR) Usage Power 500 Value Based Usage (VBU) Feature Feature Telescope 5 Name Based Usage (NBU) Quality QOS Premium 2 Name Based Multiplier (NBM) Multiplier Discount 0.5 Value Based Multiplier (VBM) Fee Shipping 100 external Value Based Usage (xVBU) Token Zone Asia 400 external Name Based Usage (xNBU) *(Disk) User scottmo 10 Multi-dimensional VBR (MVBR) The multi-dimensional Resource is a resource value that changes on a per-attribute basis, and is assumed if the Type is anything other than the known types. For example, the above line illustrates a disk resource fee of 10 for User=scottmo, whereas a different value might be applied for User=fred. Total Charge = ((((sum of all NBR and (M)VBR*value) times walltime) + (sum of all NBU and VBU*value)) times (product of all NBM and VBM*value)) + (sum of all xNBU and xVBU*value)