1. Accounting Manager Taking resource usage into your own hands Scott Jackson Scott.Jackson@pnl.gov http://www.emsl.pnl.gov/docs/mscf/gold

2. 2 Typical problems faced by HPC managers ProblemSolution Resources are hogged by the projects or project members that submit jobs most aggressively Fine-grained allocations Projects start slow then simultaneously demand their entire allocations be fulfilled at end of project cycle Expiring allocations (use-it-or-lose-it) Users can exceed their allocations until accounting is synchronized with resource management system Dynamic Charging, Reservations Would like to delegate the management and distribution of credits to established project management hierarchy Role-based AC, Nested accounts Want to share resources with other organizations but face issues of security, autonomy and accountability Strong security, Distrd accounting, Charge quotes Each site wants to track and perform historical queries on site-specific accounting statistics Custom accounting, Flexible charging, Journaling You want the accounting system to be transparent to the user Default projects, Credit accounts, Auto-generation

3. 3 IntroductionIntroduction Gold is an open source accounting system that allocates and tracks resource usage on High Performance Computers. A resource bank (allocation management system) Much like a bank, it associates a cost to computing resources and allows resource credits to be allocated to users and projects. As jobs complete or as resources are utilized, projects are dynamically charged and resource usage recorded. An accounting system Can be dynamically customized to record any type of accounting data – pacct, sar, node availability, performance data, etc. An information service Also functions as a powerful generalized information service useful in a variety of situations, such as providing meta-scheduling mappings of machines to resources, applications, accounts, users, etc.

4. 4 BackgroundBackground SciDAC Scientific Discovery through Advanced Computing – A DOE initiative to improve the impact of scientific computing Scalable Systems Software Project (SSS) Research, develop and support an integrated suite of systems software and tools for the effective management and utilization of the highest scale computational resources. 5 yr project involving 12 sites QBank Gold is based on a successful program, QBank, that has been used for years on government and university computing systems.

5. 5 Allocation Architecture Time PeriodAllocation F1Q041,000,000 F2Q041,000,000 F3Q041,000,000 F4Q041,000,000 Universe of Projects Universe of Projects Account: 1234 Name:Chemistry Account: 1234 Name:Chemistry Universe of Users Universe of Users Universe of Machines Universe of Machines Account: 1234 Name:Chemistry Account: 1234 Name:Chemistry Time PeriodAllocationCredit Limit F1Q041,000,00010,000 F2Q041,000,00010,000 F3Q041,000,00010,000 F4Q041,000,00010,000

6. 6 Dynamic Charging Accounting Manager (Gold) 0 2 1 4 3 5 6 Deposit 0 Deposits are made in Account 1 A Job is Submitted 2 A Quote is Requested Quote 3 A Reservation is Made Scheduler (Maui) 4 The Job is allowed to Start Reservation 5 The Job Completes Resource Manager (PBS, LL, LSF) 6 The Reservation is Removed Reservation Charge and a Charge is Issued

7. 7 Dynamic Charging 0 Make Deposits, etc. 1 Submit Job 2 Obtain Quote 3 Make Reservation 4 Start Job 5 Job Completes 6 Issue Charge Accounting Manager (Gold) 0 2 1 4 3 5 6 Resource Manager (PBS, LL, LSF)

8. 8 Dynamic Charging Resource Manager (PBS, LL,LSF) Allocation Manager (Gold) 0 2 1 5 3 3 7 0 Make Deposits, etc. 1 Submit Job 2 Locate Feasible Systems & Obtain Quote 3 Stage Job 4 Balance Check 5 Make Reservation 6 Start Job 7 Job Completes 8 Issue Charge Meta- Scheduler (Silver) Scheduler (Maui) 6 8 4

9. 9 Expiring Allocations

10. 10 Nested Accounts University Biology Actinide s Chem 201 Physics Chem 101 Chemistry Workshop 25 40 5040

11. 11 Web-based GUI

12. 12 Other Gold Features Strong Security Historical JournalingTransparency Features Dynamically Customizable

13. 13 Other Gold Features Flexible Charging Credit and Debit ModelsGuaranteed Quotes Powerful Querying

14. 14 Enables Grid Computing

15. 15 Distributed Accounting Gold facilitates resource sharing across administrative domains (Computational Grids) by supporting distributed accounting and allocation while preserving local autonomy Problems: Security concerns (privacy, protection from fraud) Want to maintain autonomous control of own resources Need to know how much it is going to cost you (contract) You want your own local accounting record (a receipt) Solutions: Traceback policy screening Distributed accounting Charge Quotes Strong authentication and encryption

16. 16 Custom Accounting Sites can create or modify record types on the fly enabling them to meet their custom accounting needs. This capability turns Gold into a generalized information service. [root@altix1 gold-1.0.a2.0]# gold -v gold> Object Create Name=Motorcycle gold> ObjectAttribute Create Object=Motorcycle Name=Name PrimaryKey=True gold> ObjectAttribute Create Object=Motorcycle Name=Color gold> ObjectAttribute Create Object=Motorcycle Name=HorsePower... gold> Motorcycle Create Name=Yamaha Color=Blue HorsePower=1200 Color Name HorsePower ----- ------ ---------- Blue Yamaha 1200 Successfully created 1 Motorcycle... gold> Motorcycle Modify Name==Yamaha HorsePower=1500 Color Name HorsePower ----- ------ ---------- Blue Yamaha 1500 Successfully modified 1 Motorcycles

17. 17 StatusStatus Status: Currently in beta release (4Q04) General release targeting 2Q05 Download: http://www.emsl.pnl.gov/docs/mscf/gold For more information: Scott.Jackson@pnl.gov http://www.emsl.pnl.gov/docs/mscf/gold http://sss.scl.ameslab.gov/home.shtml http://www.scidac.org/ScalableSystems