1. Accounting Manager Taking resource usage into your own hands Scott Jackson Pacific Northwest National Laboratory Scott.Jackson@pnl.gov

2. 2 Outline Background Overview Accounts, Projects, Users, Machines, Allocations Dynamic Charging Custom Accounting Distributed Accounting Schedule

3. 3 Background SciDAC Scientific Discovery through Advanced Computing – A DOE initiative to improve the impact of scientific computing Scalable Systems Software Center Research, develop and support an integrated suite of systems software and tools for the effective management and utilization of the highest scale computational resources. 5 yr project involving 12 sites QBank Gold is based on a successful program, QBank, that has been used for years on government and university computing systems.

4. 4 Overview The Gold Accounting Manager in a nutshell is: A resource bank Gold tracks and manages resource usage on High Performance Computers. It behaves like a bank in which deposits are made into accounts and charges occur at the moment of job completion. An allocation manager With allocation management, you can do much more than just track your usage profiles, you can help determine them. An accounting system Gold maintains past state and a complete transaction history. It can track and maintain arbitrary custom accounting statistics. A Grid accounting system Gold facilitates the sharing of resources within a Grid by providing distributed accounting while preserving local site autonomy.

5. 5 Typical problems faced by HPC managers ProblemSolution Resources are hogged by the projects or project members that submit jobs most aggressively Fine-grained allocations Projects start slow then simultaneously demand their entire allocations be fulfilled at end of project cycle Expiring allocations (use-it-or-lose-it) Users can exceed their allocations until accounting is synchronized with resource management system Dynamic Charging, Reservations Would like to delegate the management and distribution of credits to established project management hierarchy Role-based AC, Nested accounts Want to share resources with other organizations but face issues of security, autonomy and accountability Strong security, Distrd accounting, Charge quotes Each site wants to track and perform historical queries on site-specific accounting statistics Custom accounting, Flexible charging, Journaling You want the accounting system to be transparent to the user Default projects, Credit accounts, Auto-generation

6. 6 Accounts, Projects, Users, Machine, Allocations Universe of Projects Universe of Projects Account: 1234 Name:Chemistry Account: 1234 Name:Chemistry Universe of Users Universe of Users Universe of Machines Universe of Machines Account: 1234 Name:Chemistry Credit Limit: 1,000,000 Account: 1234 Name:Chemistry Credit Limit: 1,000,000 TimePeriodAllocation F1Q041,800,000 F2Q041,800,000 F3Q041,800,000 F4Q041,800,000

7. 7 Nested Accounts University Biology Actinide s Chem 201 Physics Chem 101 Chemistry Workshop 25 40 50 40

8. 8 Web-based GUI

9. 9 Custom Accounting Sites can create or modify record types on the fly enabling them to meet their custom accounting needs. This capability turns Gold into a generalized information service. [root@altix1 gold-1.0.a2.0]# gold -v gold> Object Create Name=Motorcycle gold> ObjectAttribute Create Object=Motorcycle Name=Name PrimaryKey=True gold> ObjectAttribute Create Object=Motorcycle Name=Color gold> ObjectAttribute Create Object=Motorcycle Name=HorsePower... gold> Motorcycle Create Name=Yamaha Color=Blue HorsePower=1200 Color Name HorsePower ----- ------ ---------- Blue Yamaha 1200 Successfully created 1 Motorcycle... gold> Motorcycle Modify Name==Yamaha HorsePower=1500 Color Name HorsePower ----- ------ ---------- Blue Yamaha 1500 Successfully modified 1 Motorcycles

10. 10 Dynamic Charging Available Balance Job Queue Action 1 Job Submission Quote Accounting Action 2 Job Start Reservation 3 Job Completion Charge Advantages of using reservations Strict quota enforcement (you can’t use more than you are allocated)

11. 11 Dynamic Charging 0 Make Deposits, etc. 1 Submit Job 2 Obtain Quote 3 Make Reservation 4 Start Job 5 Job Completes 6 Issue Charge Accounting Manager (Gold) 0 2 1 4 3 5 6 Resource Manager (PBS, LL, LSF)

12. 12 Dynamic Charging Resource Manager (PBS, LL, LSF) Scheduler (Maui) Accounting Manager (Gold) 0 2 1 4 3 5 6 0 Make Deposits, etc. 1 Submit Job 2 Obtain Quote 3 Make Reservation 4 Start Job 5 Job Completes 6 Issue Charge

13. 13 Dynamic Charging Resource Manager (PBS, LL,LSF) Allocation Manager (Gold) 0 2 1 5 3 3 7 0 Make Deposits, etc. 1 Submit Job 2 Locate Feasible Systems & Obtain Quote 3 Stage Job 4 Balance Check 5 Make Reservation 6 Start Job 7 Job Completes 8 Issue Charge Meta- Scheduler (Silver) Scheduler (Maui) 6 8 4

14. 14 Distributed Accounting Gold facilitates resource sharing across sites (Grids) Problems: Security concerns (privacy, protection from fraud) Want to maintain autonomous control of own resources Need to know how much it is going to cost you (contract) You want your own local accounting record (a receipt) Solutions: Traceback policy screening Distributed accounting Charge Quotes Strong authentication and encryption

15. 15

16. 16 Summary The Gold Paradigm: Allocation Bank Resource usage guided by management Dynamic Charging Enforces access controls Custom Accounting Generalized information service Distributed Accounting Preserves autonomy within a Grid

17. 17 Schedule 2Q04Gold alpha release becomes available (Java) 3Q04Second Gold alpha release based on Perl 4Q04Gold beta release becomes available 4Q05Production release of Gold

18. 18 Contact Information Scott Jackson Pacific Northwest National Laboratory Scott.Jackson@pnl.gov (509) 376-2205 Gold Home Page http://www.scidac.org/ScalableSystems SSS Resource Management Working Group http://sss.scl.ameslab.gov/home.shtml Scalable Systems Software Center http://www.scidac.org/ScalableSystems