1. SURAgrid Accounting Status September 2007

3. Accounting WG - Phase 1 Call for a working group at SURAgrid All-Hands March 07, to develop (at least) a recommendation for implementing accounting on SURAgrid Group formed with desire to be short-term but contribute a tangible immediately useful first step –Volunteers included: Victor Bolet, Steve Brandt, Mahantesh Halappanvar, Steve Johnson, Shawn McKee, Jerry Perez, John- Paul Robinson, Bin Zhou

4. Accounting WG Phase 1 Goals of initial SURAgrid accounting: –To develop and implement the capability to analyze SURAgrid usage in support of setting implementation priorities, planning, and input to proposals. –To monitor allocations, however these are prescribed by the resource owner at this time. –Create a system that can be implemented with current resources but also incorporated into, or expanded by, more complex system later on if needed - also tracking well with standards and best practices in this area.

5. Phase 1 Implementation Plan Determine set of common data to be collected from SURAgrid resources Document desired data collection in the form of a SURAgrid Usage Record –Based on job Usage Record standard currently progressing through the OGF User Record Working Group (http://forge.ggf.org/sf/projects/ur-wg).http://forge.ggf.org/sf/projects/ur-wg Enable data collection through local schedulers Store data in standard format (XML) and central database for access now, also later availability to another accounting package if one is selected.

6. Snapshot Status Phase 1 Work Some still to discuss but…SURAgrid Usage Record essentially complete! WG members piloting with all major local schedulers & documenting for others to follow Data from pilots is being stored in centralized database w/querying capability, hosted by ODU Will need to see if initial database & reporting package provide enough utility, can scale, etc. –“Accounting WG Phase 2” to address if not…

7. With Special Thanks to… This was a real working working group and extra appreciation goes to the following SURAgridr’s for producing the components you see today: –Victor Bolet –Steve Brandt –Mahantesh Halappanvar –Steve Johnson –Shawn McKee –John-Paul Robinson

8. SURAgrid Usage Record See handout…

9. Status of Local Scheduler Pilots PBS: Conversion to XML complete, script provided, incorporated into workflow LoadLeveler: Conversion to XML complete, script provided SGE: Basic data mapping defined, needs conversion to xml Condor: Conversion to XML complete, script provided, OSG Gratia project GridWay: Initial investigation suggests data available, needs code to extra from data store LSF?

10. Working Group Action Items GSU: work on Loadlever scripts with LSU ODU, TAMU, LSU: Loadlever and pbspro upload workflow TAMU: hammer out PBS to URWG conversion and will look at user mapping ODU, UAB, TAMI: will working on running 1.4.3 xindice (parallel development path with work-flow effort) UAB: SGE and Gridway acct data record generation, project page updates, move wiki to more formal home to facilitate future collaboration ALL: solidify project documentation with ideally step-by-step instructions

11. Status of User Documentation Currently in a Wiki, hosted by UAB –http://webapp.lab.ac.uab.edu/projects/jpr/wiki/SURAgri dAccountinghttp://webapp.lab.ac.uab.edu/projects/jpr/wiki/SURAgri dAccounting –Moving to UABgrid collaboration environment: http://projects.uabgrid.uab.edu/sg-accounting http://projects.uabgrid.uab.edu/sg-accounting Working Group members still adding detail to this resource Still need to determine best flow for information Eventually move to other location or format? –Can be hosted by UABgrid indefinitely

12. From PBS to Central Data Store Resource Manager: OpenPBS/PBS-Pro/Torque XML Conversion Tool: PBS-XML Accounting Toolkit XML Database: Xindice Report Generator: DarkSlide Figure 1. A Basic Flow Chart for Collecting SURAgrid Usage Records.

13. PBS to XML Site: install PBS-XML Accounting Toolkit Requires Java (1.4) and Ant Export code from CVS @ Sourceforge Build and Install RPM % ant buildrpm % rpm –ivh pbsxmlaccounting.rpm Set up user environment Run command Central: install Darkslide and Xindice

14. Status of Central Data Collection Update from Mahantesh

15. Additional considerations Need to revisit the “for further discussion” fields in the SURAgrid Usage Record Need a way to map local user to grid user (dn) On account naming conventions: Thinking at this time is to leave it to sites and work on mapping. It's up to local system to filter out non-suragrid users. Sites could leave all accounting data in place and use SURAgrid summary to generate local reports. Might be a good way to generate a community around this resource.

16. Next Steps Need longer term Accounting WG for Phase 2 –Insure and support implementation of SURAgrid Usage Record –Evolve record as necessary –Evaluate use and utility of the basic system and consider/select other options if needed –Participants: From the community: Mahantesh & Steve B. so far committed. Other volunteers? From SURA: Dali Wang, MFY if/as needed