1. Grid Interoperability Update on GridFTP tests Gregor von Laszewski gregor@mcs.anl.gov http://www.cogkit.org

2. Outline Proposal: Documentation consolidation Proposal: GIN GridFTP Report –Proposal for Survey –Proposal for User Identification –Proposal for identification of what tests to execute Update GridFTP –Java CoG Kit Workflow (Testing) Framework –Testing GridFTP on the TeraGrid –Testing GridFTP on the GIN Resources Summary

3. Proposal Documentation: Making Interoperability possible One aspect of interoperability is based on an effective mechanism to document and distribute the information. Proposal: Consolidate Documentation –Today: information is spread across several web sites and Wikis –Build a single entry point for the working group documentation Proposal: Improve the contents –Work on extended report, publishable article –Most important from my point of view: Improve the documentation and description on how to get accounts on the other “Grids” Proposal: Test documentation effort –Have interoperability tests executed by third parties to test effectiveness of documentation and approaches –Engage in an iterative improvement process of the documentation

4. GIN Today: Documentation Documentation links (We need your help to add to this page): –This should be done during the workshop or as response to the mail –http://wiki.nesc.ac.uk/read/gin-jobs?HomePagehttp://wiki.nesc.ac.uk/read/gin-jobs?HomePage –https://forge.gridforum.org/sf/wiki/do/viewPage/projects.gin/wiki/GINDatahttps://forge.gridforum.org/sf/wiki/do/viewPage/projects.gin/wiki/GINData Problem with OGF/GGF infrastructure –Gridforge is not used (is it too complicated? Is it too much overhead) Only charter is published –Wikis are the tool of choice for the working group as much of the information may change quickly. Proposal: use a single Wiki If wiki is used a feature is needed to archive the Wiki contents in a single document. –OGF schedules do not have pointers to the working group –Google search in OGF Web page is ineffective searching for GIN must give as first link the working group page –Search in Gridforge is non operational Example: search for GIN in wikis on Gridforge returns no results –Links in the mailing list archives are no longer operational Example links to documentation that was send by Erwin in May 06 –Transition to OGF has not been completed.

5. Proposal: GIN GridFTP Report Goal: provide an extended document that includes a formal way of reporting on the activities of this particular subtask as part of the GIN working group in a single report. Assure that the goals overlap with the working group goals Proposed Features that the report may address (in the area of GridFTP) –Survey of different implementations of GridFTP servers and clients –Identification of the users for interoperability tests and their goals Administrators Developers Users –Survey of different tools that monitor GridFTP behavior in production Grids –Identification of which tests need to be performed to satisfy Administrators Developers Users –Links to programs and documentation that can perform the tests.

6. Administrators - Developers - Users There are different dimensions to be considered when addressing interoperability issues –While administrators may care about setting up tests between virtual organizations, individual user needs are not completely addressed by this test as they are not conducted by the user itself. –Lessons learned over the past 5 years have shown that users MUST have the ability to perform a set of interoperability test themselves. This includes in most case the answer to the questions: Can I copy a file form here to there? What is the performance I get when I do the default? … –Administrator may be more interested to obtain the performance and throughput of the service. Test conducted for each individual user are typically not conducted. –Developers may be more concerned about functional or protocol interoperability. –GIN must address not only administrators and developers, but more importantly the users.

7. Update GridFTP: Testing with the Java CoG Kit Workflow Java CoG Kit Workflow –Java CoG Kit workflow can be run through a remote service, an API, or a command line tool. –Java CoG Kit contains easy to use workflow language that in the past has been able to bridge between different Globus middleware (GT2, GT3, GT4, SSH, …) –Language features include Functional programming language expressible in XML like syntax Variable Futures Explicit parallelism HTML form generation (XUL like) Two concepts: –Abstractions: copy a file from here to there regardless of the implementation / protocol –Providers: Mapping into Grid middleware or commodity technologies

8. … cont. Java CoG Kit workflow <execute executable="/bin/date" stdout="thedate" host="hot.mcs.anl.gov" provider="GT2"/> <transfer srchost="hot.mcs.anl.gov" srcfile="thedate" desthost="localhost" provider="gridftp"/> It is obvious that this can be used to test not only the file transfers but also job submissions and other things. See Also: –http://wiki.cogkit.org/index.php/Java_CoG_Kit_Documentationhttp://wiki.cogkit.org/index.php/Java_CoG_Kit_Documentation Transfers are real simple, specify source, destination, and provider Technically we could add other providers such as nordu-GridFTP, RFT, SRM, …

9. What could be tested File transfer –Directory creation –Permissions –2nd party transfers –3rd party transfers –… –All the features that are available through the Java Cog Kit abstractions –We can integrate more features if necessary Job execution –Similar … Other things –… The CoG workflow and its embedded HTML capabilities make it possible to create custom tests.

10. What did we test? First we wanted a simple test –Transfer a file form here to there –Perform the test by someone outside of the CoG Kit group to identify if the tool is Easy to install Functional Robust –Compare if the tests are similar to the once obtained from Globus native GridFTP clients Result: –In all aspects successful –Through a single shell script the testing framework can be installed and run –The results have been confirmed with Raj, who is a member of the GridFTP development team. To Do: –We moved to an SVN archive this month and need to update that information so others can replicate it if they wish)

11. GridFTPTest.sh Installing and Running –1. Requirements: Apache Ant and a Java SDK –2. Mkdir gin –3. Get cog-ftptest-install.sh script from Java CoG Kit SVN and place in gin –4. Run cog-ftptest-install.sh –5. Edit the hosts file and customize the set of sites to include all hosts you like to test host(" ” service(provider="gsiftp", type="file", url="gsiftp:// [: ]") ) –6. Run the tests: cog-workflow runtests.k – 7. The output files should appear in the "output" directory Note: We just moved to an SVN directory and some issue with sourceforge SVN causes a problem upon download. Sourcforge is working with us to resolve it. In future we intend to host this in dev.globus.org

12. GridFTP tests on on TeraGrid (Conducted by Raj K.) Verification –Download –Install –Configure –Run –No problems –Tool works as promised –Raj is a GridFTP developer and confirmed he gets the same results with the GridFTP tools

13. GridFTP test on GIN resources (Conducted by Raj K.) Just changing the hosts to the GIN resources results in Most Issues are related to the user not being able to authenticate/authorize

14. Next Steps Complete the documentation of the tool Complete the SVN transition Have a user apply to all GIN Grid resources and make sure we do not have authentication issues. –I thought this is trivial, but in fact it is not. –Even on TG this is still a challenge today as it is not as simple as just applying for an account.

15. Summary: GIN gridFTP from a users perspective –It is possible to use existing tools such as the CoG Kit to write interoperability tests that can be executed by the user. –Tools such as the Java CoG Kit even help briding interoperability issues –The issue is at this time not interoperability of gridftp servers, we can simply use the appropriate client. The Java CoG kit has demonstrated that multi- client abstractions are a feasible approach. Hence through a higher level of abstraction we can provide a considerable notion of interoperability. –Surprisingly we found that the today’s real issue is that a user has to spend days/weeks to get onto the Grid due to post account bootstraping of the users environment to use the Grid. –Proposal: We should as community make it possible that users can get on the Grid “fast” If it takes a user more than one day to get access to the about 30 Grid resources we tested with this effort need to do something about it.

16. Testing GridFTP on the TeraGrid Testing GridFTP on the GIN Resources

17. Summary