1. Open Science Grid: More compute power Alan De Smet chtc@cs.wisc.edu

2. chtc.cs.wisc.edu (CPU days each day averaged over one month) CHTC Cores In Use 1,500

3. chtc.cs.wisc.edu (CPU days each day averaged over one month) OSG Cores In Use 60,000

4. chtc.cs.wisc.edu Open Science Grid

5. chtc.cs.wisc.edu CHTC and OSG usage (CPU days each day)

6. chtc.cs.wisc.edu Challenges Solved We worry about all of this. You don’t have to. › Authentication  X.509 certificates, certificate authorities, VOMS › Interface  Globus, GridFTP, Grid universe › Validation  Linux distribution, glibc version, basic libraries

7. chtc.cs.wisc.edu Using OSG › Before universe = vanilla executable = myjob log = myjob.log queue

8. chtc.cs.wisc.edu Using OSG › After universe = vanilla executable = myjob log = myjob.log +WantGlidein = true queue

9. chtc.cs.wisc.edu Challenge: Opportunistic › OSG computers go away without notice › Solutions  Condor restarts automatically  Sub-hour jobs  Self-checkpointing  Automated checkpointing Condor’s standard universe DMTCP http://dmtcp.sourceforge.net/

10. chtc.cs.wisc.edu Challenge: Local Software

11. chtc.cs.wisc.edu Challenge: Local Software › Bare-bones Linux systems › Solution  Bring everything with you  CHTC provided MATLAB and R packages RunDagEnv/mkdag

12. chtc.cs.wisc.edu Challenge: Erratic Failures › Complex systems fail sometimes › Solution  Expect failures and automatically retry  DAGMan for retries  DAGMan POST scripts to detect problems RunDagEnv/mkdag

13. chtc.cs.wisc.edu Challenge: Bandwidth › Solutions  Only send what you need  Store large, shared files in our web cache  Read small amounts of data on the fly Condor’s standard universe Parrot http://www.cse.nd.edu/~ccl/software/parrot/