1. Computer Sciences Department University of Wisconsin-Madison condor-admin@cs.wisc.edu http://www.cs.wisc.edu/condor Developer APIs to Condor + A Tutorial on Condors Web Service Interface

2. http://www.cs.wisc.edu/condor 2 Interfacing Applications w/ Condor Suppose you have an application which needs a lot of compute cycles You want this application to utilize a pool of machines How can this be done?

3. http://www.cs.wisc.edu/condor 3 Some Condor APIs Command Line tools condor_submit, condor_q, etc DRMAA Condor GAHP JSDL RDBMS Condor Perl Module SOAP

4. http://www.cs.wisc.edu/condor 4 Command Line Tools Dont underestimate them! Your program can create a submit file on disk and simply invoke condor_submit: system(echo universe=VANILLA > /tmp/condor.sub); system(echo executable=myprog >> /tmp/condor.sub);... system(echo queue >> /tmp/condor.sub); system(condor_submit /tmp/condor.sub);

5. http://www.cs.wisc.edu/condor 5 Command Line Tools Your program can create a submit file and give it to condor_submit through stdin: PERL:fopen(SUBMIT, |condor_submit); print SUBMIT universe=VANILLA\n;... C/C++:int s = popen(condor_submit, r+); write(s, universe=VANILLA\n, 17/*len*/);...

6. http://www.cs.wisc.edu/condor 6 Command Line Tools Using the +Attribute with condor_submit: universe = VANILLA executable = /bin/hostname output = job.out log = job.log +webuser = zmiller queue

7. http://www.cs.wisc.edu/condor 7 Command Line Tools Use -constraint and –format with condor_q: % condor_q -constraint webuser==zmiller -- Submitter: bio.cs.wisc.edu : : bio.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 213503.0 zmiller 10/11 06:00 0+00:00:00 I 0 0.0 hostname % condor_q -constraint 'webuser=="zmiller"' -format "%i\t" ClusterId -format "%s\n" Cmd 213503 /bin/hostname

8. http://www.cs.wisc.edu/condor 8 Command Line Tools condor_wait will watch a job log file and wait for a certain (or all) jobs to complete: system(condor_wait job.log); can specify a timeout

9. http://www.cs.wisc.edu/condor 9 Command Line Tools condor_q and condor_status –xml option So it is relatively simple to build on top of Condors command line tools alone, and can be accessed from many different languages (C, PERL, python, PHP, etc). However…

10. http://www.cs.wisc.edu/condor 10 DRMAA DRMAA is a GGF standardized job- submission API Has C (and now Java) bindings Is not Condor-specific -- your app could submit to any job scheduler with minimal changes (probably just linking in a different library) SourceForge Project http://sourceforge.net/projects/condor-ext

11. http://www.cs.wisc.edu/condor 11 DRMAA Easy to use, but Unfortunately, the DRMAA API does not support some very important features, such as: Two-phase commit Fault tolerance Transactions

12. http://www.cs.wisc.edu/condor 12 Condor GAHP The Condor GAHP is a relatively low-level protocol based on simple ASCII messages through stdin and stdout Supports a rich feature set including two-phase commits, transactions, and optional asynchronous notification of events Is available in Condor 6.7.X

13. http://www.cs.wisc.edu/condor 13 GAHP, cont Example: R: $GahpVersion: 1.0.0 Nov 26 2001 NCSA\ CoG\ Gahpd $ S: GRAM_PING 100 vulture.cs.wisc.edu/fork R: E S: RESULTS R: E S: COMMANDS R: S COMMANDS GRAM_JOB_CANCEL GRAM_JOB_REQUEST GRAM_JOB_SIGNAL GRAM_JOB_STATUS GRAM_PING INITIALIZE_FROM_FILE QUIT RESULTS VERSION S: VERSION R: S $GahpVersion: 1.0.0 Nov 26 2001 NCSA\ CoG\ Gahpd $ S: INITIALIZE_FROM_FILE /tmp/grid_proxy_554523.txt R: S S: GRAM_PING 100 vulture.cs.wisc.edu/fork R: S S: RESULTS R: S 0 S: RESULTS R: S 1 R: 100 0 S: QUIT R: S

14. http://www.cs.wisc.edu/condor 14 JSDL and Condor GridSAM: open source web service for job submission and monitoring Condor plugin for GridSAM enables JSDL submissions to Condor.

15. http://www.cs.wisc.edu/condor 15 RDMS: Quill Job ClassAds information mirrored into an RDBMS Both active jobs and historical jobs Benefits BOTH scalability and accessibility QuillSchedd Job Queue log RDBMS Startd … Master Queue + History Tables

16. http://www.cs.wisc.edu/condor 16 Condor Perl Module Perl module to parse the job log file Recommended instead of polling w/ condor_q Call-back event model (Note: job log can be written in XML)

17. http://www.cs.wisc.edu/condor 17 Web Service Interface Simple Object Access Protocol Mechanism for doing RPC using XML (typically over HTTP or HTTPS) A World Wide Web Consortium (W3C) standard SOAP Toolkit: Transform a WSDL to a client library

18. http://www.cs.wisc.edu/condor 18 Benefits of a Condor SOAP API Condor becomes a service Can be accessed with standard web service tools Condor accessible from platforms where its command-line tools are not supported Talk to Condor with your favorite language and SOAP toolkit

19. http://www.cs.wisc.edu/condor 19 Condor SOAP API functionality Submit jobs Retrieve job output Remove/hold/release jobs Query machine status Query job status

20. http://www.cs.wisc.edu/condor 20 Getting machine status via SOAP Your program SOAP library queryStartdAds() condor_collector Machine List SOAP over HTTP

21. http://www.cs.wisc.edu/condor 21 Lets get some details…

22. http://www.cs.wisc.edu/condor 22 The API Core API, described with WSDL, is designed to be as flexible as possible File transfer is done in chunks Transactions are explicit Wrapper libraries aim to make common tasks as simple as possible Currently in Java and C# Expose an object-oriented interface

23. http://www.cs.wisc.edu/condor 23 Things we will cover Condor setup Necessary tools Job Submission Job Querying Job Retrieval Authentication with SSL and X.509 An important addition in late 6.7

24. http://www.cs.wisc.edu/condor 24 Condor setup Start with a working condor_config The SOAP interface is off by default Turn it on by adding ENABLE_SOAP=TRUE Access to the SOAP interface is denied by default Set ALLOW_SOAP and DENY_SOAP, they work like ALLOW_READ/WRITE/… See section 3.7.4 of the v6.7 manual for a description Example: ALLOW_SOAP=*/*.cs.wisc.edu

25. http://www.cs.wisc.edu/condor 25 Necessary tools You need a SOAP toolkit Apache Axis (Java) - http://ws.apache.org/axis/ Microsoft.Net - http://microsoft.com/net/ gSOAP (C/C++) - http://gsoap2.sf.net/ ZSI (Python) - http://pywebsvcs.sf.net/ SOAP::Lite (Perl) - http://soaplite.com/ You need Condors WSDL files Find them in lib/webservice/ in your Condor release Put the two together to generate a client library $ java org.apache.axis.wsdl.WSDL2Java condorSchedd.wsdl Compile that client library $ javac condor/*.java All our examples are in Java using Apache Axis

26. http://www.cs.wisc.edu/condor 26 Helpful tools The core API has some complex spots A wrapper library is available in Java and C# Makes the API a bit easier to use (e.g. simpler file transfer & job ad submission) Makes the API more OO, no need to remember and pass around transaction ids We are going to use the Java wrapper library for our examples You can download it from http://www.cs.wisc.edu/condor/birdbath/birdbath.jar Will be included in Condor release

27. http://www.cs.wisc.edu/condor 27 Submitting a job The CLI way… universe = vanilla executable = /bin/cp arguments = cp.sub cp.worked should_transfer_files = yes transfer_input_files = cp.sub when_to_transfer_output = on_exit queue 1 $ condor_submit cp.sub cp.sub: Explicit bits clusterid = X procid = Y owner = matt requirements = Z Implicit bits

28. http://www.cs.wisc.edu/condor 28 Repeat to submit multiple jobs in a single cluster Repeat to submit multiple clusters The SOAP way… 1.Begin transaction 2.Create cluster 3.Create job 4.Send files 5.Describe job 6.Commit transaction Submitting a job

29. http://www.cs.wisc.edu/condor 29 1. Begin transaction 2. Create cluster 3. Create job 4&5. Send files & describe job 6. Commit transaction Schedd schedd = new Schedd(http://…); Transaction xact = schedd.createTransaction(); xact.begin(30); int cluster = xact.createCluster(); int job = xact.createJob(cluster); File[] files = { new File(cp.sub) }; xact.submit(cluster, job, owner, UniverseType.VANILLA, /bin/cp, cp.sub cp.worked, requirements, null, files); xact.commit(); Submission from Java

30. http://www.cs.wisc.edu/condor 30 Schedds location Max time between calls (seconds) Job owner, e.g. matt Requirements, e.g. OpSys==\Linux\ Extra attributes, e.g. Out=stdout.txt or Err=stderr.txt Schedd schedd = new Schedd(http://…); Transaction xact = schedd.createTransaction(); xact.begin(30); int cluster = xact.createCluster(); int job = xact.createJob(cluster); File[] files = { new File("cp.sub") }; xact.submit(cluster, job, owner, UniverseType.VANILLA, /bin/cp, cp.sub cp.worked, requirements, null, files); xact.commit(); Submission from Java

31. http://www.cs.wisc.edu/condor 31 Querying jobs The CLI way… $ condor_q -- Submitter: localhost : : localhost ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 1.0 matt 10/27 14:45 0+02:46:42 C 0 1.8 sleep 10000 … 42 jobs; 1 idle, 1 running, 1 held, 1 unexpanded

32. http://www.cs.wisc.edu/condor 32 Also, getJobAds given a constraint, e.g. Owner==\matt\ String[] statusName = {, Idle, Running, Removed, Completed, Held }; int cluster = 1; int job = 0; Schedd schedd = new Schedd(http://…); ClassAd ad = new ClassAd(schedd.getJobAd(cluster, job)); int status = Integer.valueOf(ad.get(JobStatus)); System.out.println(Job is + statusName[status]); Querying jobs The SOAP way from Java…

33. http://www.cs.wisc.edu/condor 33 Retrieving a job The CLI way.. Well, if you are submitting to a local Schedd, the Schedd will have all of a jobs output written back for you If you are doing remote submission you need condor_transfer_data, which takes a constraint and transfers all files in spool directories of matching jobs

34. http://www.cs.wisc.edu/condor 34 Discover available files Remote file Local file Retrieving a job The SOAP way in Java… int cluster = 1; int job = 0; Schedd schedd = new Schedd(http://…); Transaction xact = schedd.createTransaction(); xact.begin(30); FileInfo[] files = xact.listSpool(cluster, job); for (FileInfo file : files) { xact.getFile(cluster, job, file.getName(), file.getSize(), new File(file.getName())); } xact.commit();

35. http://www.cs.wisc.edu/condor 35 Authentication for SOAP Authentication is done via mutual SSL authentication Both the client and server have certificates and identify themselves Possible in late-late 6.7 (available by 6.8) It is not always necessary, e.g. in some controlled environments (a portal) where the submitting component is trusted A necessity in an open environment -- remember that the submit call takes the jobs owner as a parameter Imagine what happens if anyone can submit to a Schedd running as root…

36. http://www.cs.wisc.edu/condor 36 Authentication setup Create and sign some certificates Use OpenSSL to create a CA CA.sh -newca Create a server cert and password-less key CA.sh -newreq && CA.sh -sign mv newcert.pem server-cert.pem openssl rsa -in newreq.pem -out server-key.pem Create a client cert and key CA.sh -newreq && CA.sh -sign && mv newcert.pem client-cert.pem && mv newreq.pem client-key.pem

37. http://www.cs.wisc.edu/condor 37 Authentication config Config options… ENABLE_SOAP_SSL is FALSE by default _SOAP_SSL_PORT Set this to a different port for each SUBSYS you want to talk to over ssl, the default is a random port Example: SCHEDD_SOAP_SSL_PORT=1980 SOAP_SSL_SERVER_KEYFILE is required and has no default The file containing the servers certificate AND private key, i.e. keyfile after cat server-cert.pem server-key.pem > keyfile

38. http://www.cs.wisc.edu/condor 38 Authentication config Config options continue… SOAP_SSL_CA_FILE is required The file containing public CA certificates used in signing client certificates, e.g. demoCA/cacert.pem All options except SOAP_SSL_PORT have an optional SUBSYS_* version For instance, turn on SSL for everyone except the Collector with ENABLE_SOAP_SSL=TRUE COLLECTOR_ENABLE_SOAP_SSL=FALSE

39. http://www.cs.wisc.edu/condor 39 One last bit of config The certificates we generated have a principal name, which is not standard across many authentication mechanisms Condor maps authenticated names (here, principal names) to canonical names that are authentication method independent This is done through mapfiles, given by SEC_CANONICAL_MAPFILE and SEC_USER_MAPFILE Canonical map: SSL. *emailAddress=(.*)@cs.wisc.edu.* \1 User map: (.*) \1 SSL is the authentication method,.*emailAddress….* is a pattern to match against authenticated names, and \1 is the canonical name, in this case the username on the email in the principal

40. http://www.cs.wisc.edu/condor 40 HTTPS with Java Setup keys… keytool -import -keystore truststore -trustcacerts -file demoCA/cacert.pem openssl pkcs12 -export -inkey client-key.pem -in client- cert.pem -out keystore All the previous code stays the same, just set some properties javax.net.ssl.trustStore, javax.net.ssl.keyStore, javax.net.ssl.keyStoreType, javax.net.ssl.keyStorePassword Example: java -Djavax.net.ssl.trustStore=truststore - Djavax.net.ssl.keyStore=keystore - Djavax.net.ssl.keyStoreType=PKCS12 - Djavax.net.ssl.keyStorePassword=pass Example https://…

41. http://www.cs.wisc.edu/condor 41 Questions?