1. www.epikh.eu The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) WMPROXY API Python & C++ Diego Scardaci (diego.scardaci@ct.infn.it)diego.scardaci@ct.infn.it INFN Dept. of Catania Joint EPIKH/GISELA Application Porting School Mexico City, 15-25.11.2010

2. Outline Mexico City, (Mexico) - Joint EPiKH/GISELA School, 15-25.11.2010 2 WMProxy Python API WMProxy C++ API References

3. WMProxy Python API Mexico City, (Mexico) - Joint EPiKH/GISELA School, 15-25.11.2010 3 Requirements: glite-wms-wmproxy-api-python package In your job submission script, you have to use: from wmproxymethods import Wmproxy from wmproxymethods import JobIdStruct

4. Delegation Mexico City, (Mexico) - Joint EPiKH/GISELA School, 15-25.11.2010 4 Due to problem with the wms-proxy-delegation API, we had to issue: os.system("/opt/glite/bin/glite-wms-job- delegate-proxy --endpoint https://WMSHOSTNAME:7443/glite_wms_wmproxy_serv er -o wms_proxy --noint -d NAMEOFDELEGATIONID") We have to use the NAMEOFDELAGATIONID to submit a job Then we created a client object via: url=https://WMPSHOSTNAME:7443/glite_wms_wmproxy _server client=Wmproxy(url)

5. Job Submission (1/2) Mexico City, (Mexico) - Joint EPiKH/GISELA School, 15-25.11.2010 5 Declare your JDL: jobjdl="[ requirements = other.GlueCEUniqueID == \""+ii+"\"; RetryCount = 0; JobType = \"parametric\"; Parameters="+str(bulkjobs)+“; ParameterStart=1; ParameterStep=1; Executable=\"salute_field.sh\"; InputSandbox={\"gsiftp://se001.ipp.acad.bg:2811/ pnfs/ipp.acad.bg/seegrid/salute_field.sh\"}; OutputSandbox = {\"std.out\",\"std.err\"}; StdOutput=\"std.out\“; StdError=\"std.err\“; VirtualOrganisation = \"seegrid\“; rank = 1; ]"

6. Job Submission (2/2) Mexico City, (Mexico) - Joint EPiKH/GISELA School, 15-25.11.2010 6 Register the Job using your delegation and the jobjdl just defined: delegationId = “NAMEOFDELEGATIONID” jobId = client.jobRegister(jobjdl,delegationId) Transfer InputSandbox file to destURI (WMS or Storage Element) (e.g. using GridFTP); Start the Job using the jobId returned by the client.jobRegister function: client.jobStart(jobId)

7. Get the JobID Mexico City, (Mexico) - Joint EPiKH/GISELA School, 15-25.11.2010 7 You can obtain the jobId string as: jobIdstringform=jobId.getJobId() print jobIdstringform For parametric/collection/DAG jobs: children=jobIds.getChildren() childrenjobs=[] for i in children: childrenjobs.append(i.jobid) print i.jobid

8. Check the job status and get the output Mexico City, (Mexico) - Joint EPiKH/GISELA School, 15-25.11.2010 8 You can check the job status using the standard gLite Command. WMProxy Python API doesn’t provide a function to check the Job Status. When the Job is DONE, you can download the output files: OutputFileList = getOutputFileList(jobId) This operation returns the list of URIs where the output files created during job execution have been stored in the WMS managed space and the corresponding sizes in bytes.

9. WMProxy C++ API Mexico City, (Mexico) - Joint EPiKH/GISELA School, 15-25.11.2010 9 Requirements: glite-wms-wmproxy-api-cpp-x.y package The interface is defined in $GLITE_LOCATION/include/glite/wms/wmproxyapi /wmproxy_api.h, so this file must be included always. Other useful methods are defined in $GLITE_LOCATION/include/glite/wms/wmproxyapi /wmproxy_api_utilities.h, its inclusion is so recommended.

10. Establishing context Mexico City, (Mexico) - Joint EPiKH/GISELA School, 15-25.11.2010 10 Before any call to WM proxy is made, an instance of Config Context has to be created. The first and third parameters are also strings, and could be used to set different paths for the user proxy used to authenticate connection and for the directory of trusted certificates, but if you wish to use the default values, these can be left empty. ConfigContext cfg ("","https://gilda-wms- 01.ct.infn.it:7443/glite_wms_wmproxy_server",""); Second parameter is the WMProxy endpoint; it is suggested not to hardcode it, as shown above, but to pass it either from command line or through an environment variable.

11. User Credential Delegation Mexico City, (Mexico) - Joint EPiKH/GISELA School, 15-25.11.2010 11 string delId = "myId"; string proxy = grstGetProxyReq(delId, &cfg); // Creates a delegation identifier for proxy grstPutProxy(delId,proxy,&cfg); // binds the delegation identifier created with proxy The obtained delegation identifier can be reused for several calls to WM proxy services; The JDL to be matched against resources or to be submitted must contain some mandatory Requirements and Rank attributes; when using command line client these are added by default, but if using custom clients, as, here, you must care they are present. requirements = (other.GlueCEStateStatus=="Production"); rank = (-other.GlueCEStateEstimatedResponseTime);

12. Job List Match Mexico City, (Mexico) - Joint EPiKH/GISELA School, 15-25.11.2010 12 jobListMatch calls returns a list of Computing Elements fitting job requirements as described in the JDL provided in input. The JDL must be converted first to ClassAd format, and then parsed as a string. With such parameters properly initialized jobListMatch service can be invoked. string jdlPath=argv[argc -1]; // get the filesystem path to jdl-file from CLI jdl::Ad ad; // declare a new ClassAd ad.fromFile (jdlPath); // initialize the ClassAd with file content string jdlString (ad.toString()); // convert AD to String and save it in jdlString /* call jobListMatch */ vector > matchingCEs = jobListMatch(jdlString, delegationId, &cfg);

13. Job Submission (1/2) Mexico City, (Mexico) - Joint EPiKH/GISELA School, 15-25.11.2010 13 Job submission through API can be divided in three steps: job registration; input sandbox file(s) transfer; job start. The Job Registration is a preliminary step, where WM proxy server is contacted, the job registered and if operation succeeds, a job identifier is returned: JobIdApi jobId = jobRegister(jdlString, delegationId, &cfg); The job id is returned on a struct of type JobIdApi containing also other useful infos on the request. For example, if the jdl expresses a compound job as DAG or Collection, JobIdApi contains the Node Name and the list of job IDs contained in the request.

14. Job Submission (2/2) Mexico City, (Mexico) - Joint EPiKH/GISELA School, 15-25.11.2010 14 After job registration, it needs to be checked whether the file has Input Sandbox or not. If not, you can easily proceed by invoking jobStart method which triggers job execution from WM Proxy server. jobStart(jobId.jobid, &cfg);

15. Upload the InputSandbox Mexico City, (Mexico) - Joint EPiKH/GISELA School, 15-25.11.2010 15 This method takes as input the source paths of file, and the destination URI, and it's here provided within library libglite_wmproxytest_utils.so. if (ad->hasAttribute(jdl::JDL::INPUTSB)){ vector dest_uris; vector > to_bcopied; string errors; // get local paths to transferring file vector paths = ad->getStringValue (jdl::JDL::INPUTSB); // from JobID obtain the destination bare URI where to copy files. The //vector returned has an element for each supported protocol, used is the //second (https) dest_uris = getSandboxDestURI(jobId.jobid, &cfg); // fill in a vector with all pairs source-destination jdl::toBcopied(jdl::JDL::INPUTSB, paths, to_bcopied,dest_uris[1],""); // perform file transfer if (curlTransfer (to_bcopied, errors,&cfg) < 0) { jobCancel(jobId.jobid, &cfg); exit(-1); }

16. References Mexico City, (Mexico) - Joint EPiKH/GISELA School, 15-25.11.2010 16 WMProxy Python API: http://egee-jra1-wm.mi.infn.it/egee-jra1- wm/api_docwmproxy_python/wmproxymethods.htmlhttp://egee-jra1-wm.mi.infn.it/egee-jra1- wm/api_docwmproxy_python/wmproxymethods.html WMProxy API C++: http://egee-jra1-wm.mi.infn.it/egee-jra1- wm/api_doc/api_docwmproxy_cpp/index.htmlhttp://egee-jra1-wm.mi.infn.it/egee-jra1- wm/api_doc/api_docwmproxy_cpp/index.html WMProxy API Samples and Utilities: https://grid.ct.infn.it/twiki/pub/GILDA/WMProxyCPPAPI/ wmproxy-api-examples.tar.gz WMProxy API Wiki: http://trinity.datamat.it/projects/EGEE/wiki/wiki.php?n=W MProxyAPI.APIDocumentation

17. Mexico City, (Mexico) - Joint EPiKH/GISELA School, 15-25.11.2010 17 Thank you very much!