1. Development of portlets for special jobs: parametric, collections, workflows Mario Torrisi (mario.torrisi@ct.infn.it) National Institute of Nuclear Physics - Division of Catania Web course on the development of applications for Catania Science Gateways, 30 July 2013

2. Web course on the development of applications for Catania Science Gateways2 Outline ● Job Engine – A short recap on the Job Engine seen in the last webinar ● Introduction on Special Jobs – Job Collection – Parametric Job – Workflow N-1 ● Manage Special Jobs – Grid Engine API to execute Special Job on DCIs ● Portlet template – Get code from SVN – Compile and test – Customize ● Summary and conclusion

3. Web course on the development of applications for Catania Science Gateways3 Catania Grid & Cloud Engine

4. Web course on the development of applications for Catania Science Gateways4 Job Engine ● The Job Engine is made of a set of libraries to develop applications able to submit and manage jobs on a grid infrastructure; ● It is compliant with the OGF SAGA standard; – JSAGA is the SAGA implementation adopted ● It is optimized to be used in a Web Portal running an application server (e.g. Glassfish, Tomcat,…) based on J2EE; ● It can be used also in stand-alone mode;

5. Web course on the development of applications for Catania Science Gateways5 Job Engine Easiness ● Allows to develop applications able to submit jobs on a DCI in a very short time; ● Exposes a set of very intuitive APIs; ● Provides support for MPI applications; ● The developer has only to submit the job: – The Job Engine periodically check the job status; – When the Job is done, the job output is automatically downloaded and (if set) an email is sent to notify the the user.

6. Web course on the development of applications for Catania Science Gateways6 Job Engine Features ● Performance – The delays due to grid interactions are hidden to the final users: ● the Job Engine provide asynchronous functions for each job management actions. ● Accounting – A powerful accounting system fully compliant with EGI VO Portal Policy and EGI Grid Security Traceability and Logging Policy. ● Fault tollerance – Job Engine implements an advanced mechanism to ensure job submission; – Automatic re-submission mechanism when a job is aborted.

7. Web course on the development of applications for Catania Science Gateways7 Introduction on Special Jobs Job Collection ● Job Collection: is a set of independent jobs that run in parallel ● When all jobs are successfully completed the whole collection becomes DONE Job Collection Job 1Job 2Job N...

8. Web course on the development of applications for Catania Science Gateways8 Introduction on Special Jobs Parametric Job ● Parametric Job: is a set of jobs that run in parallel having the same executable, but with different arguments (e.g., input parameters) ● When all jobs are successfully completed the whole parametric job becomes DONE Parametric Job Job 1Job 2Job N...

9. Web course on the development of applications for Catania Science Gateways9 Introduction on Special Jobs Workflow N-1 ● Workflow N-1 is a Special Job consisting of two different levels of jobs: – First level jobs is a set of independent jobs that run in parallel – Final (Collector) job collects all first level jobs' outputs and generates a file output ● When all first level jobs are successfully completed the final job is submitted and when it becomes DONE the whole workflow has considered completed Workflow N-1 Job 1Job 2Job N... Final Job

10. Web course on the development of applications for Catania Science Gateways10 Manage Special Job ● Parametrics job and Workflows N-1 are designed as generalization of Job Collection

11. Web course on the development of applications for Catania Science Gateways11 JobCollection.java ● This class handles the behaviour of the Job Collection. ● It is responsible for: – creation; – status updating; – closing of the collection when all jobs are DONE.

12. Web course on the development of applications for Catania Science Gateways12 JobParametric.java ● This class handles the behavior of the Parametric Job. ● Inherits all attributes and methods defined in its parent class. ● Adds a new attribute that specify the single executable.

13. Web course on the development of applications for Catania Science Gateways13 WorkflowN1.java ● This class handles the behavior of the Workflow N-1. ● Inherits all attributes and methods defined in its parent class ● Adds 3 new attributes: ● Overrides some JobCollection methods

14. Web course on the development of applications for Catania Science Gateways14 Example InfrastructureInfo infrastructures[] = new InfrastructureInfo[2]; String wmsList[]={"wms://wmsdecide.dir.garr.it:7443/glite_wms_wmproxy_server"}; String wmsList2[]={"wms://wms.magrid.ma:7443/glite_wms_wmproxy_server"}; infrastructures[0] = new InfrastructureInfo("gridit", "ldap://gridit-bdii-01.cnaf.infn.it:2170", wmsList, "etokenserver.ct.infn.it", "8082", "332576f78a4fe70a52048043e90cd11f", "gridit", "gridit"); infrastructures[1] = new InfrastructureInfo("EUMED","ldap://bdii.eumedgrid.eu:2170", wmsList2, "etokenserver.ct.infn.it", "8082", "bc681e2bd4c3ace2a4c54907ea0c379b", "eumed", "eumed"); 1. Define Infrastructures

15. Web course on the development of applications for Catania Science Gateways15 Example ArrayList descriptions = new ArrayList (); for (int i = 0; i < 2; i++) { GEJobDescription description = new GEJobDescription(); description.setExecutable("/bin/sh"); switch (i) { case 0: description.setArguments("hostname.sh"); description.setInputFiles("./hostname.sh"); break; case 1: description.setArguments("ls.sh"); description.setInputFiles("./ls.sh"); break; } description.setOutputPath("/tmp"); description.setOutput("myOutput-" + i + ".txt"); description.setError("myError-" + i + ".txt"); descriptions.add(description); } 2. Define Descriptions

16. Web course on the development of applications for Catania Science Gateways16 Example Job Collection submission JobCollection collection = new JobCollection("mtorrisi", "Collection - TEST - ", "/tmp", descriptions); JobCollectionSubmission tmpJobCollectionSubmission = new JobCollectionSubmission("jdbc:mysql://localhost/userstracking", "tracking_user", "usertracking", collection); tmpJobCollectionSubmission.submitJobCollection(infrastructures, "193.206.208.183:8162", 10); 3. Create JobCollection object 4. Submit Job Collection

17. Web course on the development of applications for Catania Science Gateways17 Example Parametric Job Submission JobCollection p = new JobParametric("test", "Parametric Job - mtorrisi", "mario.torrisi@ct.infn.it", "/tmp", descriptions, "hostname.sh"); JobCollectionSubmission tmpJobCollectionSubmission = new JobCollectionSubmission( "jdbc:mysql://localhost/userstracking", "tracking_user", "usertracking", p); tmpJobCollectionSubmission.submitJobCollection(infrastructures, "193.206.208.183:8162", 10); 3. Create JobParametric object 4. Submit Parametric Job *The first two steps are common to the "Job Collection" submission

18. Web course on the development of applications for Catania Science Gateways18 Example Workflow N-1 Submission (1/2) 3. Create final job description GEJobDescription finalJobDescription = new GEJobDescription(); finalJobDescription.setExecutable("ls.sh"); finalJobDescription.setArguments("./ls.sh"); String tmp = ""; for (int i = 0; i < descriptions.size(); i++) { if (tmp.equals("")) tmp = descriptions.get(i).getOutput(); else tmp += "," + descriptions.get(i).getOutput(); } finalJobDescription.setInputFiles(tmp + ",./ls.sh"); finalJobDescription.setOutputPath("/tmp"); finalJobDescription.setOutput("myOutput-FinalJob.txt"); finalJobDescription.setError("myError-FinalJob.txt"); *The first two steps are common to the "Job Collection" submission

19. Web course on the development of applications for Catania Science Gateways19 Example Workflow N-1 Submission (2/2) JobCollection wf = new WorkflowN1("mtorrisi", "Workflow N-1", "/tmp", descriptions, finalJobDescription); JobCollectionSubmission tmpJobCollectionSubmission = new JobCollectionSubmission( "jdbc:mysql://localhost/userstracking", "tracking_user", "usertracking", wf); tmpJobCollectionSubmission.submitJobCollection(infrastructures, "193.206.208.183:8162", 10); 4. Create WorkFlowN1 object 5. Submit Workflow N-1

20. Web course on the development of applications for Catania Science Gateways20 Special Job Portlet

21. Web course on the development of applications for Catania Science Gateways21 Outline ● Introductory consideration on mi- parallel-app-portlet ● Preliminary operations ● Get from SVN and deploy ● Test – Special job execution – Log extraction – MyJobs ● Customize

22. Web course on the development of applications for Catania Science Gateways22 Preliminary operations ● Before testing mi-parallel-app-portlet – Install MyJobs portlet – Check the eTokenServer service is reachableeTokenServer – Check the system date is aligned with current time – Check that Grid CA certificates are updated Update CA certificates – Follow instructions to fix GridEngine 1.5.1 ● Refer to presentation seen in the last webinar to perform the above mentioned preliminary operations

23. Web course on the development of applications for Catania Science Gateways23 Preliminary operations GridOperations table ● Insert a new line into the GridOperations table:

24. Web course on the development of applications for Catania Science Gateways24 mi-parallel-app-portlet Get from SVN and deploy ● Get code form SVN ● Run deploy command ● Watch server.log until see something like: ● Install portlet in the portal through Liferay menu: svn checkout svn://svn.code.sf.net/p/ctsciencegtwys/liferay/trunk/gilda/mi-parallel-app-portlet mi- parallel-app-portlet ant deploy [AutoDeploy] Successfully autodeployed : /home/mario/liferay-portal-6.1.1-ce-ga2/glassfish- 3.1.2/domains/domain1/autodeploy/mi-parallel-app-portlet.

25. Web course on the development of applications for Catania Science Gateways25 mi-parallel-app-portlet Test 1. Select collection type 2. Insert task number 3. Click “OK” button 4. Fill input fields 5. Insert a brief collection description 6. Submit collection

26. Web course on the development of applications for Catania Science Gateways26 Special job execution Log extraction(1/2) ● Check GridEnigne.log file under /logs folder – Jobs Submission 2013-07-26 22:35:57,270 [submitJob] INFO JSagaJobSubmission - Job Submitted: [wms://wmsdecide.dir.garr.it:7443/glite_wms_wmproxy_server]- [https://wmsdecide.dir.garr.it:9000/n_tn5247FcS3Cas4v3gfxA] […] 2013-07-26 22:36:01,046 [submitJob] INFO JSagaJobSubmission - Job Submitted: [wms://wms024.cnaf.infn.it:7443/glite_wms_wmproxy_server]- [https://wms024.cnaf.infn.it:9000/fnu6HvRxj71vmfyufqaslQ] […] 2013-07-26 22:36:04,531 [submitJob] INFO JSagaJobSubmission - Job Submitted: [wms://wms024.cnaf.infn.it:7443/glite_wms_wmproxy_server]- [https://wms024.cnaf.infn.it:9000/7lXUSv5n1em0CcrZlvQS1A] – Check job status thread: updates jobs status 2013-07-26 22:37:04,205 [checkJobStatus] INFO UsersTrackingDBInterface - Status for job [wms://wmsdecide.dir.garr.it:7443/glite_wms_wmproxy_server]- [https://wmsdecide.dir.garr.it:9000/n_tn5247FcS3Cas4v3gfxA] is RUNNING [...] 2013-07-26 22:37:04,934 [checkJobStatus] INFO UsersTrackingDBInterface - Status for job [wms://wms024.cnaf.infn.it:7443/glite_wms_wmproxy_server]- [https://wms024.cnaf.infn.it:9000/7lXUSv5n1em0CcrZlvQS1A] is RUNNING [...] 2013-07-26 22:37:05,060 [checkJobStatus] INFO UsersTrackingDBInterface - Status for job [wms://wms024.cnaf.infn.it:7443/glite_wms_wmproxy_server][https://wms024.cnaf.infn.it:9000/fnu6HvRxj71vmfyufqa slQ] is RUNNING

27. Web course on the development of applications for Catania Science Gateways27 Special job execution Log extraction(2/2) – Check job status thread: updates collection status 2013-07-26 22:37:05,697 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - Running collections: 1 2013-07-26 22:37:05,697 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - Running collection Status: RUNNING 2013-07-26 22:37:05,719 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - NEW Running collection Status: RUNNING – Complete collection 2013-07-26 22:40:04,258 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - Running collection Status: RUNNING 2013-07-26 22:40:04,300 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - NEW Running collection Status: DONE 2013-07-26 22:40:04,300 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - Closing Done Collection: JobCollection [id=67, description=Demo Collection: 27/7/2013 - 0:35:33, commonName=test, taskCounter=3] 2013-07-26 22:40:04,301 [pool-31-thread-1] INFO JobCollection - creating a tgz archive containing output files... [...] 2013-07-26 22:40:04,301 [pool-31-thread-1] INFO JobCollection - tar czvf /tmp/jobOutput/DemoCollection277201303533_67.tgz --directory=/tmp/jobOutput/ DemoCollection277201303533_67

28. Web course on the development of applications for Catania Science Gateways28 MyJobs Collection status Jobs current status Collection DONE

29. Web course on the development of applications for Catania Science Gateways29 Customize ● Customize.sh is a BASH script to clone mi-parallel-app-portlet ● Make a copy of mi-parallel-app-portlet in _portlet – cp -R mi-parallel-app-portlet _portlet ● Move into _portlet folder and edit customize.sh – AUTH_EMAIL= your@email – AUTH_NAME=' ' – AUTH_INSTITUTE=' ' – APP_OPERATIONID= – APP-NAME= ● Run customize script –./cusotmize.sh

30. Web course on the development of applications for Catania Science Gateways30 Summary and conclusion ● Catania Grid & Cloud Engine provides powerful APIs to port applications on e-Infrastructure easily ● Developers do not need to care of the differences between the middleware details ● Developers have to only prepare the job(s) description(s) and then to submit it(them) ● The JobEngine: – Periodically checks the status – Automatically downloads the outputs and notify the user ● Job Engine provides also powerful mechanisms to guarantee jobs submission (or re-submission), if they fail due infrastructure issues

31. Web course on the development of applications for Catania Science Gateways31 References ● Catania Science Gateway Framework http://www.catania-science-gateways.it/ http://www.catania-science-gateways.it/ ● Grid & Cloud Engine Javadoc http://grid.ct.infn.it/webinar-liferay/ http://grid.ct.infn.it/webinar-liferay/ ● Simple API for Grid Applications (SAGA) http://www.ogf.org/documents/GFD.90.pdf http://www.ogf.org/documents/GFD.90.pdf ● JSaga: Java implementation of SAGA specification http://grid.in2p3.fr/jsaga/ ● Training material https://gilda.ct.infn.it/wikimain

32. Web course on the development of applications for Catania Science Gateways32 Thank you!