1. 1 GridNexus © 2005 Clayton S. Ferner GridNexus UNCW GUI for Workflow Management www.gridnexus.org Clayton Ferner cferner@uncw.edu

2. 2 GridNexus © 2005 Clayton S. Ferner Outline ● Basics – Getting Started – Simple workflows – A little bit about JXPL – Libraries – Composites – Ports – User Library – Splitting Relations ● Examples – Prog – Cond – Defun – Recursion – Iteration ● Services – Web Services – Grid Services

3. 3 GridNexus © 2005 Clayton S. Ferner What is GridNexus ● GridNexus is a GUI that allows the user to create/edit/run workflows ● Based on Ptolemy II http://ptolemy.eecs.berkeley.edu/ptolemyII. Ptolemy provides the GUI and workflow features. We have extended it to provide the functionality we want (JXPL and GridServices) http://ptolemy.eecs.berkeley.edu/ptolemyII ● Release 1.02 download available www.gridnexus.org www.gridnexus.org

4. 4 GridNexus © 2005 Clayton S. Ferner Getting Started ● Download either tar/gzip or zip file from www.gridnexus.org www.gridnexus.org ● Follow instructions for installation ● Run GridNexus ● Choose File->New - >Workflow

5. 5 GridNexus © 2005 Clayton S. Ferner Getting Started ● The main frame on the right is the palette for building workflows ● The upper left frame provides the library of modules ● The lower left frame is a thumbnail of the entire workflow

6. 6 GridNexus © 2005 Clayton S. Ferner The Basics ● Sources produce data without needing input ● Sinks consume data but may have side effects (such as displaying results) ● All workflows must start with sources and end with sinks

7. 7 GridNexus © 2005 Clayton S. Ferner Simple Example 1 ● Click and drag the “Const” source to the workflow. ● Click and drag the “JxplDisplay” sink to the workflow

8. 8 GridNexus © 2005 Clayton S. Ferner Simple Example 1 ● Double-click on the Const module ● Change its value to 10 ● Click commit ● The new value is shown on the icon

9. 9 GridNexus © 2005 Clayton S. Ferner Simple Example 1 ● Input ports are on the left-hand side and output ports are on the right-hand side of each module ● Click and drag from the output port of the Const module to the JxplDisplay

10. 10 GridNexus © 2005 Clayton S. Ferner Simple Example 1 ● A link (or relation) is created between the two modules ● The output of Const is consumed by the JxplDisplay

11. 11 GridNexus © 2005 Clayton S. Ferner Simple Example 1 ● Click on the run button ( ) ● The JxplDisplay evaluates the input and produces a display window to show the results. ● Notice the output is in XML (actually JXPL)

12. 12 GridNexus © 2005 Clayton S. Ferner Simple Example 2 ● Transformers are modules that take input, transform it, and produce new output ● This example computes the express: (23 + 6) x -2

13. 13 GridNexus © 2005 Clayton S. Ferner Simple Example 2 ● The Multiplication module takes the result of the addition (its first input) and multiplies that by -2 (its second input) ● The result is consumed by JxplDisplay

14. 14 GridNexus © 2005 Clayton S. Ferner What's Going On? ● The workflow is not actually performing the operations. Instead it is creating a script (JXPL) that, when executed, produces the result ● The JxplDisplay is evaluating the script and displaying the results

15. 15 GridNexus © 2005 Clayton S. Ferner What's Going On? ● Double-click on the JxplDisplay and deselect the “Evaluate Jxpl” parameter ● This parameter tells JxplDisplay whether or not to evaluate the script that is generated

16. 16 GridNexus © 2005 Clayton S. Ferner What's Going On? ● Now when we run it, we see the actual script that is produced by the workflow ● The script is written in XML using a language developed at UNCW called JXPL

17. 17 GridNexus © 2005 Clayton S. Ferner A Little Bit about JXPL ● JXPL is based on LISP ● The corresponding LISP to the JXPL on the right looks like: (* (+ (23 6) -2)

18. 18 GridNexus © 2005 Clayton S. Ferner A Little Bit about JXPL ● Why? – XML is used to transport data between web/grid services – XML opening/closing tags LISP opening/closing parens – Everything is either an atom or a list (functions, Data Structures)

19. 19 GridNexus © 2005 Clayton S. Ferner A Little Bit about JXPL ● The transformer library is LISP-centric ● More on JXPL later (Dr. Brown) ● See www.gridnexus.orgwww.gridnexus.org

20. 20 GridNexus © 2005 Clayton S. Ferner Transformer Library

21. 21 GridNexus © 2005 Clayton S. Ferner Sources ● Const – Integers and Decimals ● Decimal – Integers and Decimals ● Symbol – Variables (use quotes 1 ) ● Rational – Fractions (use quotes 1 ) 1 Quotes are needed to prevent Ptolemy from evaluating the expressions before the JXPL evaluator gets a chance to see them.

22. 22 GridNexus © 2005 Clayton S. Ferner Examples ● Choose File->Open and select the Examples folder ● There are many examples that don't need explanation. Just give 'em a try! ● Others will be discussed later

23. 23 GridNexus © 2005 Clayton S. Ferner Composites ● A composite is a means to abstract a workflow into a single module ● From the “utilities” folder, drag the “Composite actor” to the workflow ● Right-click->Look Inside

24. 24 GridNexus © 2005 Clayton S. Ferner Composites ● This opens a new window in which we create the workflow that defines the new module ● First we need ports – Single ports take or produce single values – Multi ports take or produce multiple values

25. 25 GridNexus © 2005 Clayton S. Ferner Composites ● Toolbar icons – Single input port – Single output port – Multi input port – Multi output port ● Notice how the ports show up on the composite actor module

26. 26 GridNexus © 2005 Clayton S. Ferner Composites ● Right-click on a port and choose “Customize Name” ● Enter name for the Icon ● Selecting “Show name” causes the label to show up on the composite module

27. 27 GridNexus © 2005 Clayton S. Ferner Composites ● To make a relation from a port, hold down the shift key and click and drag ● Since a multiport can produce more than one value, we need to specify a width for the relation involving a multiport

28. 28 GridNexus © 2005 Clayton S. Ferner Composites ● For a multiport, double-click on the relation and choose Add ● The name should be “width” with a default of zero

29. 29 GridNexus © 2005 Clayton S. Ferner Composites ● Close the composite window ● We can name the composite by right- clicking and choosing “Customize Name” ● Now we are ready to use it

30. 30 GridNexus © 2005 Clayton S. Ferner User Library ● To save a module in the user library, right- click on it and choose Save Actor in Library ● The user library is stored in.GridNexus/user\ library.xml in your home directory

31. 31 GridNexus © 2005 Clayton S. Ferner User Library ● Actually, it isn't saved. It opens the user library with the new module ● You still have to choose File->Save to save the user library.

32. 32 GridNexus © 2005 Clayton S. Ferner User Library ● Alternatively, you can save an entire workflow in the library ● Choose Graph->Save in Library. ● The workflow is saved in the library as a new composite named with the filename

33. 33 GridNexus © 2005 Clayton S. Ferner Splitting Relations ● Sometime, you want the results of a module to be used as input to more than one other modules ● SetQ module assigns a value to a symbol ● The symbol X is input to both SetQ and Addition modules file: examples/setq.xml

34. 34 GridNexus © 2005 Clayton S. Ferner Splitting Relations ● To create a new splitter, click the button ● Link the output of Symbol to the splitter ● Hold down the shift key and click and drag the splitter to the input port of SetQ. file: examples/setq.xml

35. 35 GridNexus © 2005 Clayton S. Ferner Splitting Relations ● Repeat for Addition ● (Sometimes, the use of a splitter can complicate the workflow due to crossing relations. It is not necessary to split the output from Symbol “x”. Alternatively, we can simply make a duplicate Symbol “x”.) file: examples/setq.xml

36. 36 GridNexus © 2005 Clayton S. Ferner The Prog ● The Prog (short for Program) module allows one to execute separate workflows where it is not desirable to have a relationship between them file: examples/prog.xml

37. 37 GridNexus © 2005 Clayton S. Ferner The Prog ● The Prog primitive evaluates each of its inputs in turn ● It returns the results of the last one evaluated (in this case this case it returns 2x). file: examples/prog.xml

38. 38 GridNexus © 2005 Clayton S. Ferner Multi Ports ● One difficulty with multi ports is the order in which its inputs are evaluated is not evident from the GUI ● Which of the sub- workflows is executed first? Are you sure? file: examples/prog.xml

39. 39 GridNexus © 2005 Clayton S. Ferner Multi Ports ● If one deletes the SetQ -> Prog relation, then puts it back, the result is that Prog will evaluation the Multiplication before the SetQ – Not good! ● Multi Ports are best for commutative operations like Addition, but not like Prog. ● However, the alternative to using a multi port for Prog is to have a fixed number of single ports. ● When it doubt, delete all relations to a multi port and redo them

40. 40 GridNexus © 2005 Clayton S. Ferner Cond ● Cond is the branching mechanism in LISP ● A Cond List Element takes a boolean expression and the results if the boolean is true file: examples/cond.xml

41. 41 GridNexus © 2005 Clayton S. Ferner Cond ● The Cond finds the first Cond List Element whose boolean is true and returns its result ● Be careful of the order in which the Elements are linked to the multi- input port file: examples/cond.xml

42. 42 GridNexus © 2005 Clayton S. Ferner Cond ● An “else” clause should the last List Element with a boolean of “true” ● The last List Element will be returned if all others are false file: examples/cond.xml

43. 43 GridNexus © 2005 Clayton S. Ferner Defun ● Defun defines a new primitive (function) ● It takes a symbol as the function name, a set of symbols as parameters, and a workflow as the body file: examples/defun.xml

44. 44 GridNexus © 2005 Clayton S. Ferner Defun ● Once it is defined, we can call it using the Primitive module file: examples/defun.xml

45. 45 GridNexus © 2005 Clayton S. Ferner Recursion ● Of course, we have to do factorial! ● This workflow first defines factorial (using a composite to keep the workflow clean) then uses it to compute 10!. file: examples/factorial.xml

46. 46 GridNexus © 2005 Clayton S. Ferner Recursion ● Inside Factorial: – If N <= 1 return 1 – Else return N(N-1)! file: examples/factorial.xml

47. 47 GridNexus © 2005 Clayton S. Ferner Recursion ● Inside N(N-1)!: – Notice the recursive call to factorial – Remember that this generates a script that will be executed later file: examples/factorial.xml

48. 48 GridNexus © 2005 Clayton S. Ferner Iteration ● Arguments: – loop variable – initial value – step size (loop variable += step size) – termination boolean (true => terminate) – loop body file: examples/do.xml

49. 49 GridNexus © 2005 Clayton S. Ferner Web Services ● The WS Client module can serve as a general purpose client for a Web Service given the WSDL ● Double-click on the module and enter the URL of the WSDL

50. 50 GridNexus © 2005 Clayton S. Ferner Web Services ● The icon is set with the name of the service ● Ports are added for the methods of the service ● Only the method with input will be called file: examples/wsclient.xml

51. 51 GridNexus © 2005 Clayton S. Ferner Grid Services ● First, you need the jars of the Grid Service. E.g.: edu_uncw_math_mathservice.jar edu_uncw_math_mathservice_stubs.jar ● Put them in $GRIDNEXUS_HOME/lib (Linux) or C:\Program Files\GridNexus1.xx\lib (Windows) ● You will need to restart GridNexus so that it will reset the classpath

52. 52 GridNexus © 2005 Clayton S. Ferner Grid Services ● The WSRF Client module can serve as a general purpose client for a Grid Service given the following: – URL of the Factory – Class name of the Addressing Locator class

53. 53 GridNexus © 2005 Clayton S. Ferner Grid Services ● The icon is set with the name of the service ● Ports are added for the methods of the service ● An EndpointReference (epr) can be provided, which will override the Factory URL

54. 54 GridNexus © 2005 Clayton S. Ferner Grid Services ● In order to call a method that takes no parameters, pass a value of “false” ● For example, the create method takes no parameters ● The create method creates a service instance and returns the epr file: examples/wsrf_client/ MathService2Create.xml

55. 55 GridNexus © 2005 Clayton S. Ferner Grid Services ● What does this workflow do?

56. 56 GridNexus © 2005 Clayton S. Ferner Grid Services ● The destroy method also does not take a parameter ● After the destroy, the service instance no longer exists and the epr file is no longer of use file: examples/wsrf_client/ MathService2Destroy.xml

57. 57 GridNexus © 2005 Clayton S. Ferner User Library ● Don't forget to save these actors in the user library now that they are configured ● Just click and drag these to the palette and they're ready for use

58. 58 GridNexus © 2005 Clayton S. Ferner GridExec (GRAM Client) ● The GridExec module will run an application on a service using GRAM ● The Factory Contact is the URL of the server ● If a Simple command is given, GridExec will construct an RSL file to run it file: examples/job_submission/ simplejob.xml /bin/touch /tmp/test.beowulf ${GLOBUS_USER_HOME}/stdout ${GLOBUS_USER_HOME}/stderr

59. 59 GridNexus © 2005 Clayton S. Ferner GridExec (GRAM Client) ● One can also provide the RSL in a file file: examples/job_submission/ submitecho.xml /bin/echo Hello world from GridNexus /${GLOBUS_USER_HOME}/echo.out /${GLOBUS_USER_HOME}/echo.err

60. 60 GridNexus © 2005 Clayton S. Ferner GridExec (GRAM Client) ● The Factory Type indicates a scheduler, such as LSF, SGE, PBS, etc. (default is Fork). file: examples/job_submission/ submitgaussian.xml

61. 61 GridNexus © 2005 Clayton S. Ferner GridExec (GRAM Client) ● The Input and Output Files can be used to perform file staging file: examples/job_submission/ submitgaussian.xml file:///home/cferner/stdout gsiftp://gridprof1.bear.uncw.edu/home/ cferner/stdout

62. 62 GridNexus © 2005 Clayton S. Ferner File Transfer via GridFTP ● The first argument is the source and the second is the destination ● “gsftp://” indicates a fully qualified URL ● “file://” indicates a local file file: examples/job_submission/ gridFTP.xml

63. 63 GridNexus © 2005 Clayton S. Ferner File Transfer via SFTP ● To transfer files with sftp, the local machine and server must be setup to use public/private key authentication (see installation guide for details) file: examples/job_submission/ sftp_get.xml

64. 64 GridNexus © 2005 Clayton S. Ferner GridNexus UNCW GUI for Workflow Management www.gridnexus.org Clayton Ferner cferner@uncw.edu