1. 1 GridNexus © 2007 Clayton S. Ferner GridNexus UNCW GUI for Workflow Management www.gridnexus.org Clayton Ferner cferner@uncw.edu

2. 2 GridNexus © 2007 Clayton S. Ferner Outline Basics  Getting Started  Simple workflows  A little bit about JXPL  Libraries  Composites  Ports  User Library  Splitting Relations Examples  Prog  Cond  Defun  Recursion  Iteration Services  Web Services  Grid Services

3. 3 GridNexus © 2007 Clayton S. Ferner What is GridNexus GridNexus is a GUI that allows the user to create/edit/run workflows Based on Ptolemy II http://ptolemy.eecs.berkeley.edu/ptolemyII. Ptolemy provides the GUI and workflow features. We have extended it to provide the functionality we want (JXPL and GridServices) http://ptolemy.eecs.berkeley.edu/ptolemyII Release 2.0 download available www.gridnexus.org www.gridnexus.org

4. 4 GridNexus © 2007 Clayton S. Ferner Getting Started Download from www.gridnexus.org www.gridnexus.org Follow instructions for installation Run GridNexus Choose File->New - >Workflow

5. 5 GridNexus © 2007 Clayton S. Ferner Getting Started The main frame on the right is the palette for building workflows The upper left frame provides the library of modules The lower left frame is a thumbnail of the entire workflow

6. 6 GridNexus © 2007 Clayton S. Ferner The Basics Sources produce data without needing input Sinks consume data but may have side effects (such as displaying results) All workflows must start with sources and end with sinks

7. 7 GridNexus © 2007 Clayton S. Ferner Simple Example 1 Click and drag the “Const” source to the workflow. Click and drag the “JxplDisplay” sink to the workflow

8. 8 GridNexus © 2007 Clayton S. Ferner Simple Example 1 Double-click on the Const module Change its value to 10 Click commit The new value is shown on the icon

9. 9 GridNexus © 2007 Clayton S. Ferner Simple Example 1 Input ports are on the left-hand side and output ports are on the right-hand side of each module Click and drag from the output port of the Const module to the JxplDisplay

10. 10 GridNexus © 2007 Clayton S. Ferner Simple Example 1 A link (or relation) is created between the two modules The output of Const is consumed by the JxplDisplay

11. 11 GridNexus © 2007 Clayton S. Ferner Simple Example 1 Click on the run button ( ) The JxplDisplay evaluates the input and produces a display window to show the results. Notice the output is in XML (actually JXPL)

12. 12 GridNexus © 2007 Clayton S. Ferner Simple Example 2 Transformers are modules that take input, transform it, and produce new output This example computes the express: (23 + 6) ∙ -2

13. 13 GridNexus © 2007 Clayton S. Ferner Simple Example 2 The Multiplication module takes the result of the addition (its first input) and multiplies that by -2 (its second input) The result is consumed by JxplDisplay

14. 14 GridNexus © 2007 Clayton S. Ferner What's Going On? The workflow is not actually performing the operations. Instead it is creating a script (JXPL) that, when executed, produces the result The JxplDisplay is evaluating the script and displaying the results

15. 15 GridNexus © 2007 Clayton S. Ferner What's Going On? Double-click on the JxplDisplay and deselect the “Evaluate Jxpl” parameter This parameter tells JxplDisplay whether or not to evaluate the script that is generated

16. 16 GridNexus © 2007 Clayton S. Ferner What's Going On? Now when we run it, we see the actual script that is produced by the workflow The script is written in XML using a language developed at UNCW called JXPL

17. 17 GridNexus © 2007 Clayton S. Ferner A Little Bit about JXPL JXPL is based on LISP The corresponding LISP to the JXPL on the right looks like: (* (+ 23 6) -2) <jxpl:list xmlns:jxpl="http://www.jxpl.org/script">

18. 18 GridNexus © 2007 Clayton S. Ferner A Little Bit about JXPL Why?  XML is used to transport data between web/grid services  XML opening/closing tags LISP opening/closing parens  Everything is either an atom or a list (functions, Data Structures) <jxpl:list xmlns:jxpl="http://www.jxpl.org/script">

19. 19 GridNexus © 2007 Clayton S. Ferner A Little Bit about JXPL The transformer library is LISP-centric More on JXPL later See www.gridnexus.orgwww.gridnexus.org

20. 20 GridNexus © 2007 Clayton S. Ferner Transformer Library

21. 21 GridNexus © 2007 Clayton S. Ferner Sources Const – Integers and Decimals Decimal – Integers and Decimals Symbol – Variables (use quotes 1 ) Rational – Fractions (use quotes 1 ) 1 Quotes are needed to prevent Ptolemy from evaluating the expressions before the JXPL evaluator gets a chance to see them.

22. 22 GridNexus © 2007 Clayton S. Ferner Examples Choose File->Open and select the Examples folder There are many examples that don't need explanation. Just give 'em a try! Others will be discussed later

23. 23 GridNexus © 2007 Clayton S. Ferner Composites A composite is a means to abstract a workflow into a single module From the “utilities” folder, drag the “Composite actor” to the workflow Right-click->Look Inside

24. 24 GridNexus © 2007 Clayton S. Ferner Composites This opens a new window in which we create the workflow that defines the new module First we need ports  Single ports take or produce single values  Multi ports take or produce multiple values

25. 25 GridNexus © 2007 Clayton S. Ferner Composites Toolbar icons  Single input port  Single output port  Multi input port  Multi output port Notice how the ports show up on the composite actor module

26. 26 GridNexus © 2007 Clayton S. Ferner Composites Right-click on a port and choose “Customize Name” Enter name for the Icon Selecting “Show name” causes the label to show up on the composite module

27. 27 GridNexus © 2007 Clayton S. Ferner Composites To make a relation from a port, hold down the shift key and click and drag Since a multiport can produce more than one value, we need to specify a width for the relation involving a multiport

28. 28 GridNexus © 2007 Clayton S. Ferner Composites For a multiport, double-click on the relation (the link from the multiport) and choose Add The name should be “width” with a default of zero

29. 29 GridNexus © 2007 Clayton S. Ferner Composites Close the composite window We can name the composite by right- clicking and choosing “Customize Name” Now we are ready to use it

30. 30 GridNexus © 2007 Clayton S. Ferner User Library To save a module in the user library, right- click on it and choose Save Actor in Library The user library is stored in.GridNexus/user\ library.xml in your home directory

31. 31 GridNexus © 2007 Clayton S. Ferner User Library Actually, it isn't saved. It opens the user library with the new module You still have to choose File->Save to save the user library.

32. 32 GridNexus © 2007 Clayton S. Ferner User Library Alternatively, you can save an entire workflow in the library Choose Graph->Save in Library. The workflow is saved in the library as a new composite named with the filename

33. 33 GridNexus © 2007 Clayton S. Ferner Splitting Relations Sometime, you want the results of a module to be used as input to more than one other modules SetQ module assigns a value to a symbol The symbol X is input to both SetQ and Addition modules file: examples/setq.xml

34. 34 GridNexus © 2007 Clayton S. Ferner Splitting Relations To create a new splitter, click the button Link the output of Symbol to the splitter Hold down the shift key and click and drag the splitter to the input port of SetQ. file: examples/setq.xml

35. 35 GridNexus © 2007 Clayton S. Ferner Splitting Relations Repeat for Addition (Sometimes, the use of a splitter can complicate the workflow due to crossing relations. It is not necessary to split the output from Symbol “x”. Alternatively, we can simply make a duplicate Symbol “x”.) file: examples/setq.xml

36. 36 GridNexus © 2007 Clayton S. Ferner The Prog The Prog (short for Program) module allows one to execute separate workflows where it is not desirable to have a relationship between them (i.e. synchro- nization rather than data dependency) file: examples/prog.xml

37. 37 GridNexus © 2007 Clayton S. Ferner The Prog The Prog primitive evaluates each of its inputs in turn It returns the results of the last one evaluated (in this case this case it returns 2x). file: examples/prog.xml

38. 38 GridNexus © 2007 Clayton S. Ferner Multi Ports One difficulty with multi ports is the order in which its inputs are evaluated is not evident from the GUI Which of the sub- workflows is executed first? Are you sure? file: examples/prog.xml

39. 39 GridNexus © 2007 Clayton S. Ferner Multi Ports If one deletes the SetQ -> Prog relation, then puts it back, the result is that Prog will evaluation the Multiplication before the SetQ – Not good! Multi Ports are best for commutative operations like Addition, but not like Prog. However, the alternative to using a multi port for Prog is to have a fixed number of single ports. When it doubt, delete all relations to a multi port and redo them

40. 40 GridNexus © 2007 Clayton S. Ferner Cond Cond is the branching mechanism in LISP A Cond List Element takes a boolean expression and the results if the boolean is true file: examples/cond.xml

41. 41 GridNexus © 2007 Clayton S. Ferner Cond The Cond finds the first Cond List Element whose boolean is true and returns its result Be careful of the order in which the Elements are linked to the multi- input port file: examples/cond.xml

42. 42 GridNexus © 2007 Clayton S. Ferner Cond An “else” clause should the last List Element with a boolean of “true” The last List Element will be returned if all others are false file: examples/cond.xml

43. 43 GridNexus © 2007 Clayton S. Ferner Defun Defun defines a new primitive (function) It takes a symbol as the function name, a set of symbols as parameters, and a workflow as the body file: examples/defun.xml

44. 44 GridNexus © 2007 Clayton S. Ferner Defun Once it is defined, we can call it using the Primitive module file: examples/defun.xml

45. 45 GridNexus © 2007 Clayton S. Ferner Recursion Of course, we have to do factorial! This workflow first defines factorial (using a composite to keep the workflow clean) then uses it to compute 10!. file: examples/factorial.xml

46. 46 GridNexus © 2007 Clayton S. Ferner Recursion Inside Factorial:  If N <= 1 return 1  Else return N(N-1)! file: examples/factorial.xml

47. 47 GridNexus © 2007 Clayton S. Ferner Recursion Inside N(N-1)!:  Notice the recursive call to factorial  Remember that this generates a script that will be executed later file: examples/factorial.xml

48. 48 GridNexus © 2007 Clayton S. Ferner Iteration Arguments:  loop variable  initial value  step size (loop variable += step size)  termination boolean (true => terminate)  loop body file: examples/do.xml

49. 49 GridNexus © 2007 Clayton S. Ferner Web Services The WS Client module can serve as a general purpose client for a Web Service given the WSDL Double-click on the module and enter the URL of the WSDL

50. 50 GridNexus © 2007 Clayton S. Ferner Web Services The icon is set with the name of the service Ports are added for the methods of the service Only the method with input will be called file: examples/wsclient.xml

51. 51 GridNexus © 2007 Clayton S. Ferner Grid Services First, you need the jars of the Grid Service. E.g.: edu_uncw_math_mathservice.jar edu_uncw_math_mathservice_stubs.jar These are found in $GLOBUS_LOCATION/lib on the server Put in $GRIDNEXUS_HOME/lib (Linux) or C:\Program Files\GridNexus2.xx\lib (Windows) You will need to restart GridNexus so that it will reset the classpath

52. 52 GridNexus © 2007 Clayton S. Ferner Grid Services The WSRF Client module can serve as a general purpose client for a Grid Service given the following:  URL of the Factory  Class name of the Addressing Locator class

53. 53 GridNexus © 2007 Clayton S. Ferner Grid Services The class name of the Addressing Locator class can be found from the stubs jar From this example it would be: edu.uncw.math.service.M athServiceAddressingLoc ator $ jar tf edu_uncw_math_mathservice_stubs.jar... edu/uncw/math/Create.class edu/uncw/math/GetValue.class edu/uncw/math/MathPortType.class edu/uncw/math/MathResourceProperties.class edu/uncw/math/bindings/MathPortTypeSOAPBindingStub.class edu/uncw/math/service/MathService.class edu/uncw/math/service/MathServiceAddressing.class edu/uncw/math/service/MathServiceAddressingLocator.class edu/uncw/math/service/MathServiceLocator.class

54. 54 GridNexus © 2007 Clayton S. Ferner Grid Services The icon is set with the name of the service Ports are added for the methods of the service An EndpointReference (epr) can be provided, which will override the Factory URL

55. 55 GridNexus © 2007 Clayton S. Ferner Grid Services In order to call a method that takes no parameters, pass a value of “false” For example, the create method takes no parameters The create method creates a service instance and returns the epr file: examples/wsrf_client/ MathService2Create.xml

56. 56 GridNexus © 2007 Clayton S. Ferner Grid Services What does this workflow do?

57. 57 GridNexus © 2007 Clayton S. Ferner Grid Services The destroy method also does not take a parameter After the destroy, the service instance no longer exists and the epr file is no longer of use file: examples/wsrf_client/ MathService2Destroy.xml

58. 58 GridNexus © 2007 Clayton S. Ferner User Library Don't forget to save these actors in the user library now that they are configured Just click and drag these to the palette and they're ready for use

59. 59 GridNexus © 2007 Clayton S. Ferner GridExec (GRAM Client) The GridExec module will run an application on a service using GRAM The Factory Contact is the URL of the server If a Simple command is given, GridExec will construct an RSL file to run it file: examples/job_submission/ simplejob.xml /bin/echo Hello world from GridNexus ${GLOBUS_USER_HOME}/stdout ${GLOBUS_USER_HOME}/stderr

60. 60 GridNexus © 2007 Clayton S. Ferner GridExec (GRAM Client) One can also provide the RSL in a file file: examples/job_submission/ submitecho.xml /bin/echo Hello world from GridNexus RSL /${GLOBUS_USER_HOME}/stdout /${GLOBUS_USER_HOME}/stderr

61. 61 GridNexus © 2007 Clayton S. Ferner GridExec (GRAM Client) The Factory Type indicates a scheduler, such as LSF, SGE, Condor, PBS, etc. (default is Fork). file: examples/job_submission/ submitgaussian.xml

62. 62 GridNexus © 2007 Clayton S. Ferner GridExec (GRAM Client) The Input and Output Files can be used to perform file staging This only works if you can gridFTP files from the server to your local machine (ie. does not work on Windows) file:///home/cferner/stdout gsiftp://gridprof1.bear.uncw.edu/home/ cferner/stdout

63. 63 GridNexus © 2007 Clayton S. Ferner File Transfer via GridFTP The first argument is the source and the second is the destination “gsftp://” indicates a fully qualified URL “file://” indicates a local file file: examples/job_submission/ gridFTP.xml

64. 64 GridNexus © 2007 Clayton S. Ferner File Transfer via SFTP To transfer files with sftp, the local machine and server must be setup to use public/private key authentication (see installation guide for details)

65. 65 GridNexus © 2007 Clayton S. Ferner File Transfer via SFTP Note that Windows directories should use forward slash (/) instead of backslash (\) You can use backslash, but you will need double backslashes (\\) for every backslash.

66. 66 GridNexus © 2007 Clayton S. Ferner File Transfer via SFTP With sftp, one can accomplish file staging (albeit manual)

67. 67 GridNexus © 2007 Clayton S. Ferner GridNexus UNCW GUI for Workflow Management www.gridnexus.org Clayton Ferner cferner@uncw.edu