1. Introduction to Scripting Workshop February 23, 2016

2. Introduction George Garrett & The HPC Support Team Research Computing Services CUIT

3. Introduction Please Leave Feedback

4. Introduction Slides will be sent out afterwards.

5. Introduction What is a script?

6. Introduction What is a shell script?

7. Introduction What is a shell script? A file with shell commands.

8. Introduction Why use scripts?

9. Scripting Resources How Linux Works by Brian Ward Available as an E-book from Columbia University Libraries and at Safari Books OnlineE-book

10. Scripting Resources Shell and Scripting Tutorial http://linuxcommand.org/

11. Scripting Resources Advanced Bash-Scripting Guide http://tldp.org/LDP/abs/html/

12. Cunix System: cunix.columbia.edu User: Your UNI

13. Access Windows Instructions 1.Search for putty on Columbia home page 2.Select first result 3.Follow link to Putty download page 4.Download putty.exe 5.Run putty.exe

14. Access Mac Instructions 1.Run terminal

15. Access Mac (Terminal) $ ssh UNI@cunix.columbia.edu Windows (Putty) Host Name: cunix.columbia.edu

16. Access Does everyone have access?

17. Quick Review shelluser interface to the system $standard prompt symbol pwdprint working (current directory) cdchange directory.current directory lslist directory contents

18. Quick Review catprint a file sortsort the lines of a file grepprint lines matching a pattern echo hiprint hi sleep 5wait 5 seconds mancommand manual

19. Workshop Setup $ mkdir workshop $ cd workshop $ cp /tmp/workshop/*.

20. Command Line Example Word Count from “Data Science at the Command Line” - Jeroen Janssens

21. Let’s Download Some Data Gutenberg Project: http://www.gutenberg.orghttp://www.gutenberg.org Which famous novel should we use? Download plain text version using curl $ curl [URL] > novel.txt Trim header and footer, leaving only main text

22. Pipes $ cat alice.txt | grep rabbit Pipes connect output from one command to the input of another command

23. Pipes $ cat alice.txt | grep rabbit | sort You can keep on combining commands with more pipes.

24. Pipes $ cat alice.txt | grep rabbit | tr r w trtranslate characters

25. wcount $ cat wcount cat alice.txt | tr …

26. wcount $ wcount

27. wcount $ wcount -bash: wcount: command not found

28. wcount $./wcount

29. wcount $./wcount -bash:./wcount:Permission denied

30. wcount $ ls –l wcount

31. wcount $ ls –l wcount -rw-rw---- [ snip ]

32. wcount $ ls –l wcount -rw-rw---- [ snip ] $ chmod +x wcount

33. wcount $ ls –l wcount -rw-rw---- [ snip ] $ chmod +x wcount $ ls –l wcount -rwxrwx--x [ snip ]

34. wcount $./wcount Should work this time.

35. file Determine type of file. $ file wcount

36. file Determine type of file. $ file wcount wcount: ASCII text

37. wcount Choose an editor nano  Recommended default vi emacs

38. nano Nano commands are on back of cheat sheet. ^ means “hold down control”

39. Edit wcount $ nano wcount

40. #! Add “#!” to first line #!/bin/sh cat alice.txt | tr …

41. #! $./wcount Still works.

42. #! Some #! first lines you might see #!/bin/sh #!/bin/bash #!/usr/bin/perl #!/usr/bin/python

43. file Has the file type changed? $ file wcount wcount: POSIX shell script text executable

44. Variables $ file=alice.txt $ echo $file alice.txt

45. Variables 1.Add file=alice.txt to wcount. 2.Replace cat alice.txt with cat and the variable.

46. Variables #!/bin/sh file=alice.txt cat $file | tr …

47. Variables You could put $file in double quotes. Why put quotes around $file? cat “$file” | tr …

48. Command Line Parameters We’re going to change wcount so any file can be specified from the command line. $./wcount moby.txt

49. Command Line Parameters Change wcount so any file can be specified from the command line. $./wcount moby.txt  $1

50. Command Line Parameters 1.Create a new file named “param” 2.Put the #! directive on the first line 3.On next line type: echo $1 4.Save and make executable 5.Run it

51. Command Line Parameters $ cat param #!/bin/sh echo $1

52. Command Line Parameters $./param $./param alice.txt $./param aaa bbb ccc

53. Command Line Parameters Update param to print out $# echo $# echo $1 Run it with different numbers of parameters

54. Command Line Parameters Update wcount to use $1 instead of alice.txt.

55. Command Line Parameters Before: file=alice.txt After: file="$1"

56. Command Line Parameters $./wcount alice.txt

57. if if [[ condition ]] then do something fi

58. [, [[, (( [ : Standard comparison [[ : Extended comparison (( : Mathematical comparison

59. [, [[, (( [ : Standard comparison [[ : Extended comparison  (( : Mathematical comparison

60. Comparison with [[ if [[ “$count” -eq 100 ]] -eq : equals -ne : not equal -gt : greater than …etc.

61. if if [[ $# -ne 1 ]] then do something fi

62. if if [[ $# -ne 1 ]] then echo “Usage: wcount file” exit 1 fi

63. if $./wcount $./wcount alice.txt $./wcount alice.txt junk

64. if if [[ ! –f “$file” ]] then echo “Error: File $file not found.” exit 1 fi

65. File Tests Common tests -e : File exists -f : File is “regular” -d : File is a directory …many more

66. if if [[ ! –f “$file” ]] then echo “Error: File $file not found.” exit 1 fi

67. Add lines parameter Show the five most common words: $./wcount alice.txt 5

68. while i=7 while (( $i > 4 )) do echo $i i=`expr $i - 1` done

69. for for i in a b c do echo $i done for file in `ls` do ls $file done

70. for for i in {1..3} do echo “Repeat $i times” done Repeat 1 times Repeat 2 times Repeat 3 times

71. Adding Comments Everything the shell encounters after a hash mark on a line is ignored. Comments are useful documenting your script. Or to make the interpreter ignore sections of your script. # This is a comment echo “Hi” # and this is another comment

72. readyourmind A silly impolite script but it shows a few more things. read : read input case : another way to control flow

73. Questions Questions?

74. End of Slides Next week: Intro to High Performance Computing on Yeti