1. BASIC BASH SCRIPTING Katherine Holcomb UVACSE

2. Uses for Scripting Automate repetitive tasks Could also use languages like Perl or Python but it’s still handy to know some bash Write job scripts for the PBS queueing system For most users these must be a shell script (bash, ksh, or tcsh) Anything you can type on the command line can be put into a script

3. Example #!/bin/bash #This is a bash script echo “Hello world!” echo “I am done now!” #my comment The first line tells the system to start the shell and execute the commands. It is sometimes called a “shebang.”

4. Executing Your Script Suppose the previous script was in a file hello.sh By default it is just text. You must explicitly make it executable chmod a+x hello.sh Then you can just invoke it by name at the command line./hello.sh

5. Comments and Continuations All text from the # to the end of the line is ignored To continue a statement on to the next line, type \ #This is a comment echo “This line is too long I \ want it to be on two lines” Multiple statements can be placed on the same line when separated by the ; a=20; b=3

6. Sourcing If you execute the script as a standalone script, it starts a new shell -- that is what the shebang does -- if you do not use that you can type bash myscript.sh Sometimes you just want to bundle up some commands that you repeat regularly and execute them within the current shell. To do this you should not include a shebang, and you must source the script. or source

7. Variables Like all programming languages, the shell permits you to define variables. Variables and environment variables are essentially the same thing. Environment variables are conventionally written with all capital letters and are used to set properties. Shell variables are used for assignments and such. Arrays are supported Scalar variables are referenced with $ When assigning variables do not put spaces around the equals sign!

8. String Expressions and Variables Bash isn’t very good at math so much of what we do with it involves strings. String expressions Single quotes (‘hard quotes’) : the expression is evaluated literally (no substitutions) Double quotes ( “soft quotes” ): the $, \ (backslash), and ` (backtick) are taken to be shell characters $ indicates a variable name follows \ escapes the following character ` command ` command is executed and its output streams are returned --In newer versions $(command) is synonymous

9. Example echo `date` echo ‘This is worth $2’ var=2 echo “This is worth $2” echo “This is worth $var” echo “This is worth \$$var”

10. Conditionals The shell has an if else construct: if [[ ]] then commands elif #optional commands else #optional more commands fi

11. Conditions The conditions take different forms depending on what you want to compare. String comparison operators: = != -z -n equal, not equal, lexically before, lexically after (both these may need to be escaped), zero length, not zero length. Can use == but behaves differently in [] versus [[]]

12. Example: String Comparisons if [[ $name == “Tom” ]]; then echo Tom, you are assigned to Room 12 elif [[ $name == “Harry” ]]; then echo Harry, please go to Room 3 elif [[ -z $name ]]; then echo You did not tell me your name else echo Your name is $name fi

13. Numeric Comparisons Numeric comparisons are possible: -eq -ne -gt -ge -lt -le if [[ $a -eq 2 ]]; then #note ; stuff fi

14. Testing File Properties This is a very common occurrence in bash scripts. There are many operators for this. These are the most common: -e : file exists -f : file exists and is a regular file -s : file exists and has length > 0 -d : exists and is a directory

15. Example if [[ -d mydir ]]; then if [[ -f the_file ]]; then cp the_file mydir fi else mkdir mydir echo “Created mydir” fi

16. Other Conditional Operators ! : not ; negates what follows -a : and ; for compound conditionals can use && with [[]] -o : or ; for compound conditionals can use || with [[]]

17. Case Statement case expression in pattern1 ) statements ;; pattern2 ) statements ;; pattern3 ) statements ;; esac

18. Example case $filename in *.f) echo “Fortran 77 source file” ;; *.c) echo “C source file” ;; *.py) echo “Python script file” ;; *) #optional, indicates default echo “I don’t know what this file is” ;; esac

19. Loops The bash for loop is a little different from the equivalent in C/C++/Fortran (but is similar to Perl or Python) for variable in iterator do commands done

20. Examples for i in 1 2 3 4 5 ; do echo “I am on step $i” done for i in {1..5} #bash 3.0 and up do echo “I am on step $i” done for i in 0{1..9} {10..100}; do echo “File extension will be $i” done

21. Three-Expression For for (( EXPR1; EXPR2; EXPR3 )) do statements done Example: for (( i=0; i<$IMAX;i++ )); do echo $name”.”$i done

22. while loop while [ condition ] do command done One of the commands in the while loop must update the condition so that it eventually becomes false.

23. break If you need to exit a loop before its termination condition is reached you can use the break statement. while [ condition ] do if [ disaster ]; then break fi command done

24. continue To skip the commands for an iteration use continue for i in iterator; do if [[ something ]]; then continue fi command done

25. Bash Arithmetic We said earlier that bash is bad at arithmetic, but some basic operations can be performed. Expressions to be evaluated numerically must be enclosed in double parentheses. It works only with integers. x=$((4+20)) i=$(($i+1)) If you need more advanced math (even just fractions!) you must use bc.

26. bc If you really, really, really must do math in a bash script, most of the time you must use bc The syntax is very peculiar x=$(echo “3*8+$z” | bc)

27. Command-Line Arguments Many bash scripts need to read arguments from the command line. The arguments are indicated by special variables $0, $1, $2, etc. $0 is the name of the command itself The subsequent ones are the options To have a variable number of options, use the shift built-in. The special variable $# is the number of arguments (not counting the command name)

28. Example #!/bin/bash USAGE="Usage:$0 dir1 dir2 dir3...dirN" if [ "$#" == "0" ]; then echo "$USAGE" exit 1 fi while [ $# -gt 0 ]; do echo “$1” shift done

29. More Useful Example while plus case while [ $# -gt 0 ]; do case “$1” in -v) verbose=“on”;; -*) echo >&2 “USAGE: $0 [-v] [file]” exit 1;; *) break;; # default esac shift done

30. Functions function name() { statement VALUE=integer return $VALUE } the keyword function is optional in newer versions of bash. The parentheses are always left empty. Function definitions must precede any invocations.

31. Function Arguments Function arguments are passed in the caller. In the function they are treated like command-line options. #!/bin/bash function writeout() { echo $1 } writeout “Hello World”

32. Variables Variables set in the function are global to the script! #!/bin/bash myvar="hello" myfunc() { myvar="one two three" for x in $myvar do echo $x done } myfunc echo $myvar $x

33. Making Local Variables We can use the keyword local to avoid clobbering our global variables. #!bin/bash myvar="hello" myfunc() { local x local myvar="one two three" for x in $myvar ; do echo $x done } myfunc echo $myvar $x

34. Return Values Strictly speaking, a function returns only its exit status. The returned value must be an integer You can get the value with $? e.g. myfunc $1 $2 result=$?

35. String Operations Bash has a number of built-in string operations. Concatenation Just write them together (literals should be quoted) newstring=$oldstring”.ext” String length ${#string} Extract substring Strings count from 0—first character is numbered 0 Extract from pos to the end ${string:pos} Extract len characters starting at pos ${string:pos:len}

36. Clipping Strings It is very common in bash scripts to clip off part of a string so it can be remodeled. Delete shortest match from front of string ${string#substring} Delete longest match from front of string ${string##substring} Delete shortest match from back of string ${string%substring} Delete longest match from back of string ${string%substring}

37. Arrays Array variables exist but have a somewhat unusual syntax. Arrays are zero based so the first index is 0 Initialize arrays with a list enclosed in parentheses Obtaining the value of an item in an array requires use of ${} val=${arr[$i]} ${arr[@]} # All of the items in the array @ or * work in this and next example ${#arr[@]} # Number of items in the array ${#arr[0]} # Length of item zero

38. Example my_arr=(1 2 3 4 5 6) for num in ${my_arr[@]}; do echo $num done

39. Realistic Example jpg_files=(`ls *jpg`) for file in ${jpg_files[*]}; do if [[ -n $file ]]; then convert $file ${file%”.jpg”}.png fi done

40. Herefiles A herefile or here document is a block of text that is dynamically generated when the script is run CMD << Delimiter line Delimiter

41. Example #!/bin/bash # 'echo' is fine for printing single line messages, # but somewhat problematic for for message blocks. # A 'cat' here document overcomes this limitation. cat <<End-of-message ------------------------------------- This is line 1 of the message. This is line 2 of the message. This is line 3 of the message. This is line 4 of the message. This is the last line of the message. ------------------------------------- End-of-message

42. Regular Expressions Regular expressions are generalizations of the wildcards often used for simple file commands. A regular expression consists of a pattern that attempts to match text. It contains one or more of: A character set An anchor (to the line position) Modifiers Without an anchor or repeat it will find the leftmost match and stop.

43. Regex: Character sets and modifiers The character set is the set of characters that must be matched literally. Modifiers expand the potential matches. * matches any number of repeats of the pattern before it (note that this is different from its use in the shell) including 0 matches. ? Matches 0 or 1 character (also different from the shell wildcard). + matches one or more, but not 0, occurrences.. matches any single character, except newline

44. More Modifiers \ escapes the preceding character, meaning it is to be used literally and not as a regex symbol. \ can also indicate nonprinting characters, e.g. \t for tab. () group the pattern into a subexpression | pipe is or [gray|grey] equivalent to [gr(a|e)y]

45. Regex: Ranges and Repetition [] enclose a set of characters to be matched. - indicates a range of characters (must be a subsequence of the ASCII sequence) {n} where n is a digit, indicates exactly n repetitions of the preceding character or range. {n,m} matches n to m occurrences. {n,} matches n or more occurrences.

46. Regex: Anchors and Negation ^ inside square brackets negates what follows it ^ outside square brackets means “beginning of target string” $ means “end of target string”. Matches “any character in the indicated position” Note: the “target string” is usually but not always a line in a file.

47. Examples AABB* matches AAB AABB AABBBB But not AB ABB ABBBB

48. Examples (Cont.) [a-zA-Z] matches any letter [^a-z] matches anything except lower-case letters.all matches all, call, ball, mall, and so forth. Also matches shall (since it contains hall ). Regex patterns are said to be greedy since they find a match with the most generous interpretation of the pattern.

49. Extensions and Shortcuts Most shells and languages support some shortcuts: \w : [A-Za-z0-9_] \s : [ \t\r\n] some flavors have a few more rare whitespace characters \d : [0-9] \D : ^\d ^\W: ^\w ^\S: ^\s NB \D\S is not the same as ^\d\s ; in fact it matches anything. ^\d\s matches a but not 1

50. Grep, Sed and Awk grep or egrep can be used with regular expressions. sed is the stream editor. It is used to script editing of files. awk is a programming language to extract data and print reports.

51. grep examples grep “regex” filename The quotes are often needed; they make sure it’s interpreted as a regex We assume Gnu grep on Linux (it is also called egrep) egrep matches anywhere in the string grep ^root /etc/passwd grep :$ /etc/passwd

52. sed examples Sed operates on standard input and outputs to stdout unless told otherwise. Thus you must redirect. An option will tell it to overwrite the old file. This is often a mistake (if you do this, be sure to debug the sed command carefully). sed ‘command’ new Note hard quotes—best practice is to use them sed ‘s/day/night/’ new Remember, expressions are greedy; day/night here changes Sunday to Sunnight

53. awk examples awk ‘pattern {actions}’ file Similar to C, awk “action” lines end with ; For each record (line) awk splits on whitespace. The results are stored as fields and can be referenced as $1, $2, etc. The entire line is $0 and $NF indicates the number of fields in the line. awk ‘pattern1 {actions} pattern2 {actions}’ file awk ‘Smith’ employees.txt awk ‘{print $2, $NF;}’ employees.txt

54. Some Resources Linux Shell Scripting Tutorial: http://bash.cyberciti.biz/guide/Main_Page How-To: http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html Advanced Scripting Tutorial http://tldp.org/LDP/abs/html/ http://tldp.org/LDP/abs/html/

55. More Resources Regex Tutorials http://www.regular-expressions.info/ http://www.zytrax.com/tech/web/regex.htm sed tutorial http://www.grymoire.com/Unix/Sed.html awk tutorial http://www.grymoire.com/Unix/Awk.html (but note that we have gawk on Linux)