1. Lecture 5
Awk and Shell

2. Sed Drawbacks Hard to remember text from one line to another
Not possible to go backward in the file
No way to do forward references like /..../+1
No facilities to manipulate numbers
Cumbersome syntax

3. Awk
Programmable Filters

4. Why is it called AWK?
Aho
Weinberger
Kernighan

5. Awk Introduction
awk's purpose: A general purpose programmable filter that handles text (strings) as easily as numbers
This makes awk one of the most powerful of the Unix utilities
awk processes fields while sed only processes lines
nawk (new awk) is the new standard for awk
Designed to facilitate large awk programs
gawk is a free nawk clone from GNU
awk gets it’s input from
files
redirection and pipes
directly from standard input
Remember, Awk is a filter so it can get input from:
command line file arguments
redirection
pipes
standard input (terminal-keyboard)
AND can output the same way
into a pipe
redirected into a file
to standard output (screen)

6. AWK Highlights
A programming language for handling common data manipulation tasks with only a few lines of code
awk is a pattern-action language, like sed
The language looks a little like C but automatically handles input, field splitting, initialization, and memory management
Built-in string and number data types
No variable type declarations
awk is a great prototyping language
Start with a few lines and keep adding until it does what you want
Programmers spend a lot of time doing simple, mechanical, repetitive data manipulation
changing the format of data
checking its validity
finding items with some property
adding up numbers
printing reports
While all of these tasks ought to be automated, it’s a pain to have to write a special purpose program in C each time such a task comes up
AWK is a programming language designed to handle such tasks with short, simple programs, often only a couple of lines long
An AWK program is a sequence of patterns and actions that tell AWK what to look for in the input data and what to do when it finds it
The action language looks a little bit like C but there are no variable or type declarations and strings and numbers are built in data-types that are handled automatically by AWK
AWK splits input lines into fields automatically - it also handles input, initialization and memory management automatically
AWK is also a great prototyping language, start with a few lines and keep adding - Jon Bentley “Programming Pearls”

7. Structure of an AWK Program
An awk program consists of:
An optional BEGIN segment
For processing to execute prior to reading input
pattern - action pairs
Processing for input data
For each pattern matched, the corresponding action is taken
An optional END segment
Processing after end of input data
BEGIN {action}
pattern {action} .
pattern { action}
END {action}

8. Running an AWK Program There are several ways to run an Awk program
awk 'program' input_file(s)
program and input files are provided as command-line arguments
awk 'program'
program is a command-line argument; input is taken from standard input (yes, awk is a filter!)
awk -f program_file input_files
program is read from a file

9. Patterns and Actions Search a set of files for patterns.
Perform specified actions upon lines or fields that contain instances of patterns.
Does not alter input files.
Process one input line at a time
This is similar to sed

10. Pattern-Action Structure
Every program statement has to have a pattern or an action or both
Default pattern is to match all lines
Default action is to print current record
Patterns are simply listed; actions are enclosed in { }
awk scans a sequence of input lines, or records, one by one, searching for lines that match the pattern
Meaning of match depends on the pattern
Note that I and most sources will use the terms lines and records interchangeably

11. Patterns Selector that determines whether action is to be executed
pattern can be:
the special token BEGIN or END
regular expressions (enclosed with //)
arithmetic relation operators
string-valued expressions
arbitrary combination of the above
/NYU/ matches if the string “NYU” is in the record
x > 0 matches if the condition is true
/NYU/ && (name == "UNIX Tools")

12. BEGIN and END patterns
BEGIN and END provide a way to gain control before and after processing, for initialization and wrap-up.
BEGIN: actions are performed before the first input line is read.
END: actions are done after the last input line has been processed.

13. Actions
action may include a list of one or more C like statements, as well as arithmetic and string expressions and assignments and multiple output streams.
action is performed on every line that matches pattern.
If pattern is not provided, action is performed on every input line
If action is not provided, all matching lines are sent to standard output.
Since patterns and actions are optional, actions must be enclosed in braces to distinguish them from pattern.

14. An Example
ls | awk ' BEGIN { print "List of html files:" } /\.html$/ { print } END { print "There you go!" } '
List of html files: index.html as1.html as2.html There you go!

15. Variables awk scripts can define and use variables BEGIN { sum = 0 }
END { print sum }
Some variables are predefined

16. Records Default record separator is newline
By default, awk processes its input a line at a time.
Could be any other regular expression.
RS: record separator
Can be changed in BEGIN action
NR is the variable whose value is the number of the current record.

17. Fields Each input line is split into fields.
FS: field separator: default is whitespace (1 or more spaces or tabs)
awk -Fc option sets FS to the character c
Can also be changed in BEGIN
$0 is the entire line
$1 is the first field, $2 is the second field, ….
Only fields begin with $, variables are unadorned

18. Simple Output From AWK Printing Every Line Printing Certain Fields
If an action has no pattern, the action is performed to all input lines
{ print } will print all input lines to standard out
{ print $0 } will do the same thing
Printing Certain Fields
Multiple items can be printed on the same output line with a single print statement
{ print $1, $3 }
Expressions separated by a comma are, by default, separated by a single space when output

19. Output (continued) NF, the Number of Fields Computing and Printing
Any valid expression can be used after a $ to indicate the contents of a particular field
One built-in expression is NF, or Number of Fields
{ print NF, $1, $NF } will print the number of fields, the first field, and the last field in the current record
{ print $(NF-2) } prints the third to last field
Computing and Printing
You can also do computations on the field values and include the results in your output
{ print $1, $2 * $3 }

20. Output (continued) Printing Line Numbers Putting Text in the Output
The built-in variable NR can be used to print line numbers
{ print NR, $0 } will print each line prefixed with its line number
Putting Text in the Output
You can also add other text to the output besides what is in the current record
{ print "total pay for", $1, "is", $2 * $3 }
Note that the inserted text needs to be surrounded by double quotes

21. Fancier Output Lining Up Fields
Like C, Awk has a printf function for producing formatted output
printf has the form
printf( format, val1, val2, val3, … )
{ printf(“total pay for %s is $%.2f\n”, $1, $2 * $3) }
When using printf, formatting is under your control so no automatic spaces or newlines are provided by awk. You have to insert them yourself.
{ printf(“%-8s %6.2f\n”, $1, $2 * $3 ) }
First printf says %s = string, %.2f = float with two digits after the decimal point
Second printf, %-8s = string left justified in an 8 character field
%6.2f = float with two digits after the decimal right justified in a six character field

22. Selection
Awk patterns are good for selecting specific lines from the input for further processing
Selection by Comparison
$2 >= 5 { print }
Selection by Computation
$2 * $3 > 50 { printf(“%6.2f for %s\n”, $2 * $3, $1) }
Selection by Text Content
$1 == "NYU"
/NYU/
Combinations of Patterns
$2 >= 4 || $3 >= 20
Selection by Line Number
NR >= 10 && NR <= 20
Prints all lines where hourly pay is >= to 5
Prints all lines where total pay is > 50
Prints all lines where first field is “NYU”
Prints all lines having “NYU” anywhere in line
Prints all lines where hourly pay >= to 4 or total pay >= 20
Can use && for AND

23. Arithmetic and variables
awk variables take on numeric (floating point) or string values according to context.
User-defined variables do not need to be declared.
By default, user-defined variables are initialized to the null string which has numerical value 0.

24. Computing with AWK Counting is easy to do with Awk
$3 > 15 { emp = emp + 1}
END { print emp, “employees worked more than 15 hrs”}
Computing Sums and Averages is also simple
{ pay = pay + $2 * $3 }
END { print NR, “employees”
print “total pay is”, pay
print “average pay is”, pay/NR }
{pay = pay + $2 * $3 } accumulates total employee pay
END begins processing after all records have been processed
print NR, “employees” prints the number of records (employees)
print “total pay is”, pay prints total pay
print “average pay is”, pay/NR prints the average
Total pay / number of employees

25. Handling Text
One major advantage of Awk is its ability to handle strings as easily as many languages handle numbers
Awk variables can hold strings of characters as well as numbers, and Awk conveniently translates back and forth as needed
This program finds the employee who is paid the most per hour:
# Fields: employee, payrate $2 > maxrate { maxrate = $2; maxemp = $1 }
END { print “highest hourly rate:”,
maxrate, “for”, maxemp }

26. String Manipulation String Concatenation Printing the Last Input Line
New strings can be created by combining old ones
{ names = names $1 " " }
END { print names }
Printing the Last Input Line
Although NR retains its value after the last input line has been read, $0 does not
{ last = $0 }
END { print last }
This program combines all the employee names into one string by appending each name and a blank to the previous value in the variable name
The concatenation operation in Awk is represented by writing string values one after another

27. Built-in Functions
awk contains a number of built-in functions. length is one of them.
Counting Lines, Words, and Characters using length (a poor man’s wc)
{ nc = nc + length($0) + length(RS)
nw = nw + NF }
END { print NR, "lines,", nw, "words,", nc,
"characters" }
substr(s, m, n) produces the substring of s that begins at position m and is at most n characters long.
The +1 is to account for the NEWLINE at the end of each line, length doesn’t count it

28. Control Flow Statements
awk provides several control flow statements for making decisions and writing loops
If-Then-Else
$2 > 6 { n = n + 1; pay = pay + $2 * $3 }
END { if (n > 0)
print n, "employees, total pay is", pay, "average pay is", pay/n
else
print "no employees are paid more than $6/hour" }
Computes the total and average pay of employees making more than $6/hour and uses the if to defend against a divide by zero error

29. Loop Control While # interest1 - compute compound interest
# input: amount, rate, years
# output: compound value at end of each year
{ i = 1
while (i <= $3) {
printf(“\t%.2f\n”, $1 * (1 + $2) ^ i)
i = i + 1 }
uses formula value = amount(1 + rate) ^ years
^ is the exponentiation operator
\t is a TAB
# is a comment symbol, every thing following is ignored by Awk
Get into the habit of commenting your scripts, I will require it and it will help you remember what you did a week or a month later

30. Do-While Loops
Do While
do {
statement1 }
while (expression)

31. For statements For # interest2 - compute compound interest
# input: amount, rate, years
# output: compound value at end of each year
{ for (i = 1; i <= $3; i = i + 1)
printf("\t%.2f\n", $1 * (1 + $2) ^ i) }

32. Arrays Array elements are not declared
Array subscripts can have any value:
Numbers
Strings! (associative arrays)
Examples
arr[3]="value"
grade["Korn"]=40.3

33. Array Example # reverse - print input in reverse order by line
{ line[NR] = $0 } # remember each line
END { for (i=NR; (i > 0); i=i-1) { print line[i] } }

34. Useful One (or so)-liners
END { print NR }
NR == 10
{ print $NF }
{ field = $NF }
END { print field }
NF > 4
$NF > 4
{ nf = nf + NF }
END { print nf }
Print total number of input lines
Print the tenth input line
Print the last field of every line
Print the last field of the last input line
Print every input line with more than 4 fields
Print every input line in which the last field is greater than 4
Print the total number of fields in all input lines

35. More One-liners /Jeff/ { nlines = nlines + 1 } END { print nlines }
$1 > max { max = $1; maxline = $0 }
END { print max, maxline }
NF > 0
length($0) > 80
{ print NF, $0}
{ print $2, $1 }
{ temp = $1; $1 = $2; $2 = temp; print }
{ $2 = ""; print }
Print the total number of lines that contain the pattern Beth
Print the largest first field and the line that contains it (assuming some $1 is positive)
Print every line that has at least one field
Print every line longer than 80 characters
Print the number of fields in every line, followed by the line itself
Print the first two fields of every line in opposite order
Exchange the first two fields of every line and then print the modified line
Print every line after erasing the second field

36. Even More One-liners
{ for (i = NF; i > 0; i = i - 1) printf(“%s “, $i)
printf(“\n”) }
{ sum = 0
for (i = 1; i <= NF; i = i + 1)
sum = sum + $i
print sum
{ for (i = 1; i <= NF; i = i + 1) sum = sum $i }
END { print sum } }
Print in reverse order the fields of every line
Print the sums of the fields of every line
Add up all fields in all lines and print the sum
This has been a brief intro to Awk and its capabilities. We will go into more detail next week. Read printf in lab

37. Awk Variables $0, $1, $2, $NF NR - Number of records processed
NF - Number of fields in current record
FILENAME - name of current input file
FS - Field separator, space or TAB by default
OFS - Output field separator, space by default
ARGC/ARGV - Argument Count, Argument Value array
Used to get arguments from the command line
FS and OFS are good things to set in the BEGIN pattern-action pair if you don’t want the default

38. Operators
= assignment operator; sets a variable equal to a value or string
== equality operator; returns TRUE is both sides are equal
!= inverse equality operator
&& logical AND
|| logical OR
! logical NOT
<, >, <=, >= relational operators
+, -, /, *, %, ^
String concatenation

39. Built-In Functions Arithmetic String Output Special
sin, cos, atan, exp, int, log, rand, sqrt
String
length, substitution, find substrings, split strings
Output
print, printf, print and printf to file
Special
system - executes a Unix command
system(“clear”) to clear the screen
Note double quotes around the Unix command
exit - stop reading input and go immediately to the END pattern-action pair if it exists, otherwise exit the script

40. More Information
on the website

41. Lecture 5
Shell Scripting

42. What is a shell? The user interface to the operating system
Functionality:
Execute other programs
Manage files
Manage processes
Full programming language
A program like any other
This is why there are so many shells

43. Shell History There are many choices for shells
Shell features evolved as UNIX grew

44. Shell Scripts
A shell script is a regular text file that contains shell or UNIX commands
Before running it, it must have execute permission:
chmod +x filename
A script can be invoked as:
ksh name [ arg … ]
ksh < name [ args … ]
name [ arg …]

45. Shell Scripts
When a script is run, the kernel determines which shell it is written for by examining the first line of the script
If 1st line starts with #!pathname-of-shell, then it invokes pathname and sends the script as an argument to be interpreted
If #! is not specified, the current shell assumes it is a script in its own language
leads to problems

46. Simple Example
#!/bin/sh echo Hello World

47. Scripting vs. C Programming
Advantages of shell scripts
Easy to work with other programs
Easy to work with files
Easy to work with strings
Great for prototyping. No compilation
Disadvantages of shell scripts
Slow
Not well suited for algorithms & data structures

48. The C Shell C-like syntax (uses { }'s) Inadequate for scripting
Poor control over file descriptors
Can't mix flow control and commands
Difficult quoting "I say \"hello\"" doesn't work
Can only trap SIGINT
Survives mostly because of interactive features.
Job control
Command history
Command line editing, with arrow keys (tcsh)

49. The Bourne Shell Slight differences on various systems
Evolved into standardized POSIX shell
Scripts will also run with ksh, bash
Influenced by ALGOL

50. Simple Commands
simple command: sequence of non blanks arguments separated by blanks or tabs.
1st argument (numbered zero) usually specifies the name of the command to be executed.
Any remaining arguments:
Are passed as arguments to that command.
Arguments may be filenames, pathnames, directories or special options
/bin/ls -l /
ls –l /

51. Complex Commands
The shell's power is in its ability to hook commands together
We've seen one example of this so far with pipelines:
We will see others
cut –d: -f2 /etc/passwd | sort | uniq

52. Redirection of input/ouput
Redirection of output: >
example:$ ls -l > my_files
Redirection of input: <
example: $ cat <input.data
Append output: >>
example: $ date >> logfile
Arbitrary file descriptor redirection: fd>
example: $ ls –l 2> error_log

53. Multiple Redirection cmd 2>file cmd > file 2>&1
send standard error to file
standard output remains the same
cmd > file 2>&1
send both standard error and standard output to file
cmd > file1 2>file2
send standard output to file1
send standard error to file2

54. Here Documents
Shell provides alternative ways of supplying standard input to commands (an anonymous file)
Shell allows in-line input redirection using << called here documents
format
command [arg(s)] << arbitrary-delimiter
command input :
arbitrary-delimiter
arbitrary-delimiter should be a string that does not appear in text

55. Here Document Example
#!/bin/sh mail <<EOT You guys really blew it in yesterday. Good luck tomorrow. Yours, $USER
EOT

56. Shell Variables Write name=value Read: $var
Turn local variable into environment:
export variable

57. Variable Example
#!/bin/sh MESSAGE="Hello World" echo $MESSAGE

58. Environmental Variables
NAME MEANING
$HOME Absolute pathname of your home directory
$PATH A list of directories to search for
$MAIL Absolute pathname to mailbox
$USER Your login name
$SHELL Absolute pathname of login shell
$TERM Type of your terminal
$PS1 Prompt

59. Parameters A parameter is one of the following:
A variable
A positional parameter, starting at 1 (next slide)
A special parameter
To get the value of a parameter: ${param}
Can be part of a word (abc${foo}def)
Works in double quotes
The {} can be omitted for simple variables, special parameters, and single digit positional parameters.

60. Positional Parameters
The arguments to a shell script
$1, $2, $3 …
The arguments to a shell function
Arguments to the set built-in command
set this is a test
$1=this, $2=is, $3=a, $4=test
Manipulated with shift
shift 2
$1=a, $2=test
Parameter 0 is the name of the shell or the shell script.

61. Example with Parameters
#!/bin/sh # Parameter 1: word # Parameter 2: file grep $1 $2 | wc –l
$ countlines ing /usr/dict/words 3277

62. Special Parameters $# Number of positional parameters
$ Options currently in effect
$? Exit value of last executed command
$$ Process number of current process
$! Process number of background process
$* All arguments on command line
All arguments on command line individually quoted "$1" "$2" ...

63. Command Substitution
Used to turn the output of a command into a string
Used to create arguments or variables
Command is placed with grave accents ` ` to capture the output of command
$ date Wed Sep 25 14:40:56 EDT $ NOW=`date` $ sed "s/oldtext/`ls | head -1`/g" $ PATH=`myscript`:$PATH $ grep `generate_regexp` myfile.c

64. File name expansion Wildcards (patterns)
* matches any string of characters
? matches any single character
[list] matches any character in list
[lower-upper] matches any character in range lower-upper inclusive
[!list] matches any character not in list

65. File Expansion
$ /bin/ls file1 file2 $ cat file1 a $ cat file2 b $ cat file* a b
If multiple matches, all are returned and treated as separate arguments:
Handled by the shell (exec never sees the wildcards)
argv[0]: /bin/cat
argv[1]: file1
argv[2]: file2
argv[0]: /bin/cat
argv[1]: file*
NOT

66. Compound Commands Multiple commands Command groupings
Separated by semicolon
Command groupings
pipelines
Boolean operators
Subshell
( command1; command2 ) > file
Control structures

67. Boolean Operators
Exit value of a program (exit system call) is a number
0 means success
anything else is a failure code
cmd1 && cmd2
executes cmd2 if cmd1 is successful
cmd1 || cmd2
executes cmd2 if cmd1 is not successful
$ ls bad_file > /dev/null && date $ ls bad_file > /dev/null || date Wed Sep 26 07:43:

68. Control Structures
if expression then command1 else command2 fi

69. What is an expression?
Any UNIX command. Evaluates to true if the exit code is 0, false if the exit code > 0
Special command /bin/test exists that does most common expressions
String compare
Numeric comparison
Check file properties
/bin/[ is linked to /bin/test for syntactic sugar
Good example UNIX tools working together

70. Examples
if test "$USER" = "kornj" then echo "I hate you" else echo "I like you" fi
if [ -f /tmp/stuff ] && [ `wc –l < /tmp/stuff` -gt 10 ] then echo "The file has more than 10 lines in it" else echo "The file is nonexistent or small" fi

71. test Summary String based tests Numeric tests File tests Logic
-z string Length of string is 0
-n string Length of string is not 0
string1 = string2 Strings are identical
string1 != string2 Strings differ
string String is not NULL
Numeric tests
int1 –eq int2 First int equal to second
int1 –ne int2 First int not equal to second
-gt, -ge, -lt, -le greater, greater/equal, less, less/equal
File tests
-r file File exists and is readable
-w file File exists and is writable
-f file File is regular file
-d file File is directory
-s file file exists and is not empty
Logic
! Negate result of expression
-a, -o and operator, or operator
( expr ) groups an expression

72. Parsing and Quoting

73. How the Shell Parses Part 1: Read the command:
Read one or more lines a needed
Separate into tokens using space/tabs
Form commands based on token types
Part 2: Evaluate a command:
Expand word tokens (command substitution, parameter expansion)
Split words into fields
Setup redirections, environment
Run command with arguments

74. Useful Program for Testing
/ftproot/okeefe/215/showargs.c
#include <stdio.h>
int main(int argc, char *argv[]) {
int i;
for (i=0; i < argc; i++) {
printf("Arg %d: %s\n", i, argv[i]); }
return(0);

75. Shell Comments Comments begin with an unquoted #
Comments end at the end of the line
Comments can begin whenever a token begins
Examples
# This is a comment
# and so is this
grep foo bar # this is a comment
grep foo bar# this is not a comment

76. Special Characters
The shell processes the following characters specially unless quoted:
| & ( ) < > ; " ' $ ` space tab newline
The following are special whenever patterns are processed:
* ? [ ] (turn off with set -o noglob)
The following are special at the beginning of a word:
# ~
The following are special when processing assignments:
= [ ]

77. Token Types
The shell uses spaces and tabs to split the line or lines into the following types of tokens:
Control operators (|, ||)
Redirection operators (<, >, >>)
Reserved words (while, if)
Assignment tokens (foo=bar)
Word tokens (everything else)

78. Operator Tokens
Operator tokens are recognized everywhere unless quoted. Spaces are optional before and after operator tokens.
I/O Redirection Operators:
> >> >| >& < << <<- <&
Each I/O operator can be immediately preceded by a single digit
Control Operators:
| & ; ( ) || && ;;

79. Shell Quoting Quoting causes characters to loose special meaning.
\ Unless quoted, \ causes next character to be quoted. In front of new-line causes lines to be joined.
'…' Literal quotes. Cannot contain '
"…" Removes special meaning of all characters except $, ", \ and `. The \ is only special before one of these characters and new-line.

80. Quoting Examples $ cat file* a b $ cat "file*" cat: file* not found
$ cat file1 > /dev/null $ cat file1 ">" /dev/null a cat: >: cannot open FILES="file1 file2" $ cat "$FILES" cat: file1 file2 not found

81. Simple Commands A simple command consists of three types of tokens:
Assignments (must come first)
Command word tokens (name and args)
Redirections: redirection-op + word-op
The first token must not be a reserved word
Command terminated by new-line or ;
Example:
foo=bar z=`date` print $HOME x=foobar > q$$ $xyz z=3

82. Word Splitting
After parameter expansion, command substitution, and arithmetic expansion, the characters that are generated as a result of these expansions (if not inside double quotes) are checked for split characters
Default split character is space or tab
Split characters are defined by the value of the IFS variable (IFS="" disables)

83. Word Splitting Examples
FILES="file1 file2" cat $FILES a b
IFS= cat $FILES cat: file1 file2: cannot open
IFS=x v=exit print exit $v "$v" exit e it exit

84. Pathname Expansion
After word splitting, each field that contains pattern characters is replaced by the pathnames that match
Quoting prevents expansion
set –o noglob disables
Not in original Bourne shell, but in POSIX

85. Parsing Example DATE=`date` echo $foo > \ /dev/null
assignment
word
param
redirection
echo hello there
/dev/null
/bin/echo hello there
/dev/null
PATH expansion
split by IFS

86. The eval built-in eval arg …
Causes all the tokenizing and expansions to be performed again

88. Input/Output Shell Features
Standard input, output, error
Redirection
Here documents
Pipelines
Command substitution
Exit status $?
&&, ||, if, while
Environment
export, variables
Arguments
Variables
Wildcards

89. Power of the Shell
The shell is a language that lets you use programs as you would use procedures in other languages
Called with command line arguments
If used in if, while statements programs behave like functions returning a boolean value
/bin/true: Program that just does exit(0)
/bin/false: Program that just does exit(1)
If used in command substitution, programs behave like functions returning a string
Environment behaves like global variables

90. test Summary String based tests Numeric tests File tests Logic
-z string Length of string is 0
-n string Length of string is not 0
string1 = string2 Strings are identical
string1 != string2 Strings differ
string String is not NULL
Numeric tests
int1 –eq int2 First int equal to second
int1 –ne int2 First int not equal to second
-gt, -ge, -lt, -le greater, greater/equal, less, less/equal
File tests
-r file File exists and is readable
-w file File exists and is writable
-f file File is regular file
-d file File is directory
-s file file exists and is not empty
Logic
! Negate result of expression
-a, -o and operator, or operator
( expr ) groups an expression

91. Example
#!/bin/sh if test -f /tmp/stuff && \ [ `wc –l < /tmp/stuff` -gt 10 ] then echo "The file has more than 10 lines" else echo "The file is nonexistent or small" fi

92. Arithmetic No arithmetic built in to /bin/sh
Use external command /bin/expr
expr expression
Evaluates expression and sends the result to standard output
Yields a numeric or string result
expr 4 "*" 12
expr \( \) \* 2

93. for loops Different than C:
for var in list do command done
Typically used with positional params or a list of files:
sum=0 for var in do sum=`expr $sum + $var` done
for file in *.c ; do echo "We have $file" done

94. Case statement Like a C switch statement for strings:
case $var in opt1) command1 command2 ;; opt2) command ;; *) command ;; esac
* is a catch all condition

95. Case Example #!/bin/sh echo "Say something." while true
do read INPUT_STRING
case $INPUT_STRING in
hello)
echo "Hello there." ;;
bye)
echo "See ya later." *)
echo "I'm sorry?"
esac
done
echo "Take care."

96. Case Options
opt can be a shell pattern, or a list of shell patterns delimited by |
Example:
case $name in *[0-9]*) echo "That doesn't seem like a name." ;; J*|K*) echo "Your name starts with J or K, cool." ;; *) echo "You're not special." ;; esac

97. Types of Commands All behave the same way Programs Built-in commands
Most that are part of the OS in /bin
Built-in commands
Functions
Aliases

98. Built-in Commands
Built-in commands are internal to the shell and do not create a separate process. Commands are built-in because:
They are intrinsic to the language (exit)
They produce side effects on the process (cd)
They perform much better
No fork/exec

99. Important Built-in Commands
exec : replaces shell with program
cd : change working directory
shift : rearrange positional parameters
(un)set : set positional parameters
wait : wait for background proc. to exit
umask : change default file permissions
exit : quit the shell
eval : parse and execute string
time : run command and print times
export : put variable into environment
trap : set signal handlers

100. Important Built-in Commands
continue : continue in loop
break : break in loop
return : return from function
: : true
. : read file of commands into current shell; like #include

101. Reading Lines
read is used to read a line from a file and to store the result into shell variables
read –r prevents special processing
Uses IFS to split into words
If no variable specified, uses REPLY
read
read –r NAME
read FIRSTNAME LASTNAME

102. trap command
trap specifies command that should be executed when the shell receives a signal of a particular value.
trap [ [command] {signal}+]
If command is omitted, signals are ignored
Especially useful for cleaning up temporary files
trap 'echo "please, dont interrupt!"' SIGINT trap 'rm /tmp/tmpfile' EXIT

103. Functions
Functions are similar to scripts and other commands except that they can produce side effects in the callers script. The positional parameters are saved and restored when invoking a function. Variables are shared between caller and callee.
Syntax: name () { commands }

104. Aliases Like macros (#define in C)
Shorter to define than functions, but more limited
Not recommended for scripts
Example:
alias rm='rm –i'

105. Search Rules Special built-ins Functions
command bypasses search for functions
Built-ins not associated with PATH
PATH search
Built-ins associated with PATH
Executable images

106. Script Examples Rename files to lower case Strip CR from files
Emit HTML for directory contents

107. Rename files #!/bin/sh for file in * do
lfile=`echo $file | tr A-Z a-z`
if [ $file != $lfile ]
then
mv $file $lfile fi
done

108. Remove DOS Carriage Returns
#!/bin/sh
TMPFILE=/tmp/file$$
if [ "$1" = "" ]
then
tr -d '\r'
exit 0 fi
trap 'rm -f $TMPFILE' EXIT
for file in do
if tr -d '\r' < $file > $TMPFILE
mv $TMPFILE $file
done

109. Generate HTML
$ dir2html.sh > dir.html

110. The Script #!/bin/sh if test –n "$1" then cd "$1" fi cat <<HUP
<html>
<h1> Directory listing for $PWD </h1>
<table border=1>
<tr>
HUP
num=0 # global variable counting file number
for file in * do
genhtml $file # this function is on next page
done
</tr>
</table>
</html>

111. Funciton genhtml genhtml() { file=$1 echo "<td><tt>"
if [ -f $file ]
then echo "<font color=blue>$file</font>"
elif [ -d $file ]
then echo "<font color=red>$file</font>"
else echo "$file" fi
echo "</tt></td>"
# Check if this is the end of the row
num=`expr $num + 1`
if [ $num -gt 4 ]
then
echo "</tr><tr>"
num=0 }

112. Korn Shell / bash Features

113. Command Substitution Syntax
Better syntax with $(command)
Allows nesting
x=$(cat $(generate_file_list))
Backward compatible with ` … ` notation

114. Expressions Expressions are built-in with the [[ ]] operator
if [[ $var = "" ]] …
Gets around parsing issues when using /bin/test, allows checking strings against patterns
Operations:
string == pattern
string != pattern
string1 < string2
file1 –nt file2
file1 –ot file2
file1 –ef file2
&&, ||
Patterns:
Can be used to do string matching
if [[ $foo = *a* ]]
if [[ $foo = [abc]* ]]

115. Additonal Parameter Expansion
${#param} – Length of param
${param#pattern} – Left strip min pattern
${param##pattern} – Left strip max pattern
${param%pattern} – Right strip min pattern
${param%%pattern} – Right strip max pattern
${param-value} – Default value if param not set

116. Variables Variables can be arrays Indexed by number
foo[3]=test
echo ${foo[3]}
Indexed by number
${#arr} is length of the array
Multiple array elements can be set at once:
set –A foo a b c d
echo ${foo[1]}
Set command can also be used for positional params: set a b c d; print $2

117. Functions Alternative function syntax:
function name { commands }
Allows for local variables (with typeset)
$0 is set to the name of the function

118. Additional Features Built-in arithmetic: Using $((expression ))
e.g., print $(( * 8 / x ))
Tilde file expansion
~ $HOME
~user home directory of user
~+ $PWD
~- $OLDPWD

119. Control Structures Summary
if … then … fi
while … done
until … do … done
for … do … done
case … in … esac