1. Review of Awk Principles
Awk’s purpose: to give Unix a general purpose programming language that handles text (strings) as easily as numbers
This makes Awk one of the most powerful of the Unix utilities
Awk process fields while ed/sed process lines
nawk (new awk) is the new standard for Awk
Designed to facilitate large awk programs
Awk gets it’s input from
files
redirection and pipes
directly from standard input
Remember, Awk is a filter so it can get input from:
command line file arguments
redirection
pipes
standard input (terminal-keyboard)
AND can output the same way
into a pipe
redirected into a file
to standard output (screen)

2. History
Originally designed/implemented in 1977 by Al Aho, Peter Weinberger, and Brian Kernigan
In part as an experiment to see how grep and sed could be generalized to deal with numbers as well as text
Originally intended for very short programs
But people started using it and the programs kept getting bigger and bigger!
In 1985, new awk, or nawk, was written to add enhancements to facilitate larger program development
Major new feature is user defined functions

3. Other enhancements in nawk include:
Dynamic regular expressions
Text substitution and pattern matching functions
Additional built-in functions and variables
New operators and statements
Input from more than one file
Access to command line arguments
nawk also improved error messages which makes debugging considerably easier under nawk than awk
On most systems, nawk has replaced awk
On ours, both exist
Lets edit our .cshrc file to add an alias
alias awk nawk

4. Running an AWK Program There are several ways to run an Awk program
awk ‘program’ input_file(s)
program and input files are provided as command-line arguments
awk ‘program’
program is a command-line argument; input is taken from standard input (yes, awk is a filter!)
awk -f program_file_name input_files
program is read from a file

5. Awk as a Filter
Since Awk is a filter, you can also use pipes with other filters to massage its output even further
Suppose you want to print the data for each employee along with their pay and have it sorted in order of increasing pay
awk ‘{ printf(“%6.2f %s\n”, $2 * $3, $0) }’ emp.data | sort

6. Errors
If you make an error, Awk will provide a diagnostic error message
awk '$3 == 0 [ print $1 }' emp.data
awk: syntax error near line 1
awk: bailing out near line 1
Or if you are using nawk
nawk '$3 == 0 [ print $1 }' emp.data
nawk: syntax error at source line 1
context is
$3 == 0 >>> [ <<<
1 extra }
1 extra [
nawk: bailing out at source line 1

7. Structure of an AWK Program
An Awk program consists of:
An optional BEGIN segment
For processing to execute prior to reading input
pattern - action pairs
Processing for input data
For each pattern matched, the corresponding action is taken
An optional END segment
Processing after end of input data
BEGIN{action}
pattern {action} .
pattern { action}
END {action}
BEGIN and END - two special patterns provided by Awk
Both are optional and having one does not imply that you have to have the other
BEGIN executes its actions prior to reading any of the input file(s)
END executes its actions after the last record of the input file(s) has been read
After the optional BEGIN is handled, Awk reads the first line of input, checks each of its pattern-action pairs against it sequentially, executing any that match
Awk then reads the next line of input and begins over again with the first pattern-action pair
When the script is provided on the command line, the pattern-actions are enclosed in ‘ ‘ to protect them from the shell

8. BEGIN and END
Special pattern BEGIN matches before the first input line is read; END matches after the last input line has been read
This allows for initial and wrap-up processing
BEGIN { print “NAME RATE HOURS”; print “” }
{ print }
END { print “total number of employees is”, NR }
{ print “” } prints a blank line vs { print } which prints current record

9. Pattern-Action Pairs
Both are optional, but one or the other is required
Default pattern is match every record
Default action is print record
Patterns
BEGIN and END
expressions
$3 < 100
$4 == “Asia”
string-matching
/regex/ - /^.*$/
string - abc
matches the first occurrence of regex or string in the record

10. compound
$3 < 100 && $4 == “Asia”
&& is a logical AND
|| is a logical OR
range
NR == 10, NR == 20
matches records 10 through 20 inclusive
Patterns can take any of these forms and for /regex/ and string patterns will match the first instance in the record

11. Selection
Awk patterns are good for selecting specific lines from the input for further processing
Selection by Comparison
$2 >=5 { print }
Selection by Computation
$2 * $3 > 50 { printf(“%6.2f for %s\n”, $2 * $3, $1) }
Selection by Text Content
$1 == “Susie”
/Susie/
Combinations of Patterns
$2 >= 4 || $3 >= 20
Prints all lines where hourly pay is >= to 5
Prints all lines where total pay is > 50
Prints all lines where first field is “Susie”
Prints all lines having “Susie” anywhere in line
Prints all lines where hourly pay >= to 4 or total pay >= 20
Can use && for AND

12. Data Validation Validating data is a common operation
Awk is excellent at data validation
NF != 3 { print $0, “number of fields not equal to 3” }
$2 < 3.35 { print $0, “rate is below minimum wage” }
$2 > 10 { print $0, “rate exceeds $10 per hour” }
$3 < 0 { print $0, “negative hours worked” }
$3 > 60 { print $0, “too many hours worked” }

13. Regular Expressions in Awk
Awk uses the same regular expressions we’ve been using
^ $ - beginning of/end of field
. - any character
[abcd] - character class
[^abcd] - negated character class
[a-z] - range of characters
(regex1|regex2) - alternation
* - zero or more occurrences of preceding expression
+ - one or more occurrences of preceding expression
? - zero or one occurrence of preceding expression
NOTE: the min max {m, n} or variations {m}, {m,} syntax is NOT supported

14. Awk Variables $0, $1, $2, … ,$NF NR - Number of records read
FNR - Number of records read from current file
NF - Number of fields in current record
FILENAME - name of current input file
FS - Field separator, space or TAB by default
OFS - Output field separator, space by default
ARGC/ARGV - Argument Count, Argument Value array
Used to get arguments from the command line
FS and OFS are good things to set in the BEGIN pattern-action pair if you don’t want the default

15. Arrays Awk provides arrays for storing groups of related data values
# reverse - print input in reverse order by line
{ line[NR] = $0 } # remember each line
END { i = NR # print lines in reverse order
while (i > 0) {
print line[i]
i = i - 1 }

16. Operators
= assignment operator; sets a variable equal to a value or string
== equality operator; returns TRUE is both sides are equal
!= inverse equality operator
&& logical AND
|| logical OR
! logical NOT
<, >, <=, >= relational operators
+, -, /, *, %, ^
String concatenation

17. Control Flow Statements
Awk provides several control flow statements for making decisions and writing loops
If-Else
if (expression is true or non-zero){
statement1 }
else {
statement2
where statement1 and/or statement2 can be multiple statements enclosed in curly braces { }s
the else and associated statement2 are optional
Computes the total and average pay of employees making more than $6/hour and uses the if to defend against a divide by zero error

18. Loop Control While while (expression is true or non-zero) { statement1}
uses formula value = amount(1 + rate) ^ years
^ is the exponentiation operator
\t is a TAB
# is a comment symbol, every thing following is ignored by Awk
Get into the habit of commenting your scripts, I will require it and it will help you remember what you did a week or a month later

19. For for(expression1; expression2; expression3) { statement1 }
This has the same effect as:
expression1
while (expression2) {
expression3
for(;;) is an infinite loop

20. Do While
do {
statement1 }
while (expression)

21. Computing with AWK Counting is easy to do with Awk
$3 > 15 { emp = emp + 1}
END { print emp, “employees worked more than 15 hrs”}
Computing Sums and Averages is also simple
{ pay = pay + $2 * $3 }
END { print NR, “employees”
print “total pay is”, pay
print “average pay is”, pay/NR }
{pay = pay + $2 * $3 } accumulates total employee pay
END begins processing after all records have been processed
print NR, “employees” prints the number of records (employees)
print “total pay is”, pay prints total pay
print “average pay is”, pay/NR prints the average
Total pay / number of employees

22. Handling Text
One major advantage of Awk is its ability to handle strings as easily as many languages handle numbers
Awk variables can hold strings of characters as well as numbers, and Awk conveniently translates back and forth as needed
This program finds the employee who is paid the most per hour
$2 > maxrate { maxrate = $2; maxemp = $1 }
END { print “highest hourly rate:”, maxrate, “for”, maxemp }

23. Printing the Last Input Line
String Concatenation
New strings can be created by combining old ones
{ names = names $1 “ “ }
END { print names }
Printing the Last Input Line
Although NR retains its value after the last input line has been read, $0 does not
{ last = $0 }
END { print last }
This program combines all the employee names into one string by appending each name and a blank to the previous value in the variable name
The concatenation operation in Awk is represented by writing string values one after another

24. Command Line Arguments
Accessed via built-ins ARGC and ARGV
ARGC is set to the number of command line arguments
ARGV[ ] contains each of the arguments
For the command line
awk ‘script’ filename
ARGC == 2
ARGV[0] == “awk”
ARGV[1] == “filename
the script is not considered an argument

25. ARGC and ARGV can be used like any other variable
They can be assigned, compared, used in expressions, printed
They are commonly used for verifying that the correct number of arguments were provided
So, to check for the proper number of arguments in an awk script
BEGIN {if (ARGC != 2) print “I need a filename for this script”; exit}

26. ARGC/ARGV in Action #argv.awk – get a cmd line argument and display
BEGIN {if(ARGC != 2)
{print "Not enough arguments!"}
else
{print "Good evening,", ARGV[1]} }

27. BEGIN {if(ARGC != 3)
{print "Not enough arguments!"
print "Usage is awk -f script in_file field_separator"
exit}
else
{FS=ARGV[2]
delete ARGV[2]} }
$1 ~ /..3/ {print $1 "'s name in real life is", $5; ++nr}
END {print; print "There are", nr, "students registered in your class."}

28. getline
How do you get input into your awk script other than on the command line?
The getline function provides input capabilities
getline is used to read input from either the current input or from a file or pipe
getline returns 1 if a record was present, 0 if an end-of-file was encountered, and –1 if some error occurred

29. getline Function Expression Sets getline $0, NF, NR, FNR getline var
var, NR, FNR
getline <"file"
$0, NF
getline var <"file"
var
"cmd" | getline
"cmd" | getline var

30. getline from stdin #getline.awk - demonstrate the getline function
BEGIN {print "What is your first name and major? "
while (getline > 0)
print "Hi", $1 ", your major is", $2 "." }
while (getline > 0) – since getline returns –1 on failure, this ensures you don't go into an endless loop if the getline fails for some reason, such as using a non-existant file

31. getline From a File #getline1.awk - demo getline with a file
BEGIN {while (getline <"emp.data" >0)
print $0}

32. getline From a Pipe #getline2.awk - show using getline with a pipe
BEGIN {{while ("who" | getline)
nr++}
print "There are", nr, "people logged on clyde right now."}

33. Simple Output From AWK Printing Every Line Printing Certain Fields
If an action has no pattern, the action is performed for all input lines
{ print } will print all input lines on stdout
{ print $0 } will do the same thing
Printing Certain Fields
Multiple items can be printed on the same output line with a single print statement
{ print $1, $3 }
Expressions separated by a comma are, by default, separated by a single space when output

34. Computing and Printing
NF, the Number of Fields
Any valid expression can be used after a $ to indicate a particular field
One built-in expression is NF, or Number of Fields
{ print NF, $1, $NF } will print the number of fields, the first field, and the last field in the current record
Computing and Printing
You can also do computations on the field values and include the results in your output
{ print $1, $2 * $3 }

35. Putting Text in the Output
Printing Line Numbers
The built-in variable NR can be used to print line numbers
{ print NR, $0 } will print each line prefixed with its line number
Putting Text in the Output
You can also add other text to the output besides what is in the current record
{ print “total pay for”, $1, “is”, $2 * $3 }
Note that the inserted text needs to be surrounded by double quotes

36. Formatted Output printf provides formatted output
Syntax is printf(“format string”, var1, var2, ….)
Format specifiers
%c – single character
%d - number
%f - floating point number
%s - string
\n - NEWLINE
\t - TAB
Format modifiers
- left justify in column
n column width
.n number of decimal places to print

37. printf Examples printf(“I have %d %s\n”, how_many, animal_type)
format a number (%d) followed by a string (%s)
printf(“%-10s has $%6.2f in their account\n”, name, amount)
prints a left justified string in a 10 character wide field and a float with 2 decimal places in a six character wide field
printf(“%10s %-4.2f %-6d\n”, name, interest_rate, account_number > "account_rates")
prints a right justified string in a 10 character wide field, a left justified float with 2 decimal places in a 4 digit wide field and a left justified decimal number in a 6 digit wide field to a file
printf(“\t%d\t%d\t%6.2f\t%s\n”, id_no, age, balance, name >> "account")
appends a TAB separated number, number, 6.2 float and a string to a file
prints number and a string
prints a left justified string in a 10 character wide field and a float with 2 decimal places in a six character wide field
prints a right justified string in a 10 character wide field, a left justified float with 2 decimal places in a 4 digit wide field and a left justified decimal number in a 6 digit wide field
prints a TAB separated number, number, 6.2 float and a string

38. Built-In Functions Arithmetic String Output Special
sin, cos, atan, exp, int, log, rand, sqrt
String
length, substitution, find substrings, split strings
Output
print, printf, print and printf to file
Special
system - executes a Unix command
system(“clear”) to clear the screen
Note double quotes around the Unix command
exit - stop reading input and go immediately to the END pattern-action pair if it exists, otherwise exit the script

39. Built-In Arithmetic Functions
Return Value
atan2(y,x)
arctangent of y/x (-p to p)
cos(x)
cosine of x, with x in radians
sin(x)
sine of x, with x in radians
exp(x)
exponential of x, ex
int(x)
integer part of x
log(x)
natural (base e) logarithm of x
rand()
random number between 0 and 1
srand(x)
new seed for rand()
sqrt(x)
square root of x

40. Built-In String Functions
Description
gsub(r, s)
substitute s for r globally in $0, return number of substitutions made
gsub(r, s, t)
substitute s for r globally in string t, return number of substitutions made
index(s, t)
return first position of string t in s, or 0 if t is not present
length(s)
return number of characters in s
match(s, r)
test whether s contains a substring matched by r, return index or 0
sprint(fmt, expr-list)
return expr-list formatted according to format string fmt

41. Built-In String Functions
Description
split(s, a)
split s into array a on FS, return number of fields
split(s, a, fs)
split s into array a on field separator fs, return number of fields
sub(r, s)
substitute s for the leftmost longest substring of $0 matched by r
sub(r, s, t)
substitute s for the leftmost longest substring of t matched by r
substr(s, p)
return suffix of s starting at position p
substr(s, p, n)
return substring of s of length n starting at position p