Korn Shell Script Writing & HPC

Korn Shell Script Writing & HPC
Science & Technology Support Group High Performance Computing OSC (Ohio Supercomputer Center) 1224 Kinnear Road Columbus, OH

Table of Contents Decision-Making statements Introduction
What is a shell script? Script Programming Language Why use shell scripts? Korn shell scripts References The Evolution of a Script Sampling of Script Features A simple script A more versatile Script Creating Files within a Script A documented Script Script Basics Creating a shell script Executing a shell script Echoing script commands Script Variables User-defined Variables Storing a Value into a Variable Assignment Operator = Read Command Command Substitution Null Variables Special Shell Variables Positional Parameters Referencing Variables Parameter Substitution Exercises Decision-Making statements Flow Control Exit Status If-Then-Else command Conditional Tests Logical Operators Illustrative Examples Case statement

Table of Contents Script Functions HPC Designed Scripts
Looping Statements for Statement select Statement while Statement Until Statement Exercises Working with Integers Arithmetic Operations Operator Precedence Numerical Bases String vs Integer Variables Script I/O Shell Redirection Operators "Here-is" Files print read Quoting Communicating with Running Scripts Sending Signals How Scripts Receive Signals Script Functions Why Use Functions? Syntax & Use Auto-Loading Function Definition Location Command Precedence HPC Designed Scripts Timing Loops Debugging Options Batch Jobs Performance Tool Preprocessing Multiple Tools runs Multiple Tool Reports File Renaming

Introduction What is a shell script? Script Programming Language
Why use shell scripts? Korn Shell scripts References

What is a shell script? A shell script is a collection of Unix commands entered into a file. When the file (script) is executed, the commands are executed in order. Running a shell script is EXACTLY equivalent to interactively entering each command For example, say a user always does the same four steps before printing out a file: making the file owner- writable, using vi to check/change the file, printing it out, and then making the file unwritable (for safety reasons) An interactive session of a user following this procedure for the file comp.txt is shown on the next page

Interactive editing and printing
$ chmod u+w comp.txt $ vi comp.txt Here is a table summarizing OSC computers as of today. Supercomputer # Processors Proc Speed Ideal Calc. Itanium Cluster MHz GFLOPS Cray SV MHz GFLOPS Origin MHz GFLOPS IA-32 Intel MHz GFLOPS Sun Fire MHz GLOPS "comp.txt" 11 lines, 535 characters $ lpr -Pps1 comp.txt $ chmod u-w comp.txt

Making the viprint script
To make the previous editing and printing procedure into a script, all the user has to do is type the same commands they use interactively into a file Say the file is called viprint, it looks like this: $ cat viprint chmod u+w comp.txt vi comp.txt lpr -Pps1 comp.txt chmod u-w comp.txt After viprint is made executable, whenever the user wants to execute these four commands they simply type viprint as if it was a “normal” Unix command $ viprint

Script Programming Language
The real power is writing shell script comes from using special shell commands and variables that essentially make your script act like a program In this course, you will learn about the programming statements which can be used in scripts, including: Initializing and using your own script variables. Both single valued and arrays Performing operations with variables Decision-making statements (if-then-else, etc) Looping statements (for,etc) Defining and using functions in scripts Script I/O commands and procedures Above bullet reads like a “table of contents” for a text on a “traditional” programming languages such as Fortran or C, although … Script programming has less capabilities and commands

Why use shell scripts? Convenience: Run a number of Unix commands by typing a single script name Command log: Script can permanently contain a detailed set of commands with numerous options. (If interactive, complex command is gone) Usefulness: Write a script with programming commands to give the script a number of capabilities/options Take “full” use of what the shell has built-in to offer you Considering a script as a program, no compiler, loader, or libraries needed to make it executable Portability: If a script works in the shell on your machine it will work on the same shell on a different machine Easy to debug: Small number of programming statements which are relatively simple Easy to modify, expand, and customize: Just type in the new commands Run-type interaction: Can send signals to running scripts

Korn shell scripts Several types of Unix shells exist, each with their own script programming languages: Bourne (sh), Born Again shell (bash), Korn shell C shell (csh), TENEX C shell (tcsh) Features you would expect from a well-written shell: Convenient interactive features: command history, command-line editing, and filename completion Powerful scripting capabilities including reliability and efficiency This course will discuss Korn shell scripting It has all the convenient interactive features It is backward compatible with the Bourne shell (the first shell) Most scripts are sh Start-up sequence for ksh same as sh only enhanced Unix utilities and administration commands depend on sh It does not have the scripting bugs that the C shell has

Korn shell scripts What if I already use a different shell on my machine? Could switch to Korn shell (use the chsh command if you can) Keep your existing shell as your interactive shell and write your scripts as Korn shell scripts (Easily done: one line needed in the script)

References Internet URLs Usenet Newsgroups Anonymous ftp
nacphy.physics.orst.edu/rubin/melanie/node144.html (Tutorial) gonzo.tamu.edu/csh.whynot.html (Bugs in C shell scripting) Usenet Newsgroups comp.unix.shell comp.unix.questions Anonymous ftp athos.rutgers.edu/~patra/vi.html (shell-101.BetaA.Z) rtfm.mit.edu:/pub/usenet/news.answers/unix-faq/faq (Unix FAQ) OSC Unix-Series workshops (oscinfo.osc.edu) Basic Unix Intermediate Unix Introduction to Perl

The Evolution of a Script
Sampling of Scripting Features A Simple Script A More Versatile Script Creating Files within a Script A Commented Script

Sampling of Scripting Features
Next few slides will act as a course overview as we take a simple script and keep extending its capabilities The more advanced scripts will demonstrate some of the capabilities of Korn Shell scripting Don’t worry about understanding all the details at this point. A complete explanation of the scripting commands used is given in later chapters

A Simple Script In the script called triangle, a C program is compiled and then executed. The program calculates the area of the triangle and reads in the lengths of the sides from a file and outputs the result to another file. Here is an example of a user running the script: $ cat triangle cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o area area.c area < sides.in > area.dat cat area.dat rm area $ ls -l triangle -rwxr--r dje appl May 3 12:09 triangle $ cat sides.in 8 3 7 $ triangle Area of Triangle is

A Simple Script Limitation of the simple triangle script
Script only works for the one area program. What if you want to run other triangle-related programs with this same script ?

A More Versatile Script
Scripts can take arguments on the command line when they are run. The first argument is passed to a variable with a special name: $1. We can use this feature to make the triangle script work for any program. Here is the resulting new script in action: $ cat triangle cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c $1 < sides.in > $1.dat cat $1.dat rm $1 $ triangle area Area of Triangle is $ triangle perim Perimeter of Triangle is

A More Versatile Script
Limitation of versatile script: Scripts should be as independent of possible of “outside” existing files (for portability and ease of use). Since the sides.in file is so short, we should create the file within the script

Creating Files Within a Script
The easiest method for creating a small “internal” file is to use “Here-is” Unix redirection. After it is used, this temporary file is removed: $ cat triangle if [[ $2 = debug ]]; then cc -g -o $1 $1.c print "$1 meant to be debugged" else cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c cat << EOF > sides.in 8 3 7 EOF $1 < sides.in > $1.dat cat $1.dat rm $1 sides.in fi $ triangle perim Perimeter of Triangle is

A Commented Script A bad script writing style has been shown in all incarnations of the triangle script: the lack of comments. Comments are critical to another user (or you) in understanding what the script does. The special “comment” on the first line of script actually specifies the shell to be used for the script commands. This is required for users whose interactive and script shells differ. On the next page is the timing script just discussed but now with proper commenting

A Commented Script $ cat triangle #!/bin/ksh
# Compiling C code found in first parameter cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c # Using redirection to make input file cat << EOF > sides.in 8 3 7 EOF # Using while loop to run the executable 17 times let i=0 while (( i < 17 )); do $1 < sides.in >> $1.dat let i=i+1 done # Checking output cat $1.dat # Clean up files no longer needed rm $1 sides.in

Script Basics Creating a Shell Script Executing a Shell Script
Echoing Script Commands

Creating a Shell Script
A shell script is an executable file which is executed by the shell line-by-line. It can contain the following: UNIX commands shell programming statements comments Create using editor of choice Can include a #! construct in first line of script to override login shell #!/bin/ksh uses Korn shell to execute script #!/bin/csh uses C shell to execute script Recommend always put this in every Korn shell script. Makes it clear what the file is. "file" Unix command will even identify it as a Korn Shell script Name should be short and descriptive

Executing a Shell Script
There are 3 ways to execute a shell script: 1."dot" method $ . scriptname 2."just the name" method $ scriptname 3.in the background $ scriptname &

Method 1 runs the command as if you typed them in on the command line No ksh subprocess created Note that methods 2 and 3 require: execute permission for scriptname chmod +x scriptname current directory (.) must be in PATH or else must use ./scriptname

Demonstration of execution methods: $ ps PID TTY TIME CMD 31612 ttyq5 0:00 sh 31631 ttyq5 0:00 ps $ ls -l showsh -rw-r--r dje appl Nov showsh $ cat showsh print Executing the shell script showsh ps $ . showsh Executing the shell script showsh 31628 ttyq5 0:00 ps

$ chmod u+x showsh $ showsh Executing the shell script showsh PID TTY TIME CMD 31578 ttyq5 0:00 ps 31612 ttyq5 0:00 sh 31635 ttyq5 0:00 sh

Echoing Script commands
For understanding or debugging scripts it is often convenient to see the script commands shown on the monitor along with the output they produce. Accomplished using the set -x command at the beginning of the file To turn off the echoing, make the command set +x (counter-intuitive?) Also useful in batch job files and - on occasion - interactively On the next page a demonstration of the power of set -x is shown

$ see /homea/dje/ksh_script boba.osc.edu:0.0 a_opt.txt cba_opt.txt item.set.3.txt pid.txt b_opt.txt fant4.txt item.set.4.txt showsh.txt c_opt.txt item.intmod.txt no_opt.txt trap.txt $ cat see #!/bin/ksh pwd echo $DISPLAY ls *.txt $ cat see2 set -x

$ see2 + pwd /homea/dje/ksh_script + echo boba.osc.edu:0.0 boba.osc.edu:0.0 + ls a_opt.txt b_opt.txt c_opt.txt cba_opt.txt fant4.txt item.intmod.txt item.set.3.txt item.set.4.txt no_opt.txt pid.txt showsh.txt trap.txt a_opt.txt cba_opt.txt item.set.3.txt pid.txt b_opt.txt fant4.txt item.set.4.txt showsh.txt c_opt.txt item.intmod.txt no_opt.txt trap.txt

Script Variables User-defined Variables
Storing a Value into a Variable The assignment operator = The read command Command Substitution Null Variables Special Shell Variables Positional Parameters Referencing Variables Parameter Substitution Exercises

User-defined Variables
The Korn shell includes the following types of variables: User-defined variables Special shell variables User-defined variables can be initialized, used and changed from the command line or from within a shell script. Not "declared" like in a programming language. Just type the name of the variable Korn shell scripts use string and integer variables A variable name can consist of the following: letters, digits, and the underscore character first character of a variable name must be a letter or an underscore character A variable can be made read-only: once it is assigned a value, the value cannot be changed "Archaic" method: readonly variable_name Alternate method: typeset -r variable_name

Storing a Value into a Variable
There are several methods for variables to obtain values. The assignment operator = Enter the name that you have chosen for the variable followed by an equal sign and then the value that you want to store in the variable. To use the a variable's value, put the $ operator in front of it's name. (Read the $ as "contents of") The typeset command can also be used for variable assignment. More up-to-date procedure, typeset has many options. To assign a value to an integer variable, the assignment command must be proceeded by the word let On the next page, a script is shown that demonstrates all these approaches

$ cat assign street=Elm print street <-- Most common error when using variables print $street my_name="John Smith" echo $my_name neph=Chris Sellgren print $neph typeset card=ace; print $card let num=57; echo $num let num=num+3; print $num $ assign street Elm John Smith assign2.ksh[9]: Sellgren: not found ace 57 60

Second method: read in a value for a variable So far have used print and echo (archaic) commands for printing out all the strings following them and then a carriage return There is a complimentary command read which input a value into variables. The user type in as many strings as there are variables on the read line, and then hits carriage return On the next page is a demo script indicating how the read command works.

$ cat input print Please enter a ship name read ship print Please enter your full name read fname lname print Captain $lname, welcome aboard the $ship echo May I call you $fname? print Enter an integer; read num let num=2*num echo Twice your number is $num $ input Please enter a ship name Redoubt Please enter your full name Horatio Hornblower Captain Hornblower, welcome aboard the Redoubt May I call you Horatio? Enter an integer 27 Twice your number is 54

Use command substitution. The string(s) that a Unix command returns may be put into a script variable The following syntax is as follows: var_name=$(UNIX command) As with all the script commands, command substitution can be done interactively as well On the next page is a demo script illustrating this technique

136:oscb$ cat command dir=$(pwd) print I am in the directory $dir weekday=$(date +%A) print It is $weekday files=$(ls -lt | wc -l) print There a $files files in this directory me=$(whoami) print I am $me computer=$(hostname) print My computer is called $computer logged_on=$(who | cut -f1 -d' ') print These people are also on $computer: $logged_on 137:oscb$ command I am in the directory /home/dje It is Tuesday There a 277 files in this directory I am dje My computer is called oscb These people are also on oscb: osu2796 opr006 paul ysu039 osu2796 jimg mmittal jimg mmittal osu2224 osu2224 srb cls038 osu1938 jeff dje

Null Variables A variable can be set to a null value, even if previously assigned, using any of the following methods. There should be no spaces preceding or following the equal sign. If the contents of null character variables are printed out, a blank appears. If the contents of a “nullled” integer variable is outputted, a 0 is displayed. $ name= $ name='' $ name="” $ let num= $ unset varname

Null Variables All variables that don't exist are assumed null unless set -o nounset is used. Then the shell will indicate an error when an undefined variable is encountered $ unset name $ set -o nounset $ print $name ksh: name: parameter not set $ set +o nounset

Special Shell Variables
In addition to your own user-defined variables, other variables defined by the shell for the user are called special shell variables and are most useful These variables are set and updated automatically by the shell. Their values cannot be changed, but they may be referenced. The special shell variables of most use are identified by the following symbols: $# $- $? $$ $! $0 $*

The variable $# contains the number of arguments typed on the command line. $ cat numargs print The number of arguments is $# $ numargs The number of arguments is 0 $ numargs The number of arguments is 5 $ numargs "Hello World" The number of arguments is 1

The variable $- contains the shell flags (options) of the current shell. $ print $- isum $ set +u $ echo $- ism

The variable $? contains the exit status of the last command. If the exit status is 0, the command ran successfully. If it is non-zero, there was a problem $ ls file1 data account.txt $ rm file1 $ print $? $ rm dataa dataa: No such file or directory 2 We will see exit status again. It will be used to define true and false. Also, the user can set the exit status in their script by using the exit command

The variable $$ contains the Process ID (PID) of the current shell process. $ cat pid #!/bin/ksh ps print $$ $ pid PID TTY TIME CMD 34738 ttyq5 0:00 ps 34744 ttyq5 0:00 sh 34765 ttyq5 0:00 pid 34765 $ . pid 34743 ttyq5 0:00 ps 34744

The variable $! contains the process ID number of the last command sent to the background. $ sleeper & [1] $ ps PID TTY TIME CMD 29144 ttyq5 0:00 sleeper 29150 ttyq5 0:00 sh 29160 ttyq5 0:00 ps 29167 ttyq5 0:00 sleep $ compress moby_dick & [2] $ print $! 29136

The variable $0 contains the name of the script currently being execute $cat old_script print The name of this script is $0 $old_script The name of this script is old_script $mv old_script new_script $new_script The name of this script is new_script The variable $* contains all the arguments on the command line. $ cat args print The arguments are: $* $ args bob dave The arguments are: bob dave

Positional Parameters
For a script, the command line arguments are also called positional parameters. Each positional parameters refers to one individual argument on the command line. The ten positional parameters available to the script writer are referenced as follows: $1 $2 $3 $4 $5 $6 $7 $8 $9 The parameter $1 contains the first argument, $2 the second argument, and so on. Consider the following script: $ cat parms print Arg 1 is: $1 print Arg 2 is: $2 print Arg 3 is: $3 print Arg 4 is: $4 print Arg 5 is: $5

$ parms 2001 A Space Odyssey Arg 1 is: 2001 Arg 2 is: A Arg 3 is: Space Arg 4 is: Odyssey Arg 5 is: $ parms "Space, the final frontier" Arg 1 is: Space, the final frontier Arg 2 is: Arg 3 is: Arg 4 is:

Use the set command to change positional parameters. It replaces existing positional parameters with new values $ cat newpos print starting args are $* print number of args is $# print arg 1 is $1 print arg 2 is $2 set NCC 1701 Enterprise print new args are $* print arg 3 is $3

Here is an example of the use of the newpos script: $ newpos 76 trombones starting args are 76 trombones number of args is 2 arg 1 is 76 arg 2 is trombones new args are NCC 1701 Enterprise number of args is 3 arg 1 is NCC arg 2 is 1701 arg 3 is Enterprise

What if there are more than 10 positional parameters? $ cat ten_args print arg 10 is $10 $ ten_args a b c d e f g h i j arg 10 is a0 Use the shift command to perform a shift of the positional parameters n positions to the left. (Default n=1) The shell variables $# and $* also change when the shift command is used (along with the values of the positional parameters. Once a default shift is performed, the first parameter is discarded unless it is saved in a user-defined variable

Examples of shift usage: $ cat shift_it print $#: $0: $* shift shift 3 print $#: $0 $* $ shift_it It was the best of times, it was the worst of times. 12: shift_it: It was the best of times, it was the worst of times 11: shift_it: was the best of times, it was the worst of times 10: shift_it: the best of times, it was the worst of times 7: shift_it: times, it was the worst of times $ cat ten_args arg1=$1 print $arg1 $* $ ten_args

Referencing Variables
So far we have seen that the syntax for referencing a variable (obtaining its contents) is $var_name An alternate syntax which has more capabilities is ${var_name} This syntax: Allows a script to easily use more than 10 positional parameters Allows for more general use of a variable contents Allows for the number of characters in the variable’s string contents to be determined Allows for parameter substitution: replacement of the value of a variable with an different value based on specific conditions and events.

Greater than 10 positional parameters: $ cat moreargs #!/bin/ksh print arg 10 is $10 print arg 10 is ${10} print arg 11 is $11 print arg 11 is ${11} $ moreargs what rough beast, its time come at last, slouches toward Bethlehem arg 10 is what0 arg 10 is toward arg 11 is what1 arg 11 is Bethlehem

More general use $ cat copy set -x file=lone cp $file $filewolf cp $file ${file}wolf ls lone* $ copy + file=lone + cp lone Insufficient arguments Usage: cp [-aDfirRp] [-b size] [-tP -[e size]] f1 f2 cp [-aDfirRp] [-b size] [-tP -[e size]] f1 ... f2 dir + cp lone lonewolf + ls lone lonewolf lone lonewolf Length of a Variable’s contents $ cat length book="Pillars of the Earth" print length of book= ${#book} $ length length of book= 20

Parameter Substitution
Misnamed: should be variable substitution Four syntaxes for parameter substitution ${var_name: - value} ${var_name: = value} ${var_name: + value} ${var_name: ? value} The meanings and typical usages of these four methods are shown on the following pages

${parameter:-value} If parameter exists and isn't null, return its present contents; otherwise return value. Purpose: returning a default value if the variable is undefined. If parameter is null, the value returned is temporary. The null value of the variable is not permanently altered Interactive Examples: $ count="" $ print You have ${count:-5} cards You have 5 cards $ print $count $ FOO=/usr/local/bin/xx $ print edit file ${FOO:-$HOME/xx} edit file /usr/local/bin/xx $ print $FOO /usr/local/bin/xx

As the name implies, this particular syntax of parameter substitution can be used positional parameters. Consider the following script that searches for a file: $ cat fnddemo #!/bin/ksh set -x find ${2:-$HOME} -name $1 -print $ fnddemo bk.txt.Z . + find . -name bk.txt.Z -print ./bk.txt.Z $ fnddemo dale11.ps + find /homea/dje -name dale11.ps -print /homea/dje/dale11.ps

${parameter:=value} If parameter exists and isn't null, return its present contents; otherwise set it to value and then return value Purpose: Setting a variable to a default value if it is undefined. This form of parameter substitution cannot be used with positional parameters because their values can only be changed with the set command Interactive Example: $ count="" $ print You have ${count:=5} cards You have 5 cards $ print $count 5

${parameter:+value} This substitution format acts in an opposite manner to the previous two. If parameter exists and is not null, return value; otherwise return null. Purpose: Testing for the existence of a variable Interactive Example: $ cat trace set -x trace=$1; print ${trace:+"Trace mode on"} $ trace y + trace=y + print Trace mode on Trace mode on $ trace2 + trace= + print $

${parameter:?message} If parameter exists and isn't null, return its contents; otherwise print scriptname: parameter: message and abort the current script. Omitting message produces the default system message. Purpose: Catching errors that result from variables being undefined which need to be. Examples: $ cat fnd2 find ${2:-$HOME} -name ${1:?"No file specified"} -print $ fnd2 fnd2: 1: No file specified $ cat fnd3 find ${2:-$HOME} -name ${1:?} -print $ fnd3 fnd3: 1: parameter null or not set

NOTE: In all forms, the colon (:) is optional. If omitted, the operator tests for existence only, i.e., change "exists and isn't null" to "exists" in each definition.

Advanced Pattern Matching
Korn shell enhancements to the set of pattern-matching operators: Operator Meaning ${variable#pattern} If the pattern matches the beginning of the variable's value, delete the shortest part that matches and return the rest. ${variable##pattern} If the pattern matches the beginning of the variable's value, delete the longest part that matches and return the rest. ${variable%pattern} If the pattern matches the end of the variable's value, delete the shortest part that matches and return the rest. ${variable%%pattern} If the pattern matches the end of the variable's value, delete the longest part that matches and return the rest. Mnemonic: # matches the front because number signs precede numbers; % matches the rear because percent signs follow numbers.

Advanced Pattern Matching
Typical use of the above pattern-matching operators is stripping off components of pathnames such as directory prefixes and filename suffixes. Example: Assume the variable TEST has the value /home/job/src/long.file.name then: Expression Result ${TEST##/*/} long.file.name ${TEST#/*/} job/src/long.file.name ${TEST} /home/job/src/long.file.name ${TEST%.*} /home/job/src/long.file ${TEST%%.*} /home/job/src/long

Exercises 1. Write a script called lsc which executes the command ls -C. To execute this command you must give the full path name to your lsc script. Make the lsc shell script executable and run it. 2.Write a script called b which changes your current directory to /bin, displays the current directory, and then executes the lsc script created above. Make the b script executable and run it. What is your current directory when the b script is finished executing? Why? 3.Write a script called ex_on, which turn on execute permission on the first argument on the command line. 4.Modify ex_on to turn on execute permission on all arguments on the command line. 5.Write a script called 12 that prints the twelfth argument on the command line.

Exercises 6. Suppose you keep a file of your home coin collection that keeps track of how many coins you have in a given category. Lines in this file look like: 62 U.S. proofs 11 U.S. pennies ( ) 36 U.S. pennies ( ) 9 U.S. nickels ( ) Write a script that prints the N types of coins of which you have the most. The default for N is 10. The program should take one argument for the name of the input file and an optional argument for how many lines to print.

Decision Making Commands
Flow Control Exit Status II if/then/else Command Condition Tests Logical Operators Negation Operator Illustrative Examples Case Statement Exercises

Flow Control Flow control gives a programmer the power to specify that only certain blocks of shell commands are executed Korn shell supports the following decision making constructs: if/else Execute a list of statements if a certain condition is/is not true case Execute one of several lists of statements depending on the value of a variable

Exit Status II For decision making, we will look at the exit status in a different way then we did before The constructs if, while, until and the logical AND (&&) and OR (||) operators use exit status to make logical decisions: A 0 value is a logical "true" (success) A nonzero is a logical "false" (failure) There are built-in true and false commands which you can use. The true command returns an exit status of 0, the false command returns an exit status of 1.

Exit Status of a Script A script, like any other process, sets an exit status when it finishes executing. Shell scripts will finish in one of the following ways: Abort - If the script aborts due to an internal error, the exit status is that of the last command (the one that aborted the script). End - If the script runs to completion, the exit status is that of the last command in the script Exit - If the script encounters and exit command, the exit status is that set by that command Syntax for the exit command is exit [num]. When the exit command is encountered the script ends right there and the exit status is set to num

if / else Simplest type of decision making construct is the conditional embodied in Korn shell's if statement. Syntax: if condition then statements [elif condition <-- can use multiple elif clauses then statements . . .] [else statements] fi <-- must be by itself on the final line of the construct

if / else The if statement uses an exit status to determine whether or not to execute the commands. Statements are executed only if the given condition is true. If one or more elifs are used, the last else clause is an "if all else fails" part.

Condition Tests The if construct can only test exit status but that doesn't limit you to checking only whether commands ran properly or not. Using the [[ ]] operator, many different attributes can be tested: string comparisons file attributes arithmetic conditions [[ condition ]] just returns an exit status that tells whether condition is true or not [[ ]] surround expressions that include various types of relational operators.

Condition Tests: Strings
String comparison operators: Relational Operator True if ... str = pat str matches pat. str != pat str does not match pat. str1 < str2 str1 is less than str2.* str1 > str2 str1 is greater than str2.* -n str str is not null (has length greater than 0). -z str str is null (has length 0). *based on the ASCII value of their characters str refers to an expression with a string value, and pat refers to a pattern that can contain wildcards.

String Comparison Examples
$ cat compstr x=castle; y=cast* if [[ $x = $y ]] then print $x matches $y else print $x does not match $y fi $ compstr castle matches cast* $ cat compstr1 x=hello; y=helper if [[ $x > $y ]] print "$x is > $y" print "$x is not > $y" $ compstr1 hello is not > helper

String Comparison Examples
$ cat compstr2 if [[ -n $1 ]] then print First argument is $1 fi if [[ -z $1 ]] print Where is the first argument? $ compstr2 Logan First argument is Logan $ compstr2 Where is the first argument?

Condition Tests: Files
File attribute checking: Relational Operator True if ... -a file file exists -d file file is a directory -f file file is a regular file -r file there is read permission on file -s file file is not empty -w file you have write permission on file -x file you have execute permission on file, or directory search permission if it is a directory -O file you own file -G file your group id is same as file's file1 -nt file2 file1 is newer than file2 file1 -ot file2 file1 is older than file2 There are 21 such operators in total.

File Checking Demo Directory
Consider a directory that contains the following files and subdirectories $ls -lrt -rwxr--r dje appl Jun file -rw-r--r dje appl Jun my_prog -rw-r--r dje appl Jun my_scr.1 drwxrwsr-x 2 dje appl Jul scripts -rw-rw-r dje appl Sep slide1.ps drwxr-sr-x 2 dje appl May 18 11:54 probs drwxr-sr-x 2 dje appl May 18 11:54 solutions -rw-r--r dje appl May 18 11:54 court.msg -rw-r--r dje appl May 18 11:55 heel -rw-r--r dje appl May 18 11:57 greed -rwxr-xr-x 1 dje appl May 18 12:01 file.c -rwxr-xr-x 1 dje appl May 18 12:01 file.f90 -rw-r--r dje appl May 18 12:04 typescript

File Checking Examples
$ cat fexist if [[ -a $1 ]] then print File: $1 does exist else print File: $1 does not exist fi $ fexist sort.c File: sort.c does not exist $ cat ffull if [[ -s $1 ]] then print $1 has something in it print $1 is empty $ ffull heel heel is empty $ cat fx if [[ -x $1 ]] then print I can execute $1 print I cannot execute $1 $ fx file I can execute file

Condition Tests: Integers
Need to use a new test operator for integer comparisons (( … )) instead of [[ … ]] Relational Operator Meaning < less than > greater than <= less than or equal >= greater than or equal == equal != not equal

Integer Comparison Examples
$ cat intcomp if (( $1 < $2 )) then print $1 is less than $2 fi if (( $1 <= $2 )) then print $1 is less than or equal to $2 if (( $1 == $2 )) then print $1 is equal to $2 if (( $1 > $2 )) then print $1 is greater than $2 if (( $1 >= $2 )) then print $1 is greater than or equal to $2 if (( $1 != $2 )) then print $1 not equal to $2

Integer Comparison Examples
$ intcomp 3 89 3 is less than 89 3 is less than or equal to 89 3 not equal to 89 $ intcomp 13 13 13 is less than or equal to 13 13 is equal to 13 13 is greater than or equal to 13

Logical Operators || is a logical OR
Two special symbols allow the user to combine relational tests into more complex conditions && is a logical AND || is a logical OR How they work: statement1 && statement2 statement2 will execute only if statement1 was successful (returned an exit status of 0) statement1 || statement2 statement2 will execute only if statement1 was not successful (returned a nonzero exit status).

Logical Operators The logical operators && and || are commonly used with if/else commands Examples: $ ls -l -r-xr-xr-x 1 dje appl May 18 15:13 intcomp -rw-r--r dje appl May 18 15:12 length.txt $ cat setw #!/bin/ksh if [[ -f $1 && -x $1 ]] then print You are getting write permission for $1 chmod u+w $1 fi $ setw length.txt $ setw intcomp You are getting write permission for intcomp -rwxr-xr-x 1 dje appl May 18 15:13 intcomp

Logical Operators $ cat gone #!/bin/ksh
if [[ $1 = quit || $1 = bye ]] then print You have closed correctly else print Improper closing word exit 11 fi $ gone bye You have closed correctly $ echo $? $ gone quit $ gone end Improper closing word 11

The Negation Operator If an exclamation point ! is put before any of the conditions discussed, it reverses the “truth or falseness” of the condition. [[ ! -d probs ]] returns true (exit status 0) if probs is not a directory [[ ! Bob = ?o? ]] returns false (exit status 1) even though Bob matches the pattern

Combining Different Types of Conditions
If you want to combine an integer comparison with file or string comparison within the same condition, you must do two things: Use [[ … ]] for the test operator Use the integer relational operators shown below Relational Operator Comparison -lt less than -le less than or equal -eq equal -ge greater than or equal -gt greater than -ne not equal

Combining Conditionals Example
$ cat fileint #!/bin/ksh let num=$2 if [[ $1 < tact || num -ge 3 ]]; then print Pippin else print Merry fi $ fileint ace 45 Pippin $ fileint vice 2 Merry

Illustrative Examples
Test operator run interactively $ [[ -z "" ]] $ print $? $ [[ -z foo ]] 1 Treating integers strings as strings: compressed format $ if [[ 6 > 57 ]] ; then print huh?;fi huh?

The following script sets user execute permission on an ordinary, file that is not a directory: if [[ -f $1 && ! -d $1 ]]; then chmod u+x $1 fi

The following script removes the first file if it's older than the second file and the variable KEY is non-null: if [[ $1 -ot $2 && -n $KEY ]] then /bin/rm $1 fi

The following script compares two files and if no differences are found, removes the second file. Also helpful USAGE programming style: USAGE="Usage:\t $0 file1 file2" if [[ $# -ne 2 ]]; then print ${USAGE} exit 1 fi diff $1 $2 > /dev/null if [[ $? -eq 0 ]]; then /bin/rm $2 print $2 removed, $1 kept else print $1 and $2 differ

Case command Provides a multiple choice decision structure. When executed one of several statement blocks will be run. Replacement for the “elif-ladder” Lets you test strings against patterns that can contain wildcard characters. Syntax: case expression in pattern1) statements ;; pattern2) ... esac

Case Operation If expression matches one of the patterns, its corresponding statements are executed If there are several patterns separated by pipes, the expression can match any of them in order for the associated statements to be run ( | acts as logical or ) Patterns are checked in order for a match; if none is found, nothing happens When the double colons (;;) are encountered at the end of the one of the statement lists, the case terminates

Case Examples (Could have also used *.[cf] wildcard construct above)
Here's a simple script which moves C and Fortran source files to one directory and object code files to another: case $file in *.c|*.f) /bin/mv $file ${HOME}/src ;; *.o ) /bin/mv $file ${HOME}/obj;; * ) print $file not moved ;; (Default value or USAGE) esac (Could have also used *.[cf] wildcard construct above)

Case Examples The case statement is often used to specify options for a shell script. Here is a shell script called dt_fmat that allows the user to enter options that affect the way the date is displayed: case $1 in -d) print -n Date: date +"%a %h %d" ;; -t) print -n Time: date +"%T" ;; -w) print -n Weekday: date +"%a" ;; -y) print -n Year: 20 date +"%y" ;; -m) print -n Month: date +"%h" ;; *) print -n Date: date +"%a %h %d" print -n Time: esac

Case Usage $ date Sat May 19 12:01:57 EDT 2001 $ dt_fmt -d
$ dt_fmt -t Time:12:02:05 $ dt_fmt -w Weekday:Sat $ dt_fmt -y Year: 2001 $ dt_fmt -m Month:May $ dt_fmt Time:12:02:54

“Elif-Ladder” vs Case The following script shows and “elif-ladder” and a case statement that do the exact same thing. Note how the case statement has a simpler syntax if [[ $1 = cat || $1 = dog ]]; then print You have a normal pet elif [[ -z $1 ]]; then print You have no pets else print You have a wierd pet fi case $1 in cat|dog) print You have a normal pet ;; '') print You have no pets ;; *) print You have a wierd pet ;; esac Advantage of “elif-ladder”: not limited to string comparisons

Exercises 1.Write a script called char that checks a single character on the command line, c. If the character is a digit, digit is displayed. If the character is an upper or lowercase alphabetic character, letter is displayed. Otherwise, other is displayed. Have the script print an error message if the argument c is more than one character in length. 2.Write a script called mycopy that copies a source file to a directory. Add a check to see if the source file exists. If the source file does not exist, print an error message. 3.Write a script called mydir that prints the message File is a directory if the file is a directory.

Exercises 4.Write a script called ver that accepts one argument on the command line indicating the name of a file. The file is copied to another file with the same name with the addition of the extension .v2. Also, the line #Version 2 is added to the top of the file. 5.Execute the ver script on itself, creating a new version of the file called ver.v2. 6.Rewrite ver.v2 to accept a possible second argument. If two arguments are entered, the file specified by the first argument is copied to a file with the name of the second argument. If no second argument is entered, the file is copied to another file with the same name, adding the extension .v2. In either case, the line #Version2 is added to the top of the file.

Looping Commands for select while until Exercises

for Loop Command The for loop allows a section of commands to be run a fixed number of times During each time through an iteration, the loop variable is set to a different value. Syntax: for name [in list] do statements that can use $name done where name is the loop variable which can be called anything list is a list of strings (defaults to $*) separated by spaces Operation: name is set to each string in list, in order, for each iteration; the number of iterations equals the number of names in list.

for Loop Examples Outputting a Phrase $ cat simple
for word in This is a test do print $word done $ simple This is a test <-- Omitting the in This is a test phrase and instead running the script as $ simple This is a test would yield the same output.

for Loop Examples The list can be the contents of a local variable:
Check to see who is logged on the machines listed in the variable SYSTEMS="myrtle gull sandy newport daytona": for sys in $SYSTEMS do print done

for Loop Examples list can contain shell wildcards and command substitution as well: $ cat file1 for i in * do print $i done $ file1 file1 file2 $ cat file2 for i in $(ls) $ file2

for Loop Examples Can explicitly make the list all the command line arguments (also notice the alternate syntax): $ cat upper #!/bin/ksh for bug in $* ; do typeset -u day=$bug print $day done $ upper Robson Bond Green ROBSON BOND GREEN

select Command Syntax: select name [in list]
select allows you to generate simple interactive ASCII menus easily. Syntax: select name [in list] do statements that can use $name done This is the same syntax as the for loop except for the keyword select. As with for, in list defaults to $* if omitted.

select Operation Each item in list is displayed as the choice in the menu with a number preceding it. At the bottom of the menu the PS3 prompt is displayed and script is waiting for the user to select a choice by entering its number. (A default value for the PS3 prompt is provided by the system but the user can change it) The selected choice is stored in name and the selected number in a system-predefined variable called REPLY The statements in the body of the do loop are run Repeats the process forever; exit loop with break statement (or user can issue ctrl-d)

select Example The following script termselect allows you to select a terminal setting: PS3='terminal? ' select term in vt100 vt220 xterm do if [[ -n $term ]]; then TERM=$term print TERM is $term else print invalid choice fi done

select Example $ termselect 1) vt100 2) vt220 3) xterm terminal? 4
invalid choice terminal? 3 TERM is xterm

select Example $ cat travel select i in Past Present Future; do
print You have picked choice $REPLY print "You will go to the $i" break done $ travel 1) Past 2) Present 3) Future #? 1 You have picked choice 1 You will go to the Past #? 3 You have picked choice 3 You will go to the Future $

while Command Allows a section of code to be run repetitively while a certain condition holds true. Syntax: while condition do statements ... done Operation: while condition remains true (exit status 0) the body of the while loop is run [[ ]]operator can be used here as with if to specify the condition Beware of creating an infinite loop (condition must become false at some point). If condition is false when a while loop is first encountered, the loop will not be executed even once

while Example $ cat args Example: print out positional parameters:
while [[ $# -ne 0 ]]; do print $1 shift done $ args There are 7 “deadly sins” There are 7 deadly sins

while Example Example: Using a while as a counting loop:
$ cat whileint print $#: $3 let i=1 while (( i <= 5 )); do shift let i=i+1 done $ whileint 10: 5 5: 15

until Command Allows a section of code to be run repetitively until a certain condition becomes true. Syntax: until condition do statements ... done Operation: while condition remains false (exit status nonzero) the body of the until loop is run [[ ]]operator can be used here as with if to specify the condition Typically, condition is false when an until loop is first encountered; thus the loop will not be executed at least once Just about any until can be converted to a while by simply negating the condition

until Example Classic example: checking user input $ cat prompt
#!/bin/ksh until [[ $ans == no || $ans == yes ]]; do print -n 'Continue? (yes or no): ' read ans done $ prompt Continue? (yes or no): y Continue? (yes or no): sure Continue? (yes or no): ok Continue? (yes or no): yes Continue? (yes or no): n Continue? (yes or no): nope Continue? (yes or no): no way Continue? (yes or no): no $

Exercises 1.Write a script called lis that uses a for loop to display all files and directories in the current directory. 2. Write a script called dirs that displays only the names of the directory files in your current directory. 3. Using the select command, write a script that creates a menu of the abbreviations for all the months of the year. If a certain month abbreviation is chosen, the script should print out the full name of the month and and number of days in it. Change the PS3 prompt to something appropriate. 4. Using a while loop, write a script that displays a guessing game. The player is to guess a number between 0 and 59. If the guess is too low, the message “Higher” is printed. If the guess is too high the message “Lower” is printed. Guesses continues in until the player guesses the correct number. The number the user is trying to guess is taken from the seconds of the date command.

Exercises 5. Modify the script from problem 4 to keep track of the number of guesses it takes for the user to discover the correct number. Have the script print the total number of guesses that it took. 6. Rewrite the first while example script shown in this chapter (printing out the positional parameters) using the until statement instead.

Working with Integer Variables
Arithmetic Operations Operator Precedence Numerical Bases String Variables vs Integer Variables Exercises

Arithmetic Operations
So far we have seen how to give values to integer variables and compare them (in two different ways!) In this section, integers will be used in arithmetic operations Below is a list of the most common operations performed with integers Arithmetic Operator Meaning + addition - subtraction * multiplication / division with truncation % modulus division The shell interprets symbols surrounded by $(( and )) as arithmetic expressions. Variables in arithmetic expressions do not need to be preceded by dollar signs.

Arithmetic Examples Demonstration of the integer divisions:
$ cat intop let x=31 let y=8 print The integer division of $x by $y is $((x/y)) print The modulus division of $x by $y is $((x%y)) $ intop The integer division of 31 by 8 is 3 The modulus division of 31 by 8 is 7 Parentheses can give operations higher precedence: $ cat paren #!/bin/ksh let x=$(( $1 + $2 * $3 - $4 )) print $x let x=$(( ($1 + $2) * ($3 - $4) )) $ paren 23 18

Different Numerical Bases
Possible to represent integers in alternate bases: $ cat base #!/bin/ksh print -n $(( 57 )) print -n , print -n $(( 16#39)) print -n $(( 8#071)) print -n $(( 2# )) print $ base 57,57,57,57 $

String Variables vs Integer Variables
Arithmetic does not work with string variables: $ cat intstr print As integers ... if (( $1 < $2 )) then print $1 is less than $2 fi let x=$2+4; print $2+4 is $x print As strings ... if [[ $1 > $2 ]] then print $1 is greater than $2 x=$2+4; print $2+4 is $x $ intstr 3 15 As integers ... 3 is less than 15 15+4 is 19 As strings ... 3 is greater than 15 15+4 is 15+4

Exercises 1. Write a script which finds the sum of all the positional parameters. 2. Write a script to determine if the first positional parameter is prime. 3. Write a script to perform the factorial of the first positional parameter 4. Write a script that will output positive and negative integers. 5. Write a script that will convert the first positional parameter from base 10 to base 5. 6. Write a script that reads in a time in seconds and outputs as hours:minutes:sec.

Script Input / Output Input / Output Redirection “Here-is” documents
print read Quoting Exercises

Input / Output Redirection
The basic I/O redirectors >, < and | are good enough for most tasks. The Korn shell supports 16 redirectors in total. Here are some other useful ones: Redirector Function >> file Direct stdout to file, appending to file if it already exists >| file force stdout to file even if noclobber is set << label “here-is” document (see below) n> file direct file descriptor n to file n<file set file as file descriptor n >&n duplicate stdout to file descriptor n <&n duplicate stdin from file descriptor n

Input / Output Redirection
Example: This script called start which can be used to start a different script (entered as positional parameters) in the background and save both stdout and stderr in a single log file. $* > logfile 2>&1 & If we want to do the same as above but to the screen as well: $* 2>&1 | tee logfile & The command tee takes its stdin and copies it to stdout and to the file given as argument.

“Here-is” documents command << label
Special form of inline input redirection Syntax: command << label input for command .... label The shell performs command and variable substitution where appropriate unless label is surrounded by quotes

print Escape Sequence Character printed
We have already seen many examples of the print command. Here are some useful escape sequences and options; enclose escape sequences in quotes (either double or single): Escape Sequence Character printed \a bell \b backspace \r return \t tab Option Function -n omit the final newline -r ignore any escape sequence -s print to history file -un print to file descriptor n

print Example $ cat escape #!/bin/ksh print "Call me \t Ishmael"
print -r "Call me \t Ishmael" $ escape Call me Ishmael Call me \t Ishmael $ cat print.u3 print -u3 "My errors:" 3>errors $ ls errors Cannot access errors: No such file or directory $ print.u3 $ cat errors My errors:

read We have seen simple examples of read.
Shell scripts with read are useful for certain tasks such as reading in lines from a file Suppose we have the file /etc/nicks : myrtle beach_bum killian brewster daytona speedster ... which is a list of hostnames followed by nicknames. We want to assign the variable NICKNAME to the appropriate nickname depending on which machine you log into.

read machine=$(uname -n) The following code will do it:
while read host nick; do if [[ $host = $machine ]]; then NICKNAME=$nick print NICKNAME is $NICKNAME break fi done < /etc/nicks <-- read exits with status 1 when there is nothing left to read

Quoting A source of confusion for novice and experienced Korn shell script writers is when to use single quotes ‘ ‘ and when to use double quotes “ “ Throughout the workshop we have learned that certain symbols have special meanings: $name indicates the contents of the variable name, $(command) indicates the output of a command, etc. Single quotes are strong quotes: they prevent the special meaning symbols from performing their tasks. The symbols are treated just like normal characters. Double quotes are weak quotes: they allow the following actions to take place: variable and positional parameter substitution command substitution arithmetic calculations

Quoting Demonstration
$ cat quote #!/bin/ksh money=franc print "Nonsense words ${money} $1 $(date +%T) $((3*75))" print 'Nonsense words ${money} $1 $(date +%T) $((3*75))' $ quote raven Nonsense words franc raven 16:23:30 225 Nonsense words ${money} $1 $(date +%T) $((3*75))

Exercises 1.Write a shell script tfile which creates a test file called junk. Do not use a cp or mv command to accomplish this. File junk contains the following: This is a test abcdefghijklmnopqrstuvwxyz end of test 2. Create three files called data0, data1, and data2 which contain the month, day, and year respectively of your birthday. Write a single script that reads from all three of these files and then outputs your birthday to the screen

Communicating with Running Scripts
Sending Signals Signal Handling

Signal Handling We want to have control over our shell scripts even when they are interrupted by signals. Signals come in the form of control keys, the kill command and user-initiated events like logging out. Programs can be set up to "trap" specific signals and process them in their own way. Syntax of trap command: trap command sig1 sig2 ... when any of sig1, sig2, etc., are received, run command, then resume execution

Signal Handling After command finishes, the script resumes execution just after the command that was interrupted. sig1, sig2, ... can be given by name or number (name is more portable). trap without arguments lists traps that have been set

Signal Handling trap 'print You hit ctrl-c!' INT
Simple example - shell script loop : trap 'print You hit ctrl-c!' INT while true: do sleep 60 done ctrl-c will not kill this program but rather print the message and start another sleep; type ctrl-\ to stop it (then rm core).

Signal Handling trap 'print You hit ctrl-c!' INT
Enhancing the script to handle background execution (trap the kill command): trap 'print You hit ctrl-c!' INT trap 'print No kill I!' TERM while true: do sleep 60 done if run as loop &, kill %loop won't stop it; need kill -KILL %loop Note: kill -l will give the numbers associated with given signals (defined in /usr/include/sys/signal.h).

Signal Handling Examples:
If any of the signals listed (0, 1, 2, 3, or 15) are received by the program, file temp$$ will automatically be removed. Without the exit command, program execution would continue from the point where the signal was received: trap 'rm -f temp$$; exit 0' The command listed for trap is null. This specifies that the signals listed are to be ignored, and the program will try to continue where it left off: trap "" Using the null command (:) specifies that the signals listed are to be ignored for the next command only: trap :

Signal Handling nohup $* > logfile 2>&1 &
What if you want a program running in the background to continue even after you log out? Shell sends the HUP (hangup) signal to all of your background jobs when you log out. Use the command nohup to put a job into the background which will ignore HUP and QUIT (ctrl-\) signals: nohup $* > logfile 2>&1 &

Signal Handling Most common signals: Signal Name Note 1 HUP Hangup
2 INT ctrl-c 3 QUIT ctrl-\ 9 KILL Cannot be trapped 15 TERM default kill signal 24 TSTP ctrl-z

Script Functions Why Use Functions? Syntax & Use Auto-Loading
Function Definition Location Command Precedence Exercises

Functions Improves shell's programmability Syntax:
already in memory (unless autoloaded) modular programing Syntax: function functname { shell commands } or, functname () {

Functions Delete a function definition with $ unset -f functname
Display all defined functions: $ functions

Functions Two important differences between functions and shell scripts run by name (Method 2): functions do not run in a subprocess; behave more like a script run by Method 1 functions have precedence over scripts Where to put them? enter on command line (no longer available if you log out and log back in) .profile (only available in login shell) $ENV file (available to all child shells) in $FPATH directory (Korn shell will use autoload feature) Function names should begin with a letter or underscore The closing brace (}) must appear on a line by itself

Functions Example: Run a program and view the output $ myrun () > {
> cd $WORKDIR > a.out < data.in > more data.out > /bin/rm data.in > mv data.out data.in > }

Exercises 1.Write a script which sets the PAGER environment variable. The script should accept input values from: command-line arguments: -p pager or -ppager shell environment variable PAGER default setting: more -s Include a usage line. (Hint: You should try using the pattern matching operators introduced in this section) 2. Write a script that implements a simple version of the shell's built-in whence command. It should check each of the directories on your PATH for a command given as an argument, and print the full pathname if the command is found. Only files that are executable by you (and are not directories!) should be returned.

Exercises 3. Design functions pushd and popd which implement a "stack" of directories (as in the C shell). The stack is just a list of directories, stored in an environment variable we'll call DIRSTACK. pushd should "push" a specified directory onto the stack, i.e., prepend the directory name to DIRSTACK. popd "pops" a directory off the stack, i.e., cds to the top directory on the stack and then removes that directory from the stack. Your functions should: Allow the use of "." to specify the current directory. Return an error message if pushd is invoked without an argument, or if the specified directory does not exist. popd should return an appropriate message and do nothing if the stack is empty. Why are these implemented as functions rather than scripts? What modifications (if any) should you make to your ~/.profile in order to use them?

Exercises 4. Write a script that will change the suffix of all the files in the present directory from .old to .new

HPC Designed Scripts Repeating Program Execution Debugging Options
Batch Jobs Performance Tool Preprocessing Multiple Tool Runs Multiple Tool Reports File Renaming

Repeating Program Execution
It is often desired to run a program several times. Even a serial programs may have to run repeatedly to debug intermittent problems, find logical errors, etc. In HPC parallel processing arena, there are even more reasons to write a script that will execute code many times: Each run is with a different number of processors so that the traditional speedup graph data can be collected (and compared with Ahmdal’s Law) To perform repeated timing of the entire or partial code. Several parallel machines do not allow dedicated use of processors A severe load imbalance may come and go depending on how concurrently the various processors actually work Over synchronized program’s times can vary due to the order of arrival of the processors at barriers To gather data from program output for statistical analysis. One can also gather timing data for similar reasons. To confirm that a parallel program is preventing race conditions which can arise when one processor is interrupted by another when the former is trying to read, update, and write back to a shared memory location Repeated runs is easily done in our introductory triangle script by using a while loop. Here the script will run the timearea code which times how long the program takes to calculate the area

Timing Script $ cat triangle
cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c cat << EOF > sides.in 8 3 7 EOF let i=0 while (( i < $2 )); do $1 < sides.in >> $1.dat let i=i+1 done cat $1.dat rm $1 sides.in

Timing Script Usage & Output
$ triangle timearea 17

Debugging Options To use debugging software (dbx, Totalview, Xbrowse, etc) programs need to be compiled with certain options. The debugging option is traditionally –g but has been improved lately to include various levels of optimizations while the debugging is being performed (G0,G1,…) For all debugging options, the compiler will make a symbol table which connects a program identifier (variable name, etc.) with a memory address. This allows the debugger to track the values of any variables as the program is executed line by line On next page we show a variation of the introductory triangle script in which a decision is made- through the use of a second positional parameter- if debugging will be implemented

Debugging Script $ cat triangle if [[ $2 = debug ]]; then
cc -g -o $1 $1.c print "$1 ready to be debugged" else cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c $1 < sides.in > $1.dat cat $1.dat rm $1 fi $ triangle area Area of Triangle is $ triangle area debug area ready to be debugged

Batch Jobs There are a number of popular batch processing software that are heavily used on HPC machines: NQE, PBS, and LSF to name three. At this point in the course, we now realize that the “batch files” we are used to writing are actually just shell scripts. The header lines have the syntax and are treated as comments if the batch script is not submitted to the batch software. But since these batch scripts have designed purpose – to run the UNIX commands in them and send a log file back-we can tailor them to reflect it. For example, it is recommended that the shell command echoing feature be turned on so that the commands appear in the log file preceding their output. An example is shown on the next slides.

A Batch Script Here is a simple batch job (file sp_gravity.job):
#QSUB -r grav #QSUB –lT 3000 #QSUB –lM 10Mw #QSUB -oe set -x # echo commands cd gravity/batch ls -lrt f77 -o spg sp_gravity.f ./spg < region.rsp ls –lrt

A Log file (grav.o3613) + ls -lrt total 1512
-rw-r dje G May 18 12:22 sp_gravity.f -rw-r dje G May 18 12:22 newbedrx.cray -rwxr dje G May 18 12:31 region.prm -rw-r--r dje G May 18 12:31 region.rsp -rw-r--r dje G May 18 12:55 sp_gravity.job + f77 -o spg sp_gravity.f sp_gravity.f: MAIN xy2glqr: "sp_gravity.f", line 346: Warning: local variable "zero" never used gauleg: + ./spg + 0< region.rsp Parameter file: Gravity output file: region.out 3-D ORTHOGONAL COLLOCATION ON FINITE ELEMENTS ********** (GAUSS-LEGENDRE QUADRATURE) ... Integration over source region in progress . . .

Log file (cont.) + ls -lrt total 1712
-rw-r dje G May 18 12:22 sp_gravity.f -rw-r dje G May 18 12:22 newbedrx.cray -rwxr dje G May 18 12:31 region.prm -rw-r--r dje G May 18 12:31 region.rsp -rw-r--r dje G May 18 12:55 sp_gravity.job -rwxr-xr-x 1 dje G May 18 12:58 spg -rw-r--r dje G May 18 12:58 region.coeff -rw-r--r dje G May 18 13:00 region.out

qsub Command with options
The realization that a batch job is really just a normal shell script, is clear when the script only consists of the Unix commands. The “header” section is replaced by options to the qsub command: qsub –r grav –lT 3000 –lM 10Mw –eo < grav.ksh Where grav.ksh is the remaining Unix commands: set -x cd gravity/batch ls -lrt f77 -o spg sp_gravity.f ./spg < region.rsp ls –lrt This is the alternative ( not recommended) method for running a batch job. It will return the same log file shown on the previous slides.

Advanced Batch Scripting
Since a batch script is – just that – a script, we can using any part of the scripting language in it. Consider our “classic” compile and run script we have been using in the previous slides. If the compilation is a failure, there is no point in proceeding with the script’s commands since there will be no executable to work with. In the advanced batch script shown on the next slide we use the if/then shell command to check to see if there were no errors produced in the compilation. If there was, we exit the script (and the job,BTW). The log file returned after submitting this script is also shown. An error was deliberately inserted in the source code.

Error-Checking Batch Script
set -x cd SV1/dje/gravity f90 -o spg sp_gravity.f if [[ $? -ne 0 ]]; then exit 33 fi ./spg < region.rsp

Error-Checking Log File
+ cd SV1/dje/gravity + f90 -o spg sp_gravity.f do i=1, ^ f f90: ERROR XY2GLQR, File = sp_gravity.f, Line = 408, Column = 14 Unexpected syntax: "operand" was expected but found "EOS". Cray Fortran: Version (u15f51p42138g41109a18) Cray Fortran: Mon Jun 10, :13:34 Cray Fortran: Compile time: seconds Cray Fortran: 1130 source lines Cray Fortran: 1 errors, 0 warnings, 0 other messages, 0 ansi Cray Fortran: "explain f90-message number" gives more information about each message + [[ 1 -ne 0 ]] + exit 33

Performance Tool Preprocessing
Almost all of the sophisticated software tools that exist on HPC machines to profile your code and aid in serial, vector, and parallel optimization require the user to run the code and generate a raw data file. The tool subsequently analyzes the raw data under the user control (usually interactively) For some HPC software tools the time to gather the raw data can be long, especially if the code is complex with many functions and subroutines. Or the syntax required might be different than ever other optimization tool on that machine. Because of these reasons, it is often useful to perform the generation of raw data as a shell script. The script shown on the next slide, generates the raw data file for the prof tool on the Cray SV1.

Using the prof preprocessing script
oscb:$ cat profraw.ksh #! /bin/ksh f90 –G1 –o $1 –lprof $1.f ./$1 < $1.in # Creates the prof.data file prof –x –m prof.data $1 > $1.prof.raw # User can pick the raw output filename rm $1 oscb:$ profraw.ksh poisson oscb:$ profview poisson.prof.raw

Multiple Tool Runs For HPC tools that access Hardware Counters on the actual vectorizing processors or commodity Microprocessors (Pentiums, Itantiums, Ultra-Sparcs, MIPS R12K, etc.) there are too many counter chip to gather all the information with just one run of an optimization tool. Therefore, to access all the hardware use information, your code must be run several times. For Cray machines, 4 runs of hpm and perf are required. For Microprocessor-based MPP machines there a typically 64 counters only two of which can be read out in a single run of the appropriate tool (ssrun for the Origin 2K, collect for the Sun Fire 6800, etc.) Compiling and running an executable, with different options chosen to access various counters is an ideal task for shell programming. In the next slide, a script is shown that gathers all the hardware use info for Cray machines with the hpm and perf tools installed

total_hw.ksh #! /bin/ksh # First parameter is the source code “prefix”
cc –O4 -TARG:madd=on -o $1 $1.c hpm –g0 ./$1 > $1.hpm hpm –g1 ./$1 >> $1.hpm hpm –g2 ./$1 >> $1.hpm hpm –g3 ./$1 >> $1.hpm # Recompile with perf options cc –F –lperf –o $1 –O4 –TARG:madd=on $1.c typeset –x PERF_GROUP=0 PERF_DATA=p0.raw ./$1 typeset –x PERF_GROUP=1 PERF_DATA=p1.raw ./$1 typeset –x PERF_GROUP=2 PERF_DATA=p2.raw ./$1 typeset –x PERF_GROUP=3 PERF_DATA=p3.raw ./$1 cat p0.raw p1.raw p2.raw p3.raw >perf.data rm $1

Saving Multiple Tool Reports
A powerful feature of HPC Optimization software tools is that they provide information on a variety of performance topics with just one run of the tool. A certain tool might have a report of subroutine time-profiling, a caller-callee tree, a summary report, etc. When interactively using such a tool, a user can pick which report to read, and then pick another and so on. It is often desirable to just save a select set of the possible reports. This can be done by running the tool several times but with a different option each time. The option determines what report will be saved in that run. Again, a natural job for a shell script. In the next slide, a script is shown that saves several reports from the flowview tool used on Cray PVP machines. (12 reports are available from this tool)

flowview report selection script
oscb:$ cat flowchoice.ksh #! /bin/ksh # Script assumes raw data file flow.data exists # First parameter is prefix of the source code pre=$1 shift for opt in $x do flowview –L$opt >> pre.flow done oscb$ flowchoice.ksh neuralnet m h u k

Renaming Files Specifically, renaming the prefixes or suffixes of a large number of files. Many HPC parallel programs will produce hundreds of files which contain data at certain time steps, or spatial extent. Data visualization files are one example of this type of output. Say the user needs to put a new suffix or prefix on a large set of files in order that they not be overwritten by a subsequent data set. After this course, one’s mind should naturally think “ I bet a script could do this for me”. You would win that bet. On the next slide is shown a script that will change the suffix of all files whose name has the syntax strain001.txt, strain002.txt to strain500.txt to the same prefix but with “txt” replaced by “old”. The script can easily be made more general purpose as well.

Renaming script oscb$ cat suffix.ksh #set –x for file in $* do
mv $file ${file%.*}.old done oscb$ suffix.ksh strain*.txt

Korn Shell Script Writing & HPC

Similar presentations

Presentation on theme: "Korn Shell Script Writing & HPC"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Korn Shell Script Writing & HPC

Similar presentations

Presentation on theme: "Korn Shell Script Writing & HPC"— Presentation transcript:

Similar presentations

About project

Feedback