1. Tool Building 1: Shell CS 360 tool1

2. Page 2 tools1 CS 360, WSU Vancouver What is Systems Programming? (again) hardware drivers OS services system programs applications interfaces users implementation layers implementing building Testing Installing deploying System programs typically create environments for applications

3. Page 3 tools1 CS 360, WSU Vancouver Some Tools That System Programmers Create For a software engineering team: 4 organize versions of product sources 4 package designs & sources for peer review 4 build the product 4 test product 4.... For a web site operator: 4 update site per new look and prices 4 ensure file security in place 4... For a LAN administrator 4 add a new hire to all address books 4 install new software in all workstations 4... Such tools manipulate files and directories

4. Page 4 tools1 CS 360, WSU Vancouver What Does the Shell Provide?  Interactive interface to all system capabilities 4 navigate & manipulate file system 4 invoke system commands & programs 4 customize your environment  Programming environment for system programmers 4 automate above interactions 4 glue together smaller programs 4 prototype solutions

5. Page 5 tools1 CS 360, WSU Vancouver Software Structure of Unix, Again Programs: executable files 4 numerous & specific Shell: command line environment 4 few & powerful users shell programs unix libraries unix kernel

6. Page 6 tools1 CS 360, WSU Vancouver Shell Examples Find words in a file 4 C program: 50+ lines of code 4 Shell: 4 lines Find most popular word 4 C program: 75+ lines of code 4 Shell: 2 lines Outline a directory structure 4 C program alone: 100+ lines of code 4 Shell plus C program: 30 lines Clone a directory structure 4 C program: 150+ lines of code 4 Shell: 1 line Shell can reduce code bulk dramatically

7. Page 7 tools1 CS 360, WSU Vancouver Use C and Shell Together C programming = data structures + algorithms Shell programming = file system + utilities

8. Page 8 tools1 CS 360, WSU Vancouver The Shell is a Scripting Language A professional uses both language types

9. Page 9 tools1 CS 360, WSU Vancouver Which Shell? Unix architecture allows arbitrary shells The most popular have evolved and become quite powerful Always avoid religious wars and become productive! Bourne shell "SH" C shell "CSH" Korn shell "KSH" GNU Born Again shell "BASH" editors "EMACS" & "VI" our choice

10. Page 10 tools1 CS 360, WSU Vancouver Basic Shell Concepts Command Lines Pipes Useful Utilities

11. Page 11 tools1 CS 360, WSU Vancouver Canonical Shell Command Format command [ - switch ] … [ argument ]... file name:../bin/doit built-in verb:cd.. simple tokens:grep -e abc../src/foo.c../src/bar.c wildcards:grep -e abc../src/*.c quoted text:grep -e "abc xyz”../src/*.c shell variablesgrep -e “$text”../src/*.c quoted text:grep -e ‘$text’../src/*.c your program doesn't see the quotes or wildcards or variables single letters:ls -a -1 combined single letters:ls -a1 values:cc -o foobar foobar.c strange and wonderful:find. -print

12. Page 12 tools1 CS 360, WSU Vancouver Wildcards Expand into a List of Pathnames ls../src/*.c  ls../src/abc.c../src/xyz.c These are the most common pattern characters: 4 * matches any string of characters, including none 4 ? matches any single character 4 [a-z] matches any character in range 4 ~ your home directory 4 ~bob Bob's home directory Notes: 4 invoked programs don't see the patterns  but, if no matches, pattern is passed to program 4 * doesn't match names beginning with period when first in a pattern 4 tilde interpreted only if first character in token

13. Page 13 tools1 CS 360, WSU Vancouver Echo Command Writes To Stdout % echo hello there hello there % echo % echo "this is one argument" this is one argument % echo "*" * % echo * abc def xyz % echo $PWD /home/langd % echo "$PWD" /home/langd % echo '$PWD' $PWD % x="some value" % echo "$x" $x '$x' some value some value $x $"'$"' note well the rules for:

14. Page 14 tools1 CS 360, WSU Vancouver Many Utilities are Filters (Text In/Text Out) stdin stdout program Shell always provides your program with three open file descriptors: 4 0:"standard input" 4 1:"standard output" 4 2:"standard error" (= stdout usually) Examples: 4 translate lower to upper case:tr "a-z" "A-Z" 4 find lines with "abc":grep -e "abc" 4 delete lines with "abc":sed -e "/abc/d" 4 extract lines starting with "xyz":sed -n -e "/^xyz/p" 4 sort into ascending order:sort 4 copy input to output:cat 4 write to output:echo "hello, world!"

15. Page 15 tools1 CS 360, WSU Vancouver Use Redirection to Control Stdout and Stdin Stdout: 4 Your terminal unless redirected: Stdin: 4 Your terminal unless redirected: Stderr: 4 Your terminal unless redirected: % grep -e "abc" > my-output-file % grep -e "abc" >> my-output-file % grep -e "abc" < my-input-file % grep -e "abc" 2> my-error-file % grep -e "abc" 2> /dev/null % grep -e "abc" 2>&1 % echo "error!" >&2

16. Page 16 tools1 CS 360, WSU Vancouver Connect Filters into Pipelines % tr " " "\n" % tr " " "\n" | sort % tr " " "\n" | sort | uniq words sorted words unique words tr " " "\n" stdin sortuniq stdout Notes: 4 thus stdout of one program is the stdin for the following program 4 programs in pipe execute concurrently (no temp files used) 4 can combine with redirection % tr " " "\n" | sort | uniq result |

17. Page 17 tools1 CS 360, WSU Vancouver Use Grep & Sed to Find or Change Lines Grep finds lines: 4 grep [ -e pattern ] … display lines in stdin that contain any of the patterns Sed rewrites lines: 4 sed [ -e action ] …copy stdin to stdout, doing all actions on each line Some Sed actions: 4 /pattern/ddelete line if contains pattern 4 /pattern/r fileread (copy to stdout) file after the line the pattern matches 4 s/pattern/replace/gsubstitute replace for all occurrences of pattern Grep & Sed patterns: 4.match any character 4 [a-z]match any character from "a" to "z" 4 [^a-z]match any character except from "a" to "z" 4 ^match beginning of line 4 $match end of line (how match a dollar sign?)  pattern * match zero or more occurrences of pattern (match is greedy -- try some experiments)

18. Page 18 tools1 CS 360, WSU Vancouver Some Sed Examples Replace each occurrence of MYNAME with “Dick": Delete all lines exactly 1 character long: % sed -e "s/MYNAME/Dick/g" % sed -e '/^.$/d' why the single quotes?

19. Page 19 tools1 CS 360, WSU Vancouver Use Scripts to Package Commands for Reuse Count number of files in directory "foo" that have "doug" in name: A script named "hunt" to do the same thing: A new version that accepts two parameters: Consider these invocations of the script: 4 new-hunt foo doug betty 4 new-hunt foo 4 new-hunt % ls -1 foo | grep -e "doug" | wc -l 3 ls -1 foo | grep -e "doug" | wc -l ls -1 $1 | grep -e $2 | wc -l % new-hunt foo "doug" 3 (notice the quietness!) % hunt 3 new-hunt hunt

20. Page 20 tools1 CS 360, WSU Vancouver Scripts & Paths Shell finds programs by looking for executable files in the path: File is executable if has "x" mode: A script is a text file that is executable: The script is just a sequence of shell commands in sequence 4 can have parameters 4 supports if, while, and other programming constructs % echo $PATH.:/home/langd/bin:/net/class/cs360/bin:/usr/bin % ls -l my-program -rwxrwxr-x~~~~~~~~~~my-program % vi my-script % chmod +x my-script (" whence foo " = where foo will be found)

21. Page 21 tools1 CS 360, WSU Vancouver Use the Shell Utility Toolkit Basic line oriented processing: 4 grepfind patterns 4 sededit lines 4 cutextract columns 4 sortsort lines into specified order 4 uniqdisplay unique lines in a sorted file 4 tail, headdisplay last lines of a file, or first lines 4 wccount words, lines, and characters File system manipulation: 4 cp, mv, rm, lscopy, move, delete, list files 4 chmod, chownchange file access modes, file owner 4 findnavigate directory structure testing files for attributes 4 touchchange last-modified time on a file Miscellaneous: 4 echoecho arguments on stdout 4 catcopy a file or files to stdout 4 trtranslate characters 4 dateecho current date and time 4 mansee manual page on a command Scripting Languages 4 awkprocess files based on rich pattern matching & text manipulation 4 perldo the same with more C-like facilities documentation = man pages

22. Page 22 tools1 CS 360, WSU Vancouver Good Things to Know Editing last command: 4 Type ESC 4 Then can edit recent commands using VI keystrokes (e.g.: k = go up, i = insert) Using special keys: 4 End terminal stdin with control-D 4 Interrupt execution of a program with control-C Continuing beyond one line: 4 End with backslash to continue line 4 Unbalanced quotes etc. also continue 4 Prompt will change to indicate % hunt foo "doug >

23. Page 23 tools1 CS 360, WSU Vancouver Whiteboard Exercises Examples of scripts and pipelines I love Unix

24. Page 24 tools1 CS 360, WSU Vancouver Some Word Scripts This is a example docment, with some words. This is on line two (which follows line 1). This is on line three (which follows line 2.0). John siad, "Hello, world!". This is a example docment, with some words. This is on line two (which follows line 1). This is on line three (which follows line 2.0). John siad, "Hello, world!". test file "document1": Note: following scripts are stored in the directory " /class/cs360/lab/tools1/words" extract words find misspelled words find most popular word

25. Page 25 tools1 CS 360, WSU Vancouver Words0: tr " " "\n" % words0 < document1 This is a example docment, with some words. This is on line two (which follows line 1). This is... more follows... script: operation: goal: display words preparation: % vi words1 % chmod +x words0 words0

26. Page 26 tools1 CS 360, WSU Vancouver Words1: tr " " "\n" | sort | uniq % words1 < document1 "Hello, (which 1). 2.0). John This a docment, example follows is line on siad, some three two with words. world!". script: operation: goal: display unique words preparation: % vi words1 % chmod +x words1 words1

27. Page 27 tools1 CS 360, WSU Vancouver Words2: tr " " "\n" | \ tr "A-Z" "a-z" | \ sort | \ uniq % words2 < document1 "hello, (which 1). 2.0). a docment, example follows is john line on siad, some this three two with words. world!". script: operation: goal: display unique words treat upper case as lower case preparation: % vi words2 % chmod +x words2 words2

28. Page 28 tools1 CS 360, WSU Vancouver Words3: tr "\$,;./!()'\"" " " | \ tr -s " " "\n" | \ tr "A-Z" "a-z" | \ sort | \ uniq % words3 < document1 0 1 2 a docment example follows hello is john line on siad some this three two which with words world script: operation: goal: display unique words treat upper case as lower case treat punctuation as space collapse spans of spaces preparation: % vi words3 % chmod +x words3 words3

29. Page 29 tools1 CS 360, WSU Vancouver Spell: words3 > /tmp/$$ comm -23 /tmp/$$ /usr/share/dict/words rm /tmp/$$ % spell < document1 0 1 2 docment follows john siad words script: operation: goal: display words that are not in dictionary preparation: % vi spell % chmod +x spell spell

30. Page 30 tools1 CS 360, WSU Vancouver Popular tr "\$,;./!()'\"" " " | \ tr -s " " "\n" | \ tr "A-Z" "a-z" | \ sort | \ uniq -c | \ sort -nr | \ head -1 script: goal: find words using words3 logic sort words by number of occurrences print most popular word % popular < document1 4 line operation: popular

31. Page 31 tools1 CS 360, WSU Vancouver Clone a Directory Structure cp -mr $1 $2 script: goal: copy file or directory recursively preserve ownership, modes, and times % clone /home/langd /backup operation: preparation: % vi clone % chmod +x clone clone

32. Page 32 tools1 CS 360, WSU Vancouver Outline a Directory Structure % find foo -type d foo foo/bin foo/lab foo/lab/intro foo/lab/intro/my-directory foo/lab/dev1 foo/lab/io foo/demo foo/demo/a foo/demo/b foo/hello foo/example foo/example/one foo/example/two foo/example/two/three "find" walks directories: (lots of wonderful switches!) % outline foo foo /bin /lab /intro /my-directory /dev1 /io /demo /a /b /hello /example /one /two /three What we want: We will create this using a small program that replaces text to left of slashes with blanks

33. Page 33 tools1 CS 360, WSU Vancouver Outline a Directory Structure find $1 -type d | unslash script: % outline foo foo /bin /lab /intro /my-directory /dev1 /io /demo /a /b /hello /example /one /two /three goal: outline a directory structure preparation: % vi outline % chmod +x outline % vi unslash.c % cc -o unslash unslash.c operation: outline

34. Page 34 tools1 CS 360, WSU Vancouver Implementation of Unslash Outline the logic: 4 read stdin lines –if not the first line: overwrite with blanks all text to left of rightmost slash –write the line 4 return Refine the logic: 4 key variables: –let "line" be an array that holds the line just read –let i index each character in line –let s index the rightmost slash or -1 if no slash 4 to do the overwriting: –set s by examining each character (left-to-right) –replace each line[0..s-1] with a blank Translate logic into C 4 next slide...

35. Page 35 tools1 CS 360, WSU Vancouver Code for Unslash is Simple #include "lineio.h" #define LINEMAX 1000 int main (int argc, char *argv[]) { char line[LINEMAX];/* input line & work area */ int n;/* number of line 0, 1,... */ int i;/* examine line[i] next */ int s;/* line[s] is a slash or s == -1 */ n = 0; while (getline (line)) { /* overwrite text, if not first line */ if (n > 0) { /* set s to index of rightmost slash or to -1 */ i = 0; s = -1; while (line[i] != '\0') { if (line[i] == '/') s = i; ++i; } /* replace line[0..s-1] with blanks */ i = 0; while (i < s) { line[i] = ' '; ++i; } /* write line and increment n */ putline (line); ++n; } note how we find rightmost slash unslash.c

36. Page 36 tools1 CS 360, WSU Vancouver Discussion... In this situation: 4 by using a Shell utility, we had an "almost" solution immediately 4 by reformulating the problem as a filter, we could write a simple C program to complete the solution This is a general approach for tool building with the shell: 1.Use files and directories 2.Use utilities whenever possible 3.Fill gaps with small C programs 4.Connect components with pipes 5.Format data as text