Intermediate Unix Presented July 29 th, 2001 by: “Robin” R. Battey Evgeny Roubinchtein with tips, suggestions, and corrections by: Hannah Tang

The “File System” Under UNIX, (almost) everything is a “file”: –Normal files –Directories –Hardware –Sockets –Pipes Things that are not files: –Users –Groups –Processes

Ownership Files have two owners –Every file has exactly one user owner –Every file has exactly one group owner Everyone is a user –Users are in at least one group Processes have owners, too (known as an “id”) –Every process has exactly one user id –Every process has at least on group id Users and groups are really just numbers with names –Every username is mapped to a single numeric “uid” –Every groupname is mapped to a single numeric “gid”

Who am I? Commands that tell you who you are: –whoami displays your username –id displays your username and groups Commands that tell you who others are: –finger [ ] displays info for –id [ ] displays info for Commands that change who you are: –su “switch user” to –login login as a different user

File Permissions Every file has three access levels: –user(the user owner of the file) –group(the group owner of the file) –other(everyone else) At each level, there are three access types: –read(looking at the contents) –write(altering the contents) –execute(executing the contents)

Strange Things There are three “strange” permissions: –setuid(run program as user owner) –setgid(run program as group owner) –text(stay in swap after executing) Directories act differently –“write” (creating/deleting files) –“execute”( cd -ing to that directory) –“setuid”(ignored) –“setgid”(created files have same group owner) –“text” (deletion restricted to user owned files)

Examining Permissions A “long” ls listing shows file permissions: zanfur]$ id uid=8774(zanfur) gid=100(users) groups=100(users),101(mp3) zanfur]$ ls -l total 524 -rwxr-xr-x 1 zanfur users Jul 31 00:18 bash prw-r--r-- 1 zanfur users 0 Jul 31 00:24 fifo -rw-r--r-- 1 zanfur users 0 Jul 31 00:18 file drwxr-xr-x 2 zanfur users 4096 Jul 31 00:16 normal drwxrws--T 2 zanfur mp Jul 31 00:14 shared drwxrwxrwt 2 zanfur users 4096 Jul 31 00:14 tmp drwxrwxr-x 2 zanfur www 4096 Jul 31 00:15 www zanfur]$ _

Permissions Listing Breakdown FieldDescriptionValid Values ? file type - (normal file), d (directory), p (fifo), b, c, s -? user read - (no read permissions), r (read permissions) --? user write - (no write permissions), w (write permissions) ---? user execute - (no exec), x (exec), S (setuid), s (both) ----?----- group read - (no read permissions), r (read permissions) -----?---- group write - (no write permissions), w (write permissions) ?--- group execute - (no exec), x (exec), S (setgid), s (both) ?-- other read - (no read permissions), r (read permissions) ?- other write - (no write permissions), w (write permissions) ? other execute - (no exec), x (exec), T (text), t (both)

What You Can Do With Permissions PermissionFileDirectory r (read) Read a fileList files in … w (write) Write a fileCreate a file in … Rename a file in … Delete a file in … x (execute) Execute a file (eg shell script) Read a file in … Write to a file in … Execute a file/shell script in …

Changing Ownership Changing user ownership: –The command is “change owner”: chown –but, you can only do it if you are root –so just copy it instead Changing group ownership: –The command is “change group”: chgrp –but, you can only do it if you are in group –and you must be the user owner of the file

Changing Permissions The “change mode” command: chmod [,…] string of: u, g, o, a (user, group, other, all) one of +, -, = (gets, loses, equals) string of: r, w, x, s, t, u, g, o (read, write, execute, set-id, text, same as user, same as group, same as other), Examples: chmod u+rwx,go-w foobar chmod g=u,+t temp/ chmod u=rwx,g=rwxs,o= shared/

Process Management What can you do with it? –Start programs in the background –Run more than one program per terminal –Kill bad and/or crashing programs –Suspend programs mid-execution –List all jobs running in a shell –Move foreground jobs to the background –More …

Three States of a Process Foreground –Attached to keyboard –Outputs to the screen –Shell waits until the process ends Background, running –Not attached to keyboard –Might output to the screen –Shell immediately gives you another prompt Background, suspended –Paused mid-execution –Can be resumed in background or foreground

Background Processes Listing jobs: –jobs lists background “jobs” and job #’s –ps lists processes and their process id (“pid”) –% expands to the process id of the job Stopping foreground jobs –Press ^Z (Ctrl-Z) in the terminal window Starting a process in the background –Append a & character to the command line –Examples: ls –lR > ls-lR.out & xemacs my_program.cc & Resuming a stopped job –In the foreground: fg [ ] –In the background: bg [ ]

Killing Processes The “kill” command: kill [- ] Send to process The “killall” command: killall [- ] Send to all processes that start with Useful signals ( kill –l for the complete list): TERMthe default, “terminate”, kills things nicely KILLwill kill anything, but not nicely HUP“hangup”, used to reload configurations STOPstops (suspends) a running process

What are environment variables? The environment is an array of strings of the form NAME=VALUE Each process (program) has its own copy of the environment Processes started by the shell get a (“deep”) copy of the shell’s current environment Each process can examine and modify its own (and only its own) environment The “meaning” of environment variables is merely a matter of established conventions

Viewing your shell’s environment To view a single environment variable’s value: echo $ For example: echo $HOME will display /homes/iws/evgenyr To view all environment variables at once: –tcsh/csh/bash: printenv –sh/ksh: set

Setting environment variables tcsh/csh: –setenv –Example: setenv PRINTER ps329 bash/ksh: –export = –There is no space on either side of ‘=’! –Example: export PRINTER=ps329 sh: – = ; export !

Appending to environment variables Appending /uns/bin/ to your PATH –bash/ksh: export PATH=$PATH:/uns/bin –tcsh/csh: setenv PATH=$PATH:/uns/bin Prepending is similar: –bash/ksh: export INFOPATH=/uns/info:$INFOPATH –tcsh/csh: setenv INFOPATH /uns/info:$INFOPATH

Common environment variables The “meaning” of environment variables is purely a matter of convention. However, these environment variables are quite common: –PATH, INFOPATH, MANPATH, EDITOR, VISUAL, PAGER, HOME, MAIL, USER The man page for a program will usually list the environment variables the program pays attention to.

Useful shell features (a preview of upcoming attractions) Aliases Redirecting input and output Command substitution Scripts

Shell as a work-saver: aliases Aliases: textual substitution, similar to C-preprocessor macros. Syntax: –bash/ksh: alias =’ ’ –tcsh/csh alias ’ ’ When you type, the shell “substitutes” Typing alias by itself lists all your current aliases unalias removes the alias Check the csh/tcsh man page for extra features

Alias examples Always print postscript files double-sided –tcsh/csh: alias lpr ’lpr –Zduplex’ –bash/ksh: alias lpr=’lpr –Zduplex’ Collect a few tree-friendly options to enscript: –tcsh/csh: alias print \ ’enscript –2rhB –SDuplex=DuplexNoTumble’ Line continuation character, just like in C/C++!

What is input/output redirection? Normally, a program’s standard output is displayed on user’s terminal, and its standard input comes from the keyboard. Redirecting the output of a program means asking the shell to put the program’s output ( stdout [C++’s cout ]) into a file. Redirecting the input of a program means asking the shell to feed a file as the program’s standard input ( stdin [C++’s cin ]). Note: redirection works with files.

Why redirect program’s output? You may want to save output and examine it at your leisure: –if the program generates a lot of output –if the program takes a long time to run You may want to present the output to another person Having the program write to standard output may make the program simpler to write

Standard output vs Standard error By convention, “normal” output of a program is sent to standard output ( stdout [C++’s cout ]), while debugging or error output is sent to standard error ( stderr [C++’s cerr ]).

How to redirect program’s output? To redirect just the standard output: > To redirect just the standard error: sh/ksh/bash: 2> csh/tcsh: ( > STDOUT ) >& STDERR To redirect both standard output and standard error: csh/tcsh/bash: >& sh/ksh/bash: > 2>&1

> vs. >> Both > and >> will create the output file, if it doesn’t already exist If the file does exist, then: –Using > to redirect output will overwrite the output file: ls > newlisting printenv > my_environment –Using >> to redirect output will append to the output file cat ch1 ch2 ch3 > book cat ch4 ch5 ch6 >> book

Why redirect program’s input? To run the program repeatedly with the same (or similar input) Having the program read from standard input may make the program simpler to write.

How to redirect program’s input? Simple! Example sort < my_grades.txt head < really_long_book.txt

Piping Piping is connecting programs together by using the output of one program as the input to the next. Syntax: | | … | A simple example (view a sorted file-listing a page at a time): ls | sort | less Note: piping deals with the input/output of programs (that is, stdin, stdout, stderr )

Why piping? Because “the whole is bigger than the sum of its parts. By combining Unix utilities in a pipeline, you can build tools “on-the-fly” as you need them.

Piping examples How many.c files are in this directory? ls *.c | wc –l What files were modified most recently? ls –t | head What processes am I running? ps auxw | grep Make an alias for the above, for other Linux boxen too: alias myps=’ ps auxw | grep `id –un`’

The cat utility Especially useful: cat –When used with pipes, “copies” stdin to stdout cat can be used to redirect out of a pipe! Example: make a file containing currently running processes ps aux | grep evgenyr | cat > my_processes –When used with a file, “copies” the file into stdout cat can be used to place a file’s contents into a pipe! Example: list all the items in a list in alphabetical order cat my_grocery_list.txt | sort | uniq

Command substitution (aka “what’s up with the backquotes?”) Command substitution means that the shell substitutes a command’s output for the command itself. To get the shell to perform command substitution, enclose the command in backquotes (`)

Shell as a work-saver: scripts Instead of typing the same series of commands over and over again, put them in a file and have the shell execute the (commands in the) file! The file must have execute permission The first line of the file should be: #! (e.g., /bin/bash) There must be no space (or any other character) between ‘ # ’ and ‘ ! ’. Whether to put space after ‘ ! ’ is a matter of style Shell also has control flow, tests, etc. The shell tutorial on the ACM web page goes into much more detail.

Intermezzo: quoting Problem: some characters are special to the shell ( *, ? ), but you would like to pass those characters to the programs the shell runs as-is. So you quote the character(s): –A single character: by prefixing it with a \ (backslash) –A string of characters: by enclosing them in Single quotes: no characters are special. None Double quotes: some characters (notably $ and ` ) are still special. You’ll need to prefix those with a backslash to pass to the program as-is.

Quoting examples List all the files in the current directory: ls * List the file or directory called ‘ * ’ (or complain if it’s not there): ls \* Print $HOME: echo ’$HOME’ Print the path to your home directory: echo ”$HOME” Print `id –un`: echo ’`id –un`’ Print your login: echo ”`id –un`”

Next stop: utilities (No, not electricity, water, and sewer) diff and patch grep find and xargs

diff and patch You have two versions of a file; how do you see what’s changed between the two versions? –diff shows the differences between two files; you’ll want to use the –u or –c option to diff to produce output in human-friendly format: diff –u my_file my_file.orig | less –patch applies differences to a file diff –u my_file my_file.orig > my_patch patch my_file < mypatch my_file is now the same as my_file.orig

grep – search for patterns grep searches for patterns in texts: grep grep uses regular expressions to describe the string(s) to search for In a regular expression, most characters are treated as-is, but a few are magic –Ordinary characters just represent themselves –Magic characters do special things

grep ’s magic characters A few different kinds of magic characters: –Some characters “anchor” (parts of) a regular expressions to specific places in the string: ^ - The beginning of a line $ - The end of a line –A dot (. ) matches “any one character whasoever” (except for newline) –[ ] form a character class: [aeiou] – any single vowel [a-zA-z0-9] – Any single letter or digit [^0-9] – Any single character that isn’t a digit Here, ‘ ^ ’ means “not”

Even more of grep ’s magic characters Quantifiers say how many times the preceeding “thing” should be repeated: –* means “zero or more times” –? Means “zero or one time”

Frequently used grep options -i : do case-insensitive search -n : print line numbers -v : print lines that do not match -l : only list the files from which output would have been printed

grep examples Print all non-blank lines: grep –v ’^$’ my_file.txt Print all the lines on which my_function is called (or declared): grep –n ’my_function *(’ my_code.c Show all the xterms I am running: ps auxw | grep ”`id –un`.*xterm”

find Traverse the tree(s) rooted at, and for each “thing” found, evaluate the until the result of is known. The evaluation of “short-circuits”, just like expressions in C/C++. –false && some_expression can never be true Options apply to the entire find command, not the individual expression find

Components of a find expression An expression is zero or more primaries connected by operators Two kinds of primaries: –Tests: just return true or false –Actions: do something, in addition to returning true or false –Operators work just as they do in C/C++. ! / -not -a / -and -o / -or, (comma) –Parentheses are used for grouping, just as in C/C++.

find examples Print all the *.c* and *.h* files in and below the current directory find. –name ’*.[ch]*’ –a –print Show line numbers myfunction calls in the above *.c* files find. –name ’*.c*’ | xargs grep –n ’myfunction.*(’ Change permissions on files under your home directory so they’re inaccessible to everyone but you (except for files in the www directory) cd; find. –path ”./www*” –prune –o –exec chmod go-rwx {} \; How big are my *.c* files? expr `find –name ’*.c*’ –printf ”%k + ”` 0 Read the find manual ( info find ) for more examples

xargs : combine arguments Feeds standard input as arguments to Often used with find ( find … | xargs grep ) But it doesn’t have to be used with find : cat filelist | xargs cat {} > concatenated xargs

Finding more information – at CSE Use info and man Peruse the “see also” section of the man pages Peruse info ’s i (search index) command The uw-cs.lab-help group Look at your.login,.cshrc, etc. to see how support has set things up

Finding more info – on the web (comp.unix questions FAQ) The support web page: about/ugradcomputing.html (Shameless plug) the ACM tutorials page: