3 File Processing Mauro Jaskelioff

Introduction More UNIX commands for handling files Regular Expressions and Searching files Redirection and pipes Bash facilities

In UNIX everything is a file! Directories and programs are all files! Devices (keyboard, mouse, screen, memory, hard disks, etc) are files Input and Output channels are read and written like files. All of these things can be manipulated like files

More UNIX commands

Review of commands seen so far who, date, finger, passwd man pwd, cd, mkdir, rmdir cp, mv, rm, ls cat, more, less, head, tail lpr chmod, umask

Linking Files For every file we have –Its entry in its directory, with information such as: filename, actual location on disk. –Its contents on the disk with a reference count / binusrhome zlizmj foo.txt 1

Linking Files (2) ln adds a link to the contents and increments the reference count ln /home/zlizmj/foo.txt /usr/bin/bar.sh This is called a hardlink It does not copy the contents of the file rm decreases the reference count. Erases contents from file if the reference count reaches 0 / bin bar.sh usrhome zlizmj foo.txt 2

Linking Files (3) ln -s adds a symbolic link to t ln –s /home/zlizmj/foo.txt /usr/bin/bar.sh Symbolic links are visible with ls –l rm just erases the symbolic link 1

Getting Information about Files file gives the content types of the specified files (e.g., text, binary, directory, program) file /bin/* wc counts the number of words, lines and characters in a file -l for lines -c for characters -w for words with no argument, it reads from the keyboard

Finding Files find searches in a directory hierachy find -name -print $ find /usr/share/doc/ -name 'post*' -print /usr/share/doc/postgresql /usr/share/doc/postgresql /html/postmaster-shutdown.html /usr/share/doc/postgresql /html/postmaster-start.html $ which will find things which are on your PATH which shows which would be executed if we typed command. $ which tail /usr/bin/tail

sort –prints out lines of a file sorted into alphabetical order can sort on fields within lines can sort numerical entries (-n) flags to remove duplicates, reverse sort, etc. cmp –tests whether two files are identical and reports position of first character where they are (shows 0 if they are identical) Sorting and Comparing Files

Sorting and Comparing Files (2) comm –gives three column output of lines in first, but not second; in second, but not first; and in both diff -c –gives the differences with 4 or 5 lines either side to show context

Regular Expressions and Searching Inside Files

grep pattern searches the specified files for the specified pattern and prints out all lines that contain it, e.g.: grep “that” poem will print every line in poem containing the word that Searching Inside Files

Regular Expressions grep “That” poem will only find the string “That” in poem if it has an upper case ‘T’ followed by lower case ‘hat’ Regular expressions are much more powerful notation for matching many different text fragments with a single expression –i.e. could wish to find “That”, “that”, “tHaT”, etc.

Regular Expressions (2) Search expressions can be very complex and several characters have special meanings –to insist that That matches only at the start of the line use grep “^That” poem –to insist that it matches only at the end use grep “That$” poem –a dot matches any single character so that grep “c.t” poem matches cat, cbt, cct, etc.

Square brackets allow alternatives: –grep “[Tt]hat$” poem An asterisk allows zero or more repetitions of the preceding match –grep “^-*$” poem for lines with only -’s or empty –grep “^--*$” poem for lines with only -’s and at least one - –grep “Bengal.*Sumatra” poem for lines with Bengal followed sometime later by Sumatra Many flags to: –display only number of matching lines, ignore case, precede each line by its number on the file and so forth Regular Expressions (3)

Redirection and Pipes Connecting all the tools!

Input and Output in UNIX UNIX considers input and output to programs to be “streams of data” –Could be from/to the user –Could be from/to a file –Could be from/to another program

Redirecting Input and Output Input and output need not only involve the keyboard and screen, it is possible to redirect them to and from files Each UNIX command has at least one input channel and two output channels: –STDIN (0) Input channel –STDOUT (1) Output channel –STDERR (2) Output channel More input and output are usually created by commands to read and write from files that are specified in arguments

STDIN Stands for standard input This is where programs expect to find their input STDIN is set by default to read input from the keyboard If you want to read the input from a file instead, use <

STDOUT Stands for standard output This is where programs (usually) write any output they generate. By default STDOUT appears in your terminal window If you want to save the output to a file instead, use > –The file will be created –If the file already exists then it will be overwritten You can also use >> which appends the output onto a file’s contents

STDERR Stands for standard error This is where programs usually write error messages. So even if you are redirecting the normal output to a file, you can still see error messages on the screen You can redirect STDERR using 2>

By default, UNIX attaches STDIN to the keyboard and STDOUT and STDERR to the screen stdin stdout stderr input files output files Redirecting Input and Output (2)

Use > to redirect STDOUT to a file ls mydir > temp temp will be created or overwritten to contain the normal output of the ls command, although error messages still go to the screen Use 2>&1 to redirect both outputs ls mydir > temp 2>&1 >> is like > except that it appends to an existing file instead of overwriting it ls anotherdir >> temp < redirects the standard input Redirecting Input and Output (3)

Piping Like redirection except that it attaches input and output to other commands instead of files User can build a pipeline of connected commands each of which operates on the output of the one before This is why so many commands take input from stdin when no files are given as arguments (e.g., cat, more, sort, grep, wc) Uses the pipe symbol, ‘|’

ls | more gives paged output from an ls What about: who | grep zlizmj who | grep zlizmj | wc -l who | sort file /etc/* | grep “ascii” Complex pipes can be saved permanently as shell scripts or aliases screen keyboard (stdin) (stdout) (stderr)ls more (stderr) screen Redirecting Input and Output with Pipes

Hidden Files Files whose names start with a dot do not show up in a straight ls command Instead use the -a flag, (i.e., ls -a) These are often special files for configuring the system or different applications.login.bash_logout.bashrc.profile.bash_profile

Hidden Files (2) You can permanently customise your environment by editing your.profile Once you’ve edited it you can apply your changes immediately –Type source.profile (source reads and executes a file) OR –Log out and log in again. The commands in.profile are executed every time you log in

UNIX Shell The UNIX command line interface is called the ‘shell’ There are many different shells, for example csh, bash, tsh, and usually you will run only one type of shell in a login session Different types of shell have different built-in commands, although the core commands are common

Review of Lecture 1 Editing the command line –DELETE or back space to delete the last character (also ^H) –^D to delete the next character –^W to delete the last word –Alt-U to delete the entire line –^C to interrupt most commands –^A and ^E to go to the beginning or end of the line (^X means press the Control key and X at the same time)

Bash facilities There is a history of previously entered commands (called events) that you can see with the command history You can recall and modify these with !!Previous event again !! stringprevious event with string added !n event number n !-nthe nth previous event !prefixlast event that began with prefix !*all the arguments of the last event !^,!$first and last arguments of last event !:5fifth argument of last event many more see bash manual The most useful of them all: ^R –^R searches interactively in the history. –Press enter when you found the one you like (or right arrow to edit it)

Aliases (bash) To define shorthand for complex commands alias name definition defines an alias alias h=history alias ls='ls -F' alias alone shows you current aliases unalias name removes an alias

The.bash_profile file Whenever a login shell starts up it executes this file. Can be used to automatically create aliases and set history length. Could contain: set HISTSIZE=200 alias h=history alias print='lpr -Pmyprint'

Directory stacks (bash) Lets you remember old directories when you change to new ones –pushdputs a new directory on top of the stack –popdremoves it and goes back to the previous one –dirsshows the stack

Directory stacks (bash) (2) / x yzcb a /x/y /a/b /x/y cd /x/y pushd /a/b popd

More UNIX … UNIXhelp for Users: CERN Unix users guide: Commonly Used Unix Commands: Unix Fundamentals: Unix 101: Introduction to Unix Systems Administration: 1.html 1.html

Next Lecture Introduction to the EMACS editor

Summary More UNIX commands Redirection and Pipes Bash facilities