2 Some Simple UNIX Filters (Commands That Use Both Standard Input & Standard Output) File LevelContent Levelprcmp, comm, diffsortuniqhead,tailcut, pastetrgrepThere are a lot of others!!
3 Formatting Output: pr Command pr prepares files for printing by adding formatting, headers, footers, …Some options: -k prints in k columns-n numbers the lines of output-d double spaces output-l n sets length to n lines-w m sets width to m charsExample: a.out | pr –n –d –l 64 Prints output with linenumbers, double spacing,and 64 lines per page
4 Comparing Files: cmp Command cmp compares 2 files and stops when it finds a differenceThe comparison is character by character (byte by byte).Option: -l lists all byte differences in the filesExamples: cmp file1 file2file1 file2 differ: char 12, line 3cmp –l file1 file2 | wc –lDisplays the number of differences in the 2 files.
5 Comparing Files: comm Command comm compares 2 files and lists 3 columns of information:1. lines unique to 1st file lines unique to 2nd file lines common to both filesThe files must be sorted.Options: -1, -2, -3 indicates the columns to drop in the outputExamples: comm -3 file1 file2 Lists all the unique lines inboth files.comm –l2 file1 file2 Lists all the lines common to
6 Comparing Files: diff Command diff compares 2 files and lists the instructions neededto make the files the same. Here’s an example:$ diff file1 file23c Change line 3< This is line 3 of file from this---> This is line 3 of file to this.7a8> This is line 8 from file Add this line afterline 7 of file1.
7 Extracting Vertical Data: cut Command cut extracts vertical slices of data from a file. Either columns (-coption) or fields (-f option) of data may be extracted. A delimiter (-doption) may be defined to separate fields. Default delimiter is tab.Examples: cut –c1-4,8,15- file1 Extracts chars 1 thru 4, the 8thcharacter, and characters 15 thru end of each line from file1. Nowhitespace in the column list!!cut –d: -f1-3 file2 Extracts fields 1 thru 3 from file2. Thefields are separated by the : characterwho | cut –d” “ –f1 Lists the names of all users logged in.
8 Joining Vertical Data: paste Command paste vertically joins 2 files together. A delimiter (-d option)may be defined to separate fields. Default delimiter is tab. The-s option joins lines of a single file together.Examples: paste file1 file2 Displays file1 and file2 side by side.paste –s –d”::\n” addressbookrick>>>>
9 Displaying Files: head and tail Commands head displays the top of a file. (1st 10 lines, by default).tail displays the end of the file (last 10 lines, by default).Options: -n x or –x displays 1st (last) x lines of the file.-f continuously displays the end of a file as it grows. Thisoption is for the tail command only. You must use theinterrupt key to stop monitoring the file growth.Examples: ls –t | head –n Displays the file most recently edited.tail –f install.log Continuously displays the log file asit grows. Use interrupt key to stop.
10 Ordering Files: sort Command sort reorders the lines of a file in ascending (descending) order.The default order is ASCII: whitespace, numbers, uppercase, andfinally, lowercase letters.Options: -k n sort on the nth field of the line-tchar use char as the field delimiter-n sort numerically-r reverse order sort-u remove repeated lines-m list merge sorted files in list
11 sort ExamplesExamples: sort –t: -k 2 list Sort on the 2nd field of file list. Fields areseparated by :sort –t: -k 5.7 –r list Sort file list in reverse order on the7th character of 5th field. Fields separated by :sort –n list Numerically sort file list, assumed to contain numbers.sort –m file1 file2 Sorted files file1 and file2 are merged.cut –d: -f3 list | sort –u Extract the 3rd field from list &sort that field, removing the repeated lines.
12 Removing Duplicates: uniq Command uniq displays a presorted file, removing all the duplicate lines from it.If 2 files are specified, uniq reads from the first and writes to the second.Options: -u lists only the lines that are unique-d lists only the lines that are duplicates-c counts the frequency of occurrencesExamples: sort list | uniq – xlist Sorts file list; uniq reads fromstdin and writes the output to xlist.uniq –c list Displays count of each unique line in the file list.
13 Character Manipulation: tr Command tr translates characters from one format to another. Input always comesfrom standard input. Arguments don’t include filenames. General form is:tr options expression1 expression2 standard inputexpression1 is the set of characters to change; expression2 is whatthey change to. (The expressions should be equal length.)Examples: tr ‘+-’ ‘*/’ < math In the file math, replace all +’s with *’s andreplace all -’s with /’s.head –n 3 list | tr ‘[a-z]’ ‘[A-Z]’ The 1st 3 linesof the file list are translated to uppercase.
14 tr Command Options Options: -d delete characters from the input stream -s compress multiple consecutive characters (squeeze)-c complementing value of expressionExamples: tr –d ‘/’ < dates Remove all /’s from the file dates.tr –s ‘ ’ < names Replace all strings of blanks with a singleblank in the file names.tr –cd ‘:’ < file1 In the file file1, delete everything that isn’ta colon (:). All that’s left is a file full of :’s
15 Finding Patterns in Files with grep grep searches a file and displays the lines containing a pattern. Form is:grep options pattern filesIf more than 1 file is listed, the filename is also displayed in the output.Some options: -i ignore case when matching-n display line numbers as well as lines-c displays a count of the number of occurrencesExamples: grep “professor” college.lst Displays all lines in file,college.lst, that contain the string professor.grep –i “Rick” college.lst Displays all lines in the file,college.lst, that contain the string Rick. (Also finds rick, RICK, rIcK, …)
16 Regular Expressions in grep Regular expressions are metacharacter patterns used in ways differentfrom how the shell uses them.Regular Expression Meaning* 0 or more of the previous character. a single character[pqr] a single p or q or r[c1-c2] a single char in the ASCII range of c1 thru c2[^pqr] a single character not p nor q nor r^abc abc at the beginning of lineabc$ abc at the end of the line
17 Example Regular Expressions with grep grep “g*” file1 Displays all lines in file1 that contain nothing or g, gg, ggg, …grep “.*” file1 Displays all lines in file1 that contain nothing or any # of charsgrep “[1-3]” file1 Displays all lines in file1 that contain a digit between 1 & 3.grep “[^a-zA-Z]” file1 Displays all lines in file1 that contain a non-alphabeticcharactergrep “^Rick$” file1 Displays all lines in file1 that contain only Rickgrep “^$” file1 Displays all lines in file1 that contain nothinggrep “R[aeiou]ck” file1 Displays all lines in file1 that contain Rack, Reck,Rick, Rock or Ruck
18 Putting It All Together An author wants to count the frequency of words used in a book chapter.1. Put each word on a separate line:tr “ \011” “\012\012” < chapter2. Strip out everything that isn’t an alphabetic character or newline:tr -cd “[a-zA-Z\012]”3. Sort the list: sort4. Count the word frequency: uniq –c5. Put it all together:tr “ \011” “\012\012” < chapter |tr –cd “[a-zA-Z\012]” | sort | uniq -c