1. Chapter 3 UNIX Utilities II By C. Shing ITEC Dept Radford University

2. Objectives Understand how to filter files Understand how to sort files Understand how to search for files Understand how to compare files Understand how to archive, transform and compress/uncompress files Understand how to schedule processes Understand hard and soft links Understand how to attach different file systems Understand how to find every character in file

3. Filters (Find file Content) grep (Global or Get Regular Expression and Print), egrep (Extended grep), fgrep (Fast grep) uniq awk (Aho, Weinberger, Kernighan): C control structures sed (Stream Editor)

4. Filters I.grep, egrep, fgrep: filter out lines that do/don’t have specified pattern in the file content 1.grep Syntax: grep -option pattern filename  Option:  i: ignore case  l: list files containing the pattern  n: display line #  v: displays lines that don't match the pattern  w: match to whole word only  Pattern: regular expression

5. Filters- grep PatternExplain.match any single character *The preceding item match 0 or more characters ^abegins with a a$end with a

6. Filters- grep PatternExplain \.match the. character (Do not use meta- character meaning of.) [a-z]match any character within a to z range [^a-s] tvmatch any word not starting or continue within a to s range but end with tv

7. Filters- grep Examples: –grep smith /etc/passwd show all lines that have smith pattern –grep -wn lower /usr/include/* use word: lower and show line # –grep -ni lower /usr/include/* ignore case and show line number –ls -l | grep ^d show directory –ls -l | grep ‘^..w’ show files that owner can write –cut -d: -f5 /etc/passwd | grep -i ' smith$' | more show under field 5, any line has word end with smith case insensitive

8. Filters- egrep 2.egrep Syntax: egrep -option pattern filename or grep –e -option pattern filename  Option:  i: ignore case  l: list files containing the pattern  n: display line #  v: displays lines that don't match the pattern  w: match to whole word only  Pattern: extended regular expression

9. Filters- egrep Extended (extra) Pattern Explain +The preceding item match 1 or more characters ?The preceding item match 0 or 1 character |or ()put regular expression in ()

10. Filters- egrep Examples: –egrep sm.?ith: /etc/passwd all lines contain 0 or 1 of any character after sm –egrep -wn stdio \| STDIO /usr/include/stdio.h any line containing stdio or STDIO –egrep -ni to+ /usr/include/* | grep too all lines contain too –ls -l /devices| egrep –v ^d show all non-directory files (device has major #: routine #, minor #: index in the routine) –cut -d: -f5 /etc/passwd | egrep -i ' smith.+$' | more any line containing smith followed by 1 or more characters

11. Filters- fgrep 3.fgrep Syntax: fgrep -option string filename  Option:  i: ignore case  l: list files containing the pattern  n: display line #  v: displays lines that don't match the pattern  w: match to whole word only

12. Filters- fgrep Examples: –fgrep smith /etc/passwd –fgrep –wn lower /usr/include/*

13. Filters- uniq II.uniq: filter out duplicated adjacent lines Syntax: uniq -option [infile [outfile]]  Option:  number: # of fields to be skipped (delimiter: space)  c: precedes # of duplications

14. Filters- uniq Examples: –cut –d: -f4 /etc/passwd | uniq -c list # of same adjacent GID lines

15. Compare - cmp cmp: find 1 st byte that is different Syntax: cmp -option file1 file2  Option:  l: list offset and line number of 1 st mismatched byte  S: all output inhibited

16. Compare - diff diff, sdiff: display differences so file1 can be converted to file2 (sdiff shows actual lines involved) Syntax: diff -option file1 file2 or sdiff -option file1 file2  Option:  i: ignore case

17. Compare - diff Output form: 1.File1Line# a File2Line#Start, File2Line#Stop > 2.File1Line#Start, File1Line#Stop d lineCount < 3.File1Line#Start, File1Line#Stop c File2Line#Start, File2Line#Stop < --- >

18. Compare - diff Example: –diff /etc/auto_net_rucs /etc/auto_net_rucs2 –diff /etc/auto_net_rucs /etc/auto_net_ruacad

19. Sort Sort: in ascending order Syntax: sort –option file  Option:  t: c-delimiter default - space or tab  +n1: n1 is the index number for the starting sort field  -n2: n2 is the stopping sort field index number (non- inclusive)  b: ignore leading blanks  f: ignore case for sort fields  n: sort fields in numerical order  M: sort fields in month order  r: sort in reverse (descending) order  +n3, -n4: more comparison when equal in using +n1,-n2

20. Sort Examples: –sort +0 -1 -b /usr/include/stdio.h starts from and to the 1st field (index=0), ignore leading spaces and sort /usr/include/stdio.h in ascending order (delimiter is space or tab). –sort -t: +4 -5 -b +0r /etc/passwd | more Use : as the delimiter of fields, sort in ascending order according the Real Name (index=4) in /etc/passwd, ignore leading spaces; if sorted fields are equal, then sort by the username in descending order –sort -t: +2 -3n /etc/passwd | more sort by uid (numerically)

21. Search (Find file/directory name/attribute) Find: search the expression from the path Syntax: find path expression

22. Search ExpressionExplain -name patternpattern can use meta-characters such as * -type chFile type: ch can be d, b, c -user usernameor use uid -group groupnameor use gid -atime -daycountaccessed within the previous # of days -perm octPermission with oct number

23. Search ExpressionExplain -mtime -daycountmodified within the previous # of days -ctime -daycountchanged within the previous # of days -printprint matched in screen -lslist file attributes -exec commandexecutes command when found. If specify {} in command, it is replaced by all found items end with \;

24. Search ExpressionExplain expr1 -a expr2expr1 and expr2 expr1 -o expr2expr1 or expr2 -maxdepth levelsDefault is recursively to leaf

25. Search Examples: –find ~ -name '*.c' -print | more It recursively goes down from your home directory to find any c files and print on screen. –find. -mtime -14 –print | more It displays all files that has been modified within the past 2 weeks. -find ~ -perm 4701 –ctime -7 superuser needs check any executable set to suid within last 7 days(maybe hacker)

26. Search Examples: –(find / -name handler.py > /dev/tty) > & /dev/null This finds the python file and sends errors to /dev/null, and results are displayed on the terminal, so you don’t see error in screen –find / -name handler.py >& errorfile The error will goes to the file errorfile

27. Search (Cont.) Examples: (Cont.) –find. -name a.out -ls -exec rm {} \; It recursively goes down from the current directory to delete a.out files and print those names on screen. (This rm command may execute 100 times if there are 100 a.out files found) Note: a more efficient way is to use xargs command as below: (Linux only) find. -name a.out -ls | xargs rm (This will put all files found in the parameters of the command rm and execute once) If the command line parameters of rm has limited to 10, then we can use find. -name a.out -ls | xargs –n10 rm (This will execute the command 10 times, each time it removes 10 files.)

28. Search (Cont.) Examples: (Cont.) –find. \( -name '*.c' -o -name '*.java' \) - print | more It recursively goes down from the current directory to find any c files or java files and print on screen.

29. Filters - awk III.awk: scan every line that matches certain criteria (Programmable Text Processing ) or no condition, perform some actions to them Use C syntax and C library function Syntax: each line in form - [condition][{action}] –awkPgm (use –f): line in file awk -option -f awkPgm file –Command: line (in quotes) awk -option '[condition][{action}]' file  Option:  Fc: c – Fields delimiter tab or space - default delimiter

30. Filters - awk Condition Explain BEGIN/END pair1st line before loop is right after the BEGIN, 1st line after loop is right after the END expressioncombination of logical operators, relational operators and regular expressions /regular expr/Lines having reg. expr pattern /regular expr1/, /regular expr2/ the range between lines having reg. expr1 and reg. expr2

31. Filters - awk Action Explain if (conditional) statement [else statement] while (conditional) statement for (expr; conditional; expr) statement break continue var = expr

32. Filters - awk Action Explain print [list of expr] printf format[,list of expr] next exit list of statements

33. Filters - awk Examples:(all necessary Unix format files(Do not use DOS format) are in the folder)the folder 1.awk -F: '{print NF, $0}' /etc/passwd|more where NF: # of fields (predefined variable), $0: entire line 2.awk -F: -f awk1 /etc/passwd|more It prints 1st, 3rd fields and number of fields (no spaces in between)

34. Filters - awk 3.awk -F: -f awk2 /etc/passwd | more It prints out 1st, 3rd fields and number of fields (with spaces in between) for line 2 and 3(NR: record number) 4.awk -F: -f awk3 /etc/passwd | more It prints out line # along with each line and count total # of line s and fields. 5.awk -F: -f awk4 /etc/passwd | more It prints out from the last field back to the 1st field for each line.

35. Filters - awk 6.awk -F: -f awk5 /etc/passwd |more It prints out the line that have the pattern (t followed by one or more characters and end with e). 7.awk -F: -f awk6 /etc/passwd |more It prints out the line that contains "smith". 8.awk -F: -f awk7 /etc/passwd |more It prints out the line that contains "Smith:".

36. Filters - awk 9.awk -F: -f awk8 /etc/passwd |more It prints out all lines in between (inclusively) the beginning line that contains "root" to the ending line that contains "local“ 10.awk -f awk9 awktestlib It shows awk library functions.

37. Filters - awk 11. Example of using if.. elseExample of using if.. else

38. Filters - sed IV.sed: scan every line and perform simple editing action on lines that match certain criteria Syntax: sed -option scriptfile filename  Option:  f: scriptfile is file of command lines no option: use command line in quotes  n: suppress print duplication caused by p  e: multiple instructions in command line

39. Filters - sed CommandExplain s/expr/str filesubstitute the first occurrence of the expr by str in the file s/expr/str/g filesubstitute all occurrences of the expr by str in the file /expr1/, /expr2/w newfile Select all lines such that the 1 st one contains expr1 and the last line has expr2 and save them into a newfile /expr/d filedelete every line that contains expr in the file n1,n2d filedelete from line n1 to line n2 in the file

40. Filters - sed CommandExplain n1q fileQuit after reading the first n1 lines of the file n1,n2p fileprints duplication from line n1 to line n2 in the file (suppress the duplication: must use -n option) Address i\ text Insert text before line #address Address a\ text Append text after line #address AddrRange c\ text Change line #address range by text

41. Filters - sed Examples:(all necessary Unix format files(Do not use DOS format) are in the folder)the folder 1.sed 's/^/\$ /' awk1 > awk1.indent This gives a $ at the beginning of each line in the file awk1 and creates output file awk1.indent. 2.sed 's/^ *//' awk3 This removes leading spaces in each line of the file awk3. 3.sed -f sed1 awk3 This removes all lines that contains $ in the file awk3. 4.sed '2,4d' awk9 This removes line 2 to line 4 in the file awk9.

42. Filters - sed 5.sed '$d' awk9 This removes last line in the file awk9. 6.sed -n '2,4p' awk9 This prints from line 2 to line 4 in the file awk9. 7.sed 5q awk9 This prints from line 1 to line 5 in the file awk9. 8.sed –f sedinsert awk9 This inserts 1 st and last line. 9.sed –f sedchange awk9 This changes the last line. 10.sed –f sedreplace awk9 This replace all “printf” by “fprintf” 11.sed ‘1,$s/the/THE/g’ awk9 replace the word ‘the’ by ‘THE’ in the entire file awk9 12.sed –n –e ‘1,2p’ –e ‘$p’ awk9 prints line 1,2 and the last line of the file awk9 13.sed –n –e ‘/ /,/ /w table.html’ –e ‘/ /,/ /w frame.html’ entire.html extract table and frame parts of the entire.html and save them into individual files

43. Archive cpio (not in MS-DOS): small files tar (also available in MS-DOS) : backup on tape - single volume dump (or ufsdump): incremental backup can store in many volumes –restore (or ufsstore): recover

44. Archive - cpio cpio Syntax: cpio –option [file][directory]  Option:  o: takes standard input and creates output cpio format files with those filenames  v: displays those filenames  i: takes a cpio format file from standard input and creates those filenames  d: creates directory  t: creates table of contents  p: copies files into the directory  l: creates links (instead of copying) in the directory

45. Archive - cpio Examples: –mkdir tmp cp *.c tmp cd tmp ls *.c | cpio -ov > backup It creates a cpio format output file conatining all c files –ls -l backup rm -f *.c ls cpio -i < backup It recreates all original files from the cpio file. ls -l cd.. rm -rf tmp

46. Archive - tar tar Syntax: tar –option [tarfilename] filelist  Option:  c: creates tar format file  f: tarfile  r: unconditionally append to the tarfile  t: creates table of contents  v: verbose output  x: extract from tarfile  Z: compress file

47. Archive - tar Examples: –tar -cvf tarfile *.c tar all c files to the current firectory –tar -tvf tarfile read current contents of tarfile –tar -rvf tarfile *.txt unconditionally adds all txt files to the end of the tarfile –tar -tvf tarfile | more mkdir tmp mv tarfile tmp cd tmp –tar -xvf tarfile `tar -tf tarfile | grep filecp.c ` It only untar the file filecp.c from the tarfile. –tar czf tarcompressfile achive and then compress file

48. Archive – dump/restore dump Syntax: dump level f option dumpfile filesystem where dumpfile is defaulted to /etc/rmt0  Option:  v: verify each volumn  w: displays filesystems that need to be dumped

49. Archive – dump/restore Examples: –dump 0 fv /dev/rmt0 /dev/da0

50. Archive – dump/restore restore Syntax: restore -option f dumpfile files where dumpfile is defaulted to /etc/rmt0  Option:  t: show table of contents  x: restore only files specified  i: interactive mode

51. Archive - dump/restore Examples: –restore -x f /dev/rmt0 file1.c file2.c

52. Transform tr: translate from string1character set to string2 character set, respectively. If len(string2)<len(string1), pad the last character in string2 as long as possible Syntax: tr –option string1 string2  Option:  c: complement of string1 set  d: string1 set deleted  s: show once for duplicated characters

53. Tee Duplicates input and save it in a file and creates standard output Example: who | tee who.txt | wc –l This saves users in the file who.txt and output number of users in screen

54. Transform Example –tr “[:lower:]” “[:upper:]” < /usr/include/stdio.h | more (Solaris) change from lower case to upper case –tr “a-d” “A-D” < /usr/include/stdio.h | more –tr –cs “a-z” “\015” < /usr/include/stdio.h | more for non-(a-z) character, replace by new-line character only once

55. Compress/Uncompress Compress/Uncompress (System V) Syntax: –compress -option file –uncompress file.Z  Option:  c: send result to screen, instead of replacing the old file by a.Z file  v: verbose output

56. Compress/Uncompress Examples: –compress –v filename –uncompress –v filename.Z

57. Compress/Uncompress gzip/gunzip Syntax: –gzip -option file –gunzip file.gz  Option:  c: send result to screen, instead of replacing the old file by a.gz file  f: force GNU free compress program

58. Compress/Uncompress zip/unzip (Linux) Syntax: –zip -option file.zip file1 file2 … –unzip file.zip  Option:  r: recursively descend down its tree to the leaves PKZIP/PKUNZIP compress program

59. Schedule Process Periodic execution: crontab only users in cron.allow can use if the file exists all users in cron.deny are not allowed to use if the file exists One time execution: at similar for files: at.allow and at.deny One time execution when CPU is available: batch e.g. batch < job.sh

60. Schedule Process - crontab crontab: schedule a crontab process periodically Syntax: crontab -option  Option:  l: lists contents of a registered crontab file  e: update and reregister the registered crontab file  r: unregister the registered crontab file

61. Schedule Process - crontab A crontab file must be in the Unix form (not DOS): –minute hour day month weekday command See the file crontab.cron in the folderfolder Note: 1.Avoid to use * for minute field 2.All fields: –List: e.g. 2,4,6 –Range: e.g. 2-6/2 (as above) 2-6

62. Schedule Process - crontab Examples: –crontab crontab.cron This registers the crontab file –crontab -e This can edit the crontab file and reregister it –crontab -l This lists the contents of the crontab file –crontab -r This unregister the crontab file.

63. Schedule Process - at at: schedule a one-time process Syntax: –at time < shellscript This starts to run the shellscript at the future time. –at -l This checks all future jobID. –at -r jobID This removes the future jobID.

64. Schedule Process - at Examples: (See the file at.csh in the folder)folder –at 4:30pm < at.csh This program at.csh will be executed at 4:30pm today. –at now +1 day < at.csh This program at.csh will be executed at exactly one day from now. –at 5pm This sets interactive mode to type in all shell command and finish it be ctrl-d

65. Unix File System Introduction

66. Link - hard Hard link Syntax: ln filename aliasename Hard link will creates an alias called aliasname having the same inode number (same file)

67. Link – hard (Cont.) Limitations: 1.Can’t link file across different file systems 2.Can’t link directory

68. Link - hard Examples: link directory –mkdir temp –ln *.c temp This creates hard links to all c files and put them in temp directory –ls –li –cd temp –ls -li

69. Link - soft Soft link Syntax: ln -option filename[directory] softlink  Option:  s: soft link Soft link creates a different file (different inode), it can link to different file systems Note: This is similar to short-cut in Windows.

70. Link - soft Example: usually used to link directory –ln -s /usr/include myincludedir –cd myincludedir –ls -l

71. Link - soft Example: link file –ln file1 file2 –ln –s file1 file3 –cat file2 –cat file3 –ls –i file* –rm –f file2 –ls –i file* –rm –f file1 –ls –i file* –ls –l file* –cat file3 file3 becomes a dangling link Check their inode and file content in file3

72. Attach/Detach File System – mount/umount Mount/umount: attach/detach file system Syntax: –mount -o options devicename mountpoint  Option:  ro: read only  rw: read and write (default) –mount list currently mounted devices –umount mountpoint detach file system

73. Attach/Detach File System – mount/umount Examples: Use jump drive on Red Hat Linux 8 –mkdir /usb –mount /dev/sd0 /usb –mount –cp * /usb –umount /usb

74. Raw File Content - od od: dump non-text file content in octal form Syntax: od –option filename  Option:  c: in character form  s[n]: search string of length n (default 3)  x: in hexadecimal form  b: in Octal form (Linux)

75. Raw File Content - od Examples: –od –c java1.class | more look at characters in bytecode file java1.class –od –cxs5 a.out | more look at characters (string length at least 5, also in hexadecimal form) in executable file a.out –od –bc a.out | more

76. Misc. whoami cut ul: transform an underline sequence file to an ASCII text file –man cut | ul –tdumb > cut.txt spell: spell checker crypt keyvalue cryptFile To encrypt the file origFile. crypt keyvalue < cryptFile To decrypt the file cryptFile.

77. Misc. time command Find user, system, and elapsed times, CPU % for running the command. paste file1 file2 Align 2 files side by side nohup command Do not kill process created by the command when logout

78. Misc. df check usage of mounted file systems dos2unix dosfilename unixfilename convert dos file to unix file unix2dos unixfilename dosfilename convert unixfile to dosfile last username: show last login time strings executable: examine executable string contents dd : create bit by bit image copy paste file1 file2: put file2 side by side with file 1

79. Perl Introduction

80. Reference Appendix A