Introduction to Unix (CA263) File Processing

Guide to UNIX Using Linux, Third Edition 2 Objectives Explain UNIX and Linux file processing Use basic file manipulation commands to create, delete, copy, and move files and directories Employ commands to combine, cut, paste, rearrange, and sort information in files Create a script file Use the join command to link files using a common field Use the awk command to create a professional- looking report

Guide to UNIX Using Linux, Third Edition 3 Understanding File Structures Files can be structured in many ways depending on the kind of data they store –Employee information can be stored on separate line separated by delimiters such as colon. This type of record is known as variable-length record. –Another way to create a records is to have fixed number of column for each column. This type of record is known as fixed-length record.

Guide to UNIX Using Linux, Third Edition 4 Understanding File Structures (continued) UNIX/Linux store data, such as letters and product records, as flat ASCII files Three kinds of regular files are –Unstructured ASCII character, you can store any kind of data in any order. You can’t retrieve a particular column, you have to print everything to get what you need. –Unstructured ASCII records, it store data as a sequence of fixed-length record, contains similar information of different persons on different rows. –Unstructured ASCII trees, it is structured as a tree of record, that can be organized as fixed-length or variable length record. Each record contains key that helps in searching record quickly.

Guide to UNIX Using Linux, Third Edition 5 Understanding File Structures (continued)

Guide to UNIX Using Linux, Third Edition 6 Manipulating Files Creating files Delete files when no longer needed Finding a file Combining files using paste command and output redirection Separating files suing cut command Sorting the contents of a file

Guide to UNIX Using Linux, Third Edition 7 Create Files Redirection sign & touch command will create an empty file. $ > accountfiles $ touch accountsfile2 Syntax touch [-options] [filename(s)] Useful Options include: -a update the access time only -m update the last time the file was modified -c prevent creating file, if it does not exist

Guide to UNIX Using Linux, Third Edition 8 Delete Files Delete files or directory permanently when no longer needed $ rm –i phonebook.bak Syntax rm [-options] [filename / directory] Useful Options include: -i display warning before deleting a file -r will remove directory and everything it contains

Guide to UNIX Using Linux, Third Edition 9 Removing Directories Remove directory permanently when no longer needed $ rmdir documents Syntax rmdir [-options] [directory] Useful Options include: -i display warning before deleting a file ??? -r will remove directory and everything it contains Note: Directory must be empty to delete with the rmdir command

Guide to UNIX Using Linux, Third Edition 10 Finding a File Finding a file helps you locate it in the directory structure $ find –i phonebook.bak Syntax: find [pathname] [-name filename] Useful Options include: Pathname:. is for current directory -name indicates that you are searching for file with specific name. You can use wild card

Guide to UNIX Using Linux, Third Edition 11 Combining Files Combining files using output redirection –cat command - concatenate text of two different files via output redirection –paste command - joins text of different files in side by side fashion

Guide to UNIX Using Linux, Third Edition 12 Sorting Sorting a file’s contents alphabetically or numerically. Syntax sort [-option] [filename] Useful Options include: -k n sort on key field specified by n -t indicates a specified char that separate fields -m merges input file that have been previously sorted -o redirect output to the specified file -d sorts in alphanumeric or dictionary order -g sorts by numeric order -r sorts in reverse order

Guide to UNIX Using Linux, Third Edition 13 Sorting Example $ sort –k 3 food >sortedfood $ Cat sortedfood Lettuce SourdoughBeef Spinach White breadChicken Beans PumpernickelMutton Carrots Whole WheatTurkey

Guide to UNIX Using Linux, Third Edition 14 Sorting –n Example The –n option to sort specifies that the first field on the line is to be considered a number. $ cat data $ sort data $ sort –n data

Guide to UNIX Using Linux, Third Edition 15 Sorting +1n Example The +1 say to skip the first field. Similarly, +5n would mean to skip the first five field. $ cat data Skip the first field in the sort $ sort +1n data

Guide to UNIX Using Linux, Third Edition 16 Creating Script Files UNIX/Linux users create shell script files to contain commands that can be run sequentially as a set – this helps with the issues of command automation and re-use of command actions UNIX/Linux users use the vi editor to create script files, then make the script executable using the chmod command with the x argument

Guide to UNIX Using Linux, Third Edition 17 Creating Script Files (continued)

Guide to UNIX Using Linux, Third Edition 18 Using the join Command on Two Files Sometimes you want to link the information in two files The join command is often used in relational database processing The join command associates information in two different files on the basis of a common field or key in those files

Guide to UNIX Using Linux, Third Edition 19 The join Command Syntax: join [-option] [file1 file2] Useful Options include: File1 and file2 are two input files that must be sorted on join field. -1 fieldnum specifies the common join field in file 1 -2 fieldnum specifies the common join field in file 2 -o specifies a list of field to output -t specifies the field separator character, space, tab or new line -a filenum produce a file for each unpairable line -e str replaces empty fields for the unpairable line in the string specified by str.

Guide to UNIX Using Linux, Third Edition 20 sed sed is a program used for editing data. It stands for stream editor. Example: To change first occurrences of “Unix” to “UNIX” on every line of intro $ cat intro The Unix operating system was pioneered by Ken Thompson. Main goal of Unix was to create Unix environment for efficient program development $ sed 's/Unix/UNIX/' intro The UNIX operating system was pioneered by Ken Thompson. Main goal of UNIX was to create Unix environment for efficient program development

Guide to UNIX Using Linux, Third Edition 21 sed with g option Example: To change all occurrences of “Unix” to “UNIX” on every line of intro $ cat intro The Unix operating system was pioneered by Ken Thompson. Main goal of Unix was to create Unix environment for efficient program development $ sed 's/Unix/UNIX/g' intro The UNIX operating system was pioneered by Ken Thompson. Main goal of UNIX was to create UNIX environment for efficient program development

Guide to UNIX Using Linux, Third Edition 22 sed with –n option Just print first 2 lines $ cat intro The Unix operating system was pioneered by Ken Thompson. Main goal of Unix was to create Unix environment for efficient program development $ sed –n '1,2p' intro The Unix operating system was pioneered by Ken Thompson. Main goal of Unix was to create Unix

Guide to UNIX Using Linux, Third Edition 23 sed with –n option Just print first lines containing Unix $ cat intro The Unix operating system was pioneered by Ken Thompson. Main goal of Unix was to create Unix environment for efficient program development $ sed –n '/Unix/p' intro The Unix operating system was pioneered by Ken Thompson. Main goal of Unix was to create Unix

Guide to UNIX Using Linux, Third Edition 24 sed with d option Delete lines 1 and 2 $ cat intro The Unix operating system was pioneered by Ken Thompson. Main goal of Unix was to create Unix environment for efficient program development $ sed '1,2d' intro environment for efficient program development Delete all lines containing Unix $ sed '/Unix/d' intro environment for efficient program development

Guide to UNIX Using Linux, Third Edition 25 A Brief Introduction to the Awk Program Awk, a pattern-scanning and processing language helps to produce professional-looking reports Awk provides a powerful programming environment that can perform actions on files that are difficult to duplicate with a combination of other commands

Guide to UNIX Using Linux, Third Edition 26 A Brief Introduction to the Awk Program (continued) Awk checks to see if the input records in specified files satisfy a pattern If so, awk executes a specified action If no pattern is provided, awk applies the action to every record

Guide to UNIX Using Linux, Third Edition 27 awk command Syntax: awk [-Fsep] [‘pattern {action}…’] [filename] Useful Options include: -F: means the field separator is colon awk ‘BEGIN { print "This is an awk print line." }’ This is an awk print line. awk –F: ‘{printf "%s\t %s\n", $1, $2}’ datafile