1. Guide To UNIX Using Linux Third Edition
Chapter 5:
Advanced File Processing
Guide To UNIX Using Linux Third Edition

2. Objectives
Use the pipe operator to redirect the output of one command to another command
Use the grep command to search for a specified pattern in a file
Use the uniq command to remove duplicate lines from a file
Use the comm and diff commands to compare two files
Use the wc command to count words, characters and lines in a file
Guide to UNIX Using Linux, Third Edition

3. Objectives (continued)
Use manipulation and transformation commands, which include sed, tr, and pr
Design a new file-processing application by creating, testing, and running shell scripts
Guide to UNIX Using Linux, Third Edition

4. Advancing Your File-Processing Skills
Selection commands focus on extracting specific information from files
Guide to UNIX Using Linux, Third Edition

5. Advancing Your File Processing Skills (continued)
Manipulation and transformation commands alter and transform extracted information into useful and appealing formats
Guide to UNIX Using Linux, Third Edition

6. Advancing Your File Processing Skills (continued)
Guide to UNIX Using Linux, Third Edition

7. Using the Selection Commands
Using the Pipe Operator
– The pipe operator (|) redirects the output of one command to the input of another
An example would be to redirect the output of the ls command to the more command
The pipe operator can connect several commands on the same command line
Guide to UNIX Using Linux, Third Edition

8. Using the Pipe Operator
Using pipe operators and connecting commands is useful when viewing directory information
Guide to UNIX Using Linux, Third Edition

9. Using the grep Command
Used to search for a specific pattern in a file, such as a word or phrase
grep’s options and wildcard support allow for powerful search operations
You can increase grep’s usefulness by combining with other commands, such as head or tail
Guide to UNIX Using Linux, Third Edition

10. Using the uniq Command Removes duplicate lines from a file
Compares only consecutive lines, therefore uniq requires sorted input
uniq has an option that allows you to generate output that contains a copy of each line that has a duplicate
Guide to UNIX Using Linux, Third Edition

11. Using the uniq Command (continued)
Guide to UNIX Using Linux, Third Edition

12. Using the uniq Command (continued)
Guide to UNIX Using Linux, Third Edition

13. Using the comm Command
Used to identify duplicate lines in sorted files
Unlike uniq, it does not remove duplicates, and it works with two files rather than one
It compares lines common to file1 and file2, and produces three column output
Column one contains lines found only in file1
Column two contains lines found only in file2
Column three contains lines found in both files
Guide to UNIX Using Linux, Third Edition

14. Using the diff Command
Attempts to determine the minimal changes needed to convert file1 to file2
The output displays the line(s) that differ
Codes in the output indicate that in order for the files to match, specific lines must be added or deleted
Guide to UNIX Using Linux, Third Edition

15. Using the wc Command
Used to count the number of lines, words, and bytes or characters in text files
You may specify all three options in one issuance of the command
If you don’t specify any options, you see counts of lines, words, and characters (in that order)
Guide to UNIX Using Linux, Third Edition

16. Using the wc Command (continued)
The options for the wc command:
–l for lines
–w for words
–c for characters
Guide to UNIX Using Linux, Third Edition

17. Using Manipulation and Transformation Commands
These commands are: sed, tr, pr
Used to edit and transform the appearance of data before it is displayed or printed
Guide to UNIX Using Linux, Third Edition

18. Introducing the sed Command
sed is a UNIX/Linux editor that allows you to make global changes to large files
Minimum requirements are an input file and a command that lets sed know what actions to apply to the file
sed commands have two general forms
Specify an editing command on the command line
Specify a script file containing sed commands
Guide to UNIX Using Linux, Third Edition

19. Translating Characters Using the tr Command
tr copies data from the standard input to the standard output, substituting or deleting characters specified by options and patterns
The patterns are strings and the strings are sets of characters
A popular use of tr is converting lowercase characters to uppercase
Guide to UNIX Using Linux, Third Edition

20. Using the pr Command to Format Your Output
pr prints specified files on the standard output in paginated form
By default, pr formats the specified files into single-column pages of 66 lines
Each page has a five-line header containing the file name, its latest modification date, and current page, and a five-line trailer consisting of blank lines
Guide to UNIX Using Linux, Third Edition

21. Designing a New File-Processing Application
The most important phase in developing a new application is the design
The design defines the information an application needs to produce
The design also defines how to organize this information into files, records, and fields, which are called logical structures
Guide to UNIX Using Linux, Third Edition

22. Designing Records
The first task is to define the fields in the records and produce a record layout
A record layout identifies each field by name and data type (numeric or nonnumeric)
Design the file record to store only those fields relevant to the record’s primary purpose
Guide to UNIX Using Linux, Third Edition

23. Linking Files with Keys
Multiple files are joined by a key: a common field that each of the linked files share
Another important task in the design phase is to plan a way to join the files
The flexibility to gather information from multiple files comprised of simple, short records is the essence of a relational database system
Guide to UNIX Using Linux, Third Edition

24. Guide to UNIX Using Linux, Third Edition

25. Creating the Programmer and Project Files
With the basic design complete, you now implement your application design
UNIX/Linux file processing predominantly uses flat files
Working with these files is easy, because you can create and manipulate them with text editors like vi and Emacs
Guide to UNIX Using Linux, Third Edition

26. Creating the Programmer and Project Files (continued)
Guide to UNIX Using Linux, Third Edition

27. Formatting Output The awk command is used to prepare formatted output
For the purposes of developing a new file-processing application, we will focus primarily on the printf action of the awk command
Awk provides a shortcut to other UNIX/Linux commands
Guide to UNIX Using Linux, Third Edition

28. Using a Shell Script to Implement the Application
Shell scripts should contain:
The commands to execute
Comments to identify and explain the script so that users or programmers other than the author can understand how it works
Use the pound (#) character to mark comments in a script file
Guide to UNIX Using Linux, Third Edition

29. Running a Shell Script
You can run a shell script in virtually any shell that you have on your system
The Bash shell accepts more variations in command structures that other shells
Run the script by typing sh followed by the name of the script, or make the script executable and type ./ prior to the script name
Guide to UNIX Using Linux, Third Edition

30. Putting it All Together to Produce the Report
An effective way to develop applications is to combine many small scripts in a larger script file
Have the last script added to the larger script print a report indicating script functions and results
Guide to UNIX Using Linux, Third Edition

31. Chapter Summary
UNIX/Linux file-processing commands are (1) selection and (2) manipulation and transformation commands
uniq removes duplicate lines from a sorted file
comm compares lines common to file1 and file2
diff tries to determine the minimal set of changes needed to convert file1 into file2
Guide to UNIX Using Linux, Third Edition

32. Chapter Summary (continued)
tr copies data read from the standard input to the standard output, substituting or deleting characters specified
sed is a file editor designed to make global changes to large files
pr prints the standard output in pages
Guide to UNIX Using Linux, Third Edition

33. Chapter Summary (continued)
The design of a file-processing application reflects what the application needs to produce
Use record layout to identify each field by name and data type
Shell scripts should contain commands to execute programs and comments to identify and explain the programs
Guide to UNIX Using Linux, Third Edition