1. Software Tools Design COEN 276
Dr. John Xiao
Winter 1998
4/5/2019
COEN276 Winter 98 John Xiao

2. Overview
Course Description
4/5/2019
COEN276 Winter 98 John Xiao

3. Grading Policy * 20% late penalty per day 4/5/2019
COEN276 Winter 98 John Xiao

4. Quality of Assignments
Programming assignments
Individual effort!
Design and Implementation
Correct
Efficient
Maintainable
Documentation
Comments
Man pages
Java Doc
4/5/2019
COEN276 Winter 98 John Xiao

5. Objectives Learn Unix Learn existing Unix tools
Learn to build new tools
practice writing software (lots of practice)
learn software design from basic to complex
4/5/2019
COEN276 Winter 98 John Xiao

6. Objectives (Continued)
Skills to learn
Unix utilities, system calls, and IPC
Shell programming
Perl programming
World-Wide Web, HTML, CGI
JavaScript
Java
X11/Motif Window programming (depent on schedule)
4/5/2019
COEN276 Winter 98 John Xiao

7. Lecture One
Unix
4/5/2019
COEN276 Winter 98 John Xiao

8. History of Unix (Continued)
In 1969, Ken Thompson at Bell Labs started on DEC PDP-7
In 1970, Ken Thompson and Dennis Ritchie moved it to PDP-11/20. During this time, Unix attracted enthusiasm and gained credibility as a small general-purpose time-sharing system
Richie designed and wrote a C compiler
In 1973, Ritchie and Thompson rewrote the kernel in C
BSD - Berkeley Software Distribution (popular due to introduction of C shell, which gave a C-like syntax.
AT&T releases Bourne shell
4/5/2019
COEN276 Winter 98 John Xiao

9. History of Unix (Continued)
In 1974, licensed to universities for educational purposes
In 1976, available for comercial use (Version 6)
In 1978, the first portable (Version 7) available
1979, Berkeley 3BSD derived from Version 7
4/5/2019
COEN276 Winter 98 John Xiao

10. History of Unix (Continued)
1983, System V from Bell Labs as industry standard
1984, Microsoft releases Xenix
1986, BSD 4.3 (last release), Bell Labs Version 9
1987, SVR4, Mach, …
4/5/2019
COEN276 Winter 98 John Xiao

11. Unix System in the 90’s SunOS (BSD) Novell Unix Free BSD AIX SCO Unix
Solaris
Linux
HPUX
Mach (Both BSD and System V emulator on Mach microkernel)
4/5/2019
COEN276 Winter 98 John Xiao

12. Why Unix So Successful Clean minimal design
Written in C, it is portable
Source code available, written in a high-level languate, easy to adapt
Kernel - manages resources ( memory, processes, file system)
Shell: command interpreter
File system: basic organization of information.
4/5/2019
COEN276 Winter 98 John Xiao

13. Why Unix So Successful (Cont.)
Rich and productive environment for programmers (Not naive user friendly)
Lots of Unix programs do trivial things in isolation, but when combined with other programs, become quite useful tools.
Provides various easy ways of combining programs
Rich set of useful tools
4/5/2019
COEN276 Winter 98 John Xiao

14. Major Unix OS features Kernel Shell File System 4/5/2019
COEN276 Winter 98 John Xiao

15. Unix Operating System Structure
OS mediates between the user and the computer
User
Application Programs
Commands/Shell Scripts
Shell
Libraries
Shell - Whenever you log in,you get placed in a program called a shell.
You can see the shell prompt.
Enter commands at the prompt.
Shell acts as a command interpreter,
taking commands and passing them on to the kernel.
Kernel: At the core of the Unix system,
loaded whenever the system is started up (booted).
Manages computer resources (memory, processes, file system)
manages memory and allocates it to each process
schedules work done by the CPU
organizes transfer of data from one part of machine to another
accepts instructions from shell and carries them out
enforces access permission on the file system
File System
Kernel
HAL
Hardware
4/5/2019
COEN276 Winter 98 John Xiao

16. Kernel Manages memory and allocates it to each process
Schedules work done by the CPU
Organizes transfer of data from one part of machine to another
Accepts instructions from shell and carries them out
4/5/2019
COEN276 Winter 98 John Xiao

17. Shell Command interpreter Create customized environments
Write shell scripts
Define command aliases
Manipulate command history
File and command completion
Edit the command line
Nice shell comparison summary at
Des criptions at shell/oview1.html
4/5/2019
COEN276 Winter 98 John Xiao

18. File System File: basic storage unit. Types:
Logical method for organizing and storing data
File: basic storage unit.
Types:
ordinary file (stores info)
directory (holds other files and directories)
special file (represents physical devices like printers, terminals, etc)
pipe (temporary file for command linkage)
4/5/2019
COEN276 Winter 98 John Xiao

19. UNIX: Multi-user Multi-tasking
More than one user can use the machine at the same time and more than one task can run at the same time
Unix is multiuser multitasking, Window NT is, Windows 3.X is not.
4/5/2019
COEN276 Winter 98 John Xiao

20. Processes In Unix, each program is started as a process.
A process is a program in execution.
Usually only one copy of a program, but there may be many processes running the same program.
To each interactive user (window):
only one process in foreground
may have several processes in background
4/5/2019
COEN276 Winter 98 John Xiao

21. Processes Process 0: Kernel bootstrap. Start process 1. kernel
kernal mode
user mode
Process 1: start other processes
/etc/init
fork
exec
fork
exec
inetd
lpd
/etc/getty
/etc/getty
condition terminal for login
httpd
exec
exec
At start up time, process 0 launched in the kernel mode (boot)
Process0 starts process1 in the user model (manual transfer of control.
Process 1 starts other processes (fork system call).
Each time a command/program is executed, the shell forks a new process for it.
/bin/login
/bin/login
check password
exec
exec
shell
shell
command interpreter
4/5/2019
COEN276 Winter 98 John Xiao

22. Unix Startup Init process Getty process last step in booting procedure
create other processes to allow the users to login
Getty process
conditions for terminal connection
waits for user-id
display login on the screen
4/5/2019
COEN276 Winter 98 John Xiao

23. Unix Login Login process Shell process (command line interpreter)
check password with the uid
execute .profile or .login (depends on default shell)
display shell prompt
Shell process (command line interpreter)
Shell prompt ($, %)
4/5/2019
COEN276 Winter 98 John Xiao

24. UNIX Process Process environment
Process id, parent-process-id, process-group-id
Opened files
Working directory
File creation mask
Real user ID and group ID
Effective user ID and group ID
Resource limits
4/5/2019
COEN276 Winter 98 John Xiao

25. UNIX Process (Continued)
Environment variables
Signal action setting
Code
A child process inherits parent’s environment.
4/5/2019
COEN276 Winter 98 John Xiao

26. Start Unix Processes use “&” to execute a task in background
Example: $ sort infile > outfile &
use ctrl-Z to suspend the foreground task, and then use bg.
use fg to move a background task to foreground.
4/5/2019
COEN276 Winter 98 John Xiao

27. Examine Processes Use ps to see the processes that you are running.
PID TTY TIME CMD
221 pts/4 4:15 netscape
201 pts/4 0:05 bash
215 pts/4 1:15 emacs-19
4/5/2019
COEN276 Winter 98 John Xiao

28. Shell: Command Interpreter
Bourne Shell: the default shell (sh)
original unix shell
does not have interactive features of newer shells
widely used for writing shell scripts
standard on Unix System V
4/5/2019
COEN276 Winter 98 John Xiao

29. Shell (Continued)
C Shell (csh): available on BSD Unix and most other systems
with syntax similar to the C language
with many enhancement over the Bourne shell.
Korn Shell (ksh): AT&T’s answer to C Shell
combine features of Bourne and C shells
very efficient
Other shells: tcsh, bash
4/5/2019
COEN276 Winter 98 John Xiao

30. Day-to-Day Use Command Function Meaning cat Display a file conCATenate
cp Copies a file CoPy
mv Renames a file or moves it MoVe
rm Delete files ReMove
lpr Sends a file to a printer Line Printer
lp (Sys V)
ls Lists the contents of a directory LiSt
chmod Changes the “mode” of permissions Change MODe
pwd Shows the current directory Print WorkingDir
cd Change current directory Change Dir
mkdir Create a directory MaKe DIR
rmdir Delete a directory ReMove DIR
ps Shows the processes on the system Process Status
man Shows info. about commands Manual
df Shows file system status Disk File
du Shows the space allocation of files Disk Utilization
grep Search for patterns in files
who Shows who is on the system WHO
4/5/2019
COEN276 Winter 98 John Xiao

31. Standard Command Format
command [options] [arguments]
wc [-c | -m | -C] [-lw] [file …]
stuff in brackets is optional
boldface words are literals (must be typed as is)
italicized (or <> enclosed) words are args (replace appropriately)
Options modify how the command works
4/5/2019
COEN276 Winter 98 John Xiao

32. Command Format (Continued)
Option syntax
options ::= -option white-space options*
option ::= noargoption* argoption | noargoption+
noargoption ::= letter
argoption ::= letter whitespace string
Examples:
cc -go zap zap.c
cc -g -o zap zap.c
4/5/2019
COEN276 Winter 98 John Xiao

33. More Examples ls [-alRF…] file-list a l R F
for listing all files including the dot files l
for long format (file type, permissions, #links, owner, group, etc) R
for recursive, list subdirectories. F
for listing directories with a trailing /
4/5/2019
COEN276 Winter 98 John Xiao

34. More Examples ps [<options>]
List the information about running processes
Example:
ps -el # print the info about all processes (e) in the long format (l)
4/5/2019
COEN276 Winter 98 John Xiao

35. On-line Documentation
For shell command, system programs, and library functions.
man [n] <command>
e.g.
man wait man 1 wait
man man man 1 man
man -k <keywords>
4/5/2019
COEN276 Winter 98 John Xiao

36. Document Format Man(ual) page format Name Synopsis
Description (options, defaults, detail desc., examples)
See Also
Bugs
4/5/2019
COEN276 Winter 98 John Xiao

37. I/O Redirection and Pipe
“>“ redirects standard output (screen) to a file
E.g. ls > dirlist
“<“ redirects standard input (keyboard) from a file
E.g.
sort < infile > outfile
mail jxiao1 < outfile
Tee: copy stdin to stdout, making a copy in zero or more files (does not buffer output).
4/5/2019
COEN276 Winter 98 John Xiao

38. I/O Redirection and Pipe
“|” pipe output of program1 to input of program2
E.g. who | wc
Or getit < in | check_it | process_it | format_id > out
“>>“ appends output to a file
E.g. ls -l >> oldfile
Exercise: find out the definition of “tee”.
4/5/2019
COEN276 Winter 98 John Xiao

39. Sequential vs. Concurrent Process
%date
%ps -ef OR %date; ps -ef; who
%who
Concurrent:
%pgm1 & prgm2 > file1 & pgm3
%make > log &
%sort +1 pdir; ((pr dir | lpr) & sort +1 local))
pr, like cat, prints the content of file in a format suitable for line printers (66 lines per page)
4/5/2019
COEN276 Winter 98 John Xiao

40. File Name Expansion
Each shell program recognizes a set of special characters called meta characters.
The metacharacters are used to create patterns to match filenames and command names.
Bourne and Korn shell meta/wildcard characters
4/5/2019
COEN276 Winter 98 John Xiao

41. File Name Expansion * matches any string (including null)
? matches any one character
[a-dA-D] matches any one character in the range
[!AB123] matches a char not in this range
\ escape
~<username> the home dir of the user (not bourne shell).
4/5/2019
COEN276 Winter 98 John Xiao

42. File Name Expansion
Assume we have the following files under the current directory: 120, 235, 2#5, a.out, c22, c*2, Doc.1, Doc.2, Doc.3, one.c,two.c, three.c
ls *.c ls c*2
ls [a-z]*[!0-9] a.* *.*
ls ??? cd ~foo
ls *
4/5/2019
COEN276 Winter 98 John Xiao

43. Filters Most UNIX utilities are filters A filter is a program which
reads input (text) from standard input only
writes output to standard output only
writes error to standard error only
may use temporary files for intermediate results
4/5/2019
COEN276 Winter 98 John Xiao

44. Filters Filters can be combined to work together using pipes
Pipe: takes stdout of one command and uses it as stdin of another command
ls | wc
4/5/2019
COEN276 Winter 98 John Xiao

45. Command Alias Assign your own name for a command
Syntax is shell dependent
alias ll ‘ls -l’ C shell
alias ll=‘ls -l’ Korn, Bourne shell
Displaying the value of an alias
alias ll (displays value)
4/5/2019
COEN276 Winter 98 John Xiao

46. File Systems File: a collection of data Directory
Just a sequence of bytes
no record or block structure required
Directory
A file that contains information for files
distinction between a directory and a file
system can alter the contents of a directory
rooted tree file structure (inverted tree)
directories can contain both files and other directories info.
hierarchuy, starting with root directory /
kernel, system directories usually found off of root directory
4/5/2019
COEN276 Winter 98 John Xiao

47. Unix File System Road Map/
/dev
/etc
/var
/bin
/tmp
/usr
/mnt
/home
passwd
hosts
…...
sue
spool
adm
include
lib
etc
bin
5bin
john
printer
mail(in)
mail(out)
uucp
messages
wtmp …
Dev - physical device files
etc - system admin files (/etc/motd, /etc/passwd)
bin - executable binary files for standard commands and utilities (often a link to /usr/bin)
home- user directories
lib - library files used by programs and lang8uages
tmp - scratch area
usr - system files shares among users. Application programs, manguages, language files.
Special files: /dev/* represents I/O devices.
4/5/2019
COEN276 Winter 98 John Xiao

48. Unix Files Everything in Unix is a file Inside the kernel
mail messages, device files, tapes, keyboard, printers, terminals, …
/dev contains device files
Inside the kernel
references to a device file are converted into hardware commands to access that device
4/5/2019
COEN276 Winter 98 John Xiao

49. Pathnames
Every file and directory in the file system can be identified by a “full path name” (route from root to file)
/home/sue/ /f1 /
home
sue
fred
docs f2 f1
4/5/2019
COEN276 Winter 98 John Xiao

50. Pathnames (cont.) Relative path name
location relative to current directory
. Current directory
.. Parent directory
if cwd is /home/sue: ls
ls ./
cd ..
ls ../fred
4/5/2019
COEN276 Winter 98 John Xiao

51. Permissions
Every directory/file has a set of permissions associated with it
three types of permissions: read ( r), write (w), and execute (x)
three sets of permission: user, group, world.
ls -l
-rwxr-xr-x root Jan test
user group others #links owner size (time of last mod) (file name)
4/5/2019
COEN276 Winter 98 John Xiao

52. Permissions (cont.)
If a directory has “x” in its permision, the dir is searchable, ie., one can do “ls” on the directory.
4/5/2019
COEN276 Winter 98 John Xiao

53. Permissions (cont.)
In Unix system, users are identified by numbers:uid, gid
Problem: how do you write a program which allows different users to access a set of files? E.g. the program “passwd” Solution: use the set-uid bit
When a user execute a program with the set-uid bit set, the user assume the identity of the owner of the program.
4/5/2019
COEN276 Winter 98 John Xiao

54. Permissions (cont.) Example
ls -l /bin/passwd
-rwsr-xr-x 1 root Jan /bin/passwd
set-uid
Set-uid bit may break the security wall. (users can run the /bin/passwd and act like root)
Only special programs can be set-uid program, particularly if the owner is root.
4/5/2019
COEN276 Winter 98 John Xiao

55. Chmod Change the access permissions of a file
chmod <permissions> <filename>
permissions can be specified as 3 octal digits, <user,group,others>, the three bits of an octal means r,w,x
Example: chmod 755 test
permissions can be specified as +x, or u+x, or g+r, …
“chmod +s test” sets the set-uid bit for file test.
4/5/2019
COEN276 Winter 98 John Xiao

56. File Systems and the I-nodes
Each file system occupies a number of cylinder groups.
Each file system has a superblock, an i-node table and files
The superblock is at a fixed location relative to the beginning of the file system. It contains a description of the file system.
Unique inode allocated for each file
file consists of : name, contents, admin info (permission, modification)
a filename in a directory is a link:
links name in directory hierarchy to the inode and to the data
4/5/2019
COEN276 Winter 98 John Xiao

57. File Systems and the I-nodes (cont.)
One can find the location of the I-node table thru superblock.
Each entry of the I-node table is an I-node, which uniquely represents a file in the file system.
superblock
I-node table
file1
file2
free
file3
free
4/5/2019
COEN276 Winter 98 John Xiao

58. File Systems and the I-nodes (cont.)
A file has several components: a name, contents, and administrative info.
An I-node contains administrative info, such as uid, gid, time of creation, modification, and access, permissions, location of the file on the disk.
A file name in a directory is a link from the name in the directory to the inode.
4/5/2019
COEN276 Winter 98 John Xiao

59. Symbolic links vs Regular Links
Regular links are recorded in reference counts (using inodes)
Symbolic links are not recorded in reference counts
A symbolic link may point across file systems
A symbolic link may point to a directory
4/5/2019
COEN276 Winter 98 John Xiao

60. Mounting File Systems /
/dev
/etc
/bin
/tmp
/usr
/mnt
/home
A file system must be mounted before it can be used
Root file system is mounted at boot time.
A file system can be mounted to a dir. of another mounted fs.
Mounting is done in memory
a file system
Mount - graft a new branch in the file system
NFS - allow user on workstation to access remote file system transparently
(network file system)
4/5/2019
COEN276 Winter 98 John Xiao