1. ITEC400 Week Two Professor Robert D’Andrea

2. Agenda
Introduction Administrative Announcements Link of the Week Expected Outcomes This Week’s Topics Next Week’s Lab Assignment Break Out Problems Upcoming Deadlines Lab assistance, questions, and chat time

3. Unix Operating System
The link below lists most all operating systems that are available. The link below has tabs for books, posters, software, and tools Definition of Operating System (OS) is a multitasking, multi- user computer operating system that exists in many variants. The original Unix was developed at AT&T's Bell Labs research center by Ken Thompson, Dennis Ritchie, and others. Think of an operating system as an octopus. It’s four pairs of arms are connected to diverse software areas within the operating system that accesses different functionality. The central body representing the kernel.

4. Unix Operating System
Common operating systems discussed in class will be:
UNIX, Linux, HP-UX, and Windows
Basic tasks performed by an operating system
Control and allocate memory for processes
Prioritize system requests for the CPU
Control input and output transmissions to and from devices
Facilitate networking and support file system management

5. Unix Operating System Services Operating System (OS) perform
Process management
Memory management
OS coordinates various types of memory
File systems
Networking
Graphical user interface (GUI) and command line
Device drivers
Security
Internal management
External management

6. Unix Operating System
Services Operating System (OS) perform Process management – Background and foreground processes Memory management - The essential requirement of memory management is to provide ways to dynamically allocate portions of memory to programs at their request, and freeing it for reuse when no longer needed. This is critical to the computer system. Random Access Memory (RAM) – Static and dynamic RAM. Cache Memory - is a cache used by the central processing unit of a computer to reduce the average time to access memory Shared Memory - The essential requirement of memory management is to provide ways to dynamically allocate portions of memory to programs at their request, and freeing it for reuse when no longer needed. This is critical to the computer system.

7. Unix Operating System
File systems - Is an abstraction to store, retrieve and update a set of files. The term identifies data structures specified by some of those abstractions, which are designed to organize multiple files as a single stream of bytes. Network protocols are specified by other abstractions, which are designed to allow files on a system to be accessed remotely. Networking – Internal and external communication. Graphical user interface (GUI) and Command line (CLI). Device drivers - Is a computer program that operates or controls a particular type of device that is attached to a computer. A driver typically communicates with the device through the computer bus or communications subsystem to which the hardware connects.

8. Unix Operating System
Security Internal management and external management. Internal security pertains to employees inside the company and visitors that come to a company to sale products, repair equipment and delivery persons. External security encompasses users needing access to company data and/or assets. The list of external users accessing data for a company could be significant depending on the nature of the business. The list includes employees working remotely, branch offices, and company suppliers.

9. This weeks expected outcome
Upon successful completion of this module, the student will be able to:
Create scripts using shell variables and program control flow.
Use man page system
find and grep commands
Use redirection and pipes to combine scripts and executables.

10. This weeks expected outcome
Authors of UNIX
Create scripts using shell/Perl variables and program control flow.
Authors of AWK
Use redirection and pipes to combine scripts and executables.
Use man page system and find script tools.
Discuss Perl Language

11. VI Text Editor
Features of the text editor “vi” Vi was the first full screen text editor written for UNIX. It was initially designed to be simple and small in size. The designers intended vi to fit on a floppy-based emergency boot system. For this specific reason, you may need to use it in an emergency recovery situation. Most Linux distributions ship with a variant of vi known as Vim. Vim has been enhanced, supporting more features than the original vi editor. You will find that most distributions of Vim support launching it by typing vi, as if it were the original vi.

12. VI Text Editor
Vi is made up of three modes, executing in one of the three modes: The command mode accepts commands, which are usually single characters. For example, o and O and I and an all enter the insert mode, but in different ways. The O and o open a line above and below, respectively where the cursor is positioned. The Ex mode is entered for saving your file. This is initiated by typing a colon (:), followed by entering a w or q character to save or not to retain the information in the file. After executing in the ex-mode, vi automatically returns to the command mode. The Insert mode allows you to enter text that appears on the terminal. To exit the insert mode, depress the Esc key, which will return you to the command mode.

13. VI Text Editor
Basic vi commands Save text file - :wq! Delete a character – x Inserts a new line immediately below the current line - o Inserts a new line immediately above the current line O Undo any changes – u Change current word – cw Move around in file using the line number - <number>G Insert information on left side of a character – i Insert information on right side of a character – a Delete a line in a file – dd

14. UNIX Environment
Executing your environment The behavior of shell commands are determined by the execution of the environment provided by the shell. The UNIX shell maintains a complete set of environment variables that provide information about different areas of the operating system. For example, there are variables for the terminal type, current working directory, and default file and directory permissions. The environment variables are passed to all scripts and programs that are not built in to the shell, but may be accessed or modified, by the program. The convention for specifying environment variables is in upper case letters.

15. UNIX Environment
To view all environment variables, type printenv To view a particular environment variable. echo $EDITOR The above statement outputs the value of the $EDITOR environment variable echo $TERM The above statement outputs the value of the $TERM environment variable. The TERM variable tells the shell what kind of terminal you are using.

16. HP-UX Executing Environment
When you log into a system, the sequence of events can differ when creating your working environment. The particular sequence of events depends on the flavor of UNIX being implemented and the default shell for your account. The HP-UX operating system subscribes to the following sequence of events at log in: The getty process provides the login: prompt seen on your terminal. The getty process reads the provided username, and invokes the login program.

17. HP-UX Executing Environment
The login program receives the username from the getty process, and prompts the user for their password. The login program consults the /etc/shadow file to verify your password. The login program turns off terminal echo so that the password characters are not displayed on the terminal. After the password has been authenticated, the default shell is invoked by accessing the information in the last field of the /etc/passwd file for that username.

18. HP-UX Executing Environment
When the shell program starts, it reads the configuration files called login scripts to configure the execution environment. On HP-UX, the /etc/profile file contains initialization parameters for ksh and sh, while the /etc/csh.login file is used for csh. After the system login scripts have been read, the shell looks for user- specific login scripts. Once the system login scripts have been read, the shell reads user login scripts. The user login scripts are stored in the user’s home directory, where a user can customize their shell environment. The sh and ksh shells look in the .profile. Ksh reads a file defined in the environment variable ENV. Csh reads a file called .cshrc, and if it is the login shell, the file is named, .login.

19. System Environment Variables
USER – contains the username. When a file or directories are accessed, the access permissions are verified against the value found in the USER variable.
HOST – is an environment variable that contains the name of the host machine that is executing your shell program. When connecting to another remote machine through telnet or ftp, the name of the host is transferred to the remote machine, and displayed on the remote terminal so the administrator can know the names of the systems he or she is logged into.

20. System Environment Variables
Linux enables you to load kernel modules with two programs: insmod and modprobe. The insmod program inserts a single module into the kernel. This process requires you to have already loaded any modules on which the module you’re loading relies. The modprobe program , by contrast, automatically loads any depended-on modules and so is generally the preferred way to do the job.

21. Processes and Tasks
General approach to foreground and background processes When you log in to a UNIX system, the kernel starts a specific shell for you, and connects your shell to your terminal. When a command is entered and executed from the shell, the shell creates a child process to execute the command, and connects the child process to your terminal. By connecting the child process to your terminal, the shell allows you to communicate by sending input to the child process, and receiving output from it. When the child process finishes, all resources are relinquished (reaped) back to the system, your shell regains access to the terminal, redisplays the shell prompt, and waits for your next command.

22. Processes and Tasks
Processes that require users to be active participants (like processing Excel) must execute in the foreground in order to run. These jobs are termed "interactive," and must periodically update the display monitor, and accept input from the user, and allow access to the terminal interface. There are processes that do not require active participation once they are started. For example, a subroutine that initially accepts input data like a pattern and data file from the main script, uses the pattern to find a match in the data file, and writes matched information to an output file. Non-interactive jobs do not accept input commands, display output data on your terminal, and are disconnected from your terminal, and releasing the terminal for interactive use. This type of process is referred to as a “background” process.

23. Processes and Tasks
It is possible to log out of a system, and leave the background processes executing. The down side of this action is not being able to reconnect a background process to a terminal after logging out. If the background process identification information (PID) is known, the process can be terminated from a terminal. The number of background processes executing at the same time can be many, where a foreground process can only execute one process at a time. The reason is because you only have one terminal for viewing and one keyboard at your terminal.

24. Conceptual Appearance of UNIX

25. UNIX File Types
Command: ls -l drwxrwxrwx permissions (Directory) -rwxrwxrwx permissions (File) lrwxrwxrwx permissions (Symbolic link) -rwx dandrear faculty (Hard link) crw root root audio (Character device) brw-rw root disk aztcd (Block device)

26. UNIX Operating System
Pipe Command Users can connect the standard output of one command into the standard input of another command by using the pipeline operator (|). Demonstrate command usage: ps -ef ps –ef | wc –l ps –ef | awk ‘{print $2}’ ps –ef | grep dandrear ls –l | cut –c1-3 who –b (Time of last system boot) who –r (Print current run level) who –s (Print short version of user on system)

27. UNIX Operating System
File Descriptor A process associates a number with each file it has open. This number is called a file descriptor. When you log in, your first process has the following three open files connected to your terminal. Standard input: File descriptor 0 is open for reading. Standard output: File descriptor 1 is open for writing. Standard error: File descriptor 2 is open reading.

28. UNIX Operating System
Manual (man) Command UNIX-like command documentation is known as “man”. Each page is a self-contained document. The Manual sections are split into eight numbered sections: 1 General commands. 2 System calls 3 C library functions 4 Special files (usually devices, those found in /dev) and drivers. 5 File formats and conventions 6 Games and screensavers 7 Miscellaneous 8 System administration commands and daemons

29. UNIX Operating System
grep Command The “grep” command searches the named input file(s) for lines in a file containing a given pattern. When a pattern is found, each line is reported to standard output. Demonstrate: grep [options] pattern file(s) grep inc_A ~dandrear/xyz grep inc_A prog1 prog2 grep inc_A *

30. UNIX Operating System
find Command The “find” command lists all pathnames that are in each of the given directories. Demonstrate: find ~dandrear –type d –print find ~dandrear –type f -print find . –print find / -type f –print

31. UNIX Operating System
Redirections Many UNIX commands take text-like input and produce text-like output. It's sometimes useful to be able to control where the input comes from and output goes (via redirection), or even pass the output from one command to another's input (via pipes).

32. UNIX Operating System
Redirection (cont’d) Redirect the standard output of a command to a file. date > /tmp/date_saved Redirect the standard input of a command so that it reads from a file instead of from your terminal. cat < ~dandrear/Winter_2018_Solutions/foobar Append the standard output of a command to a file. cat foobar_2 >> foobar_1

33. UNIX Operating System
Authors of UNIX Dennis Ritchie Computer Scientist Dennis Ritchie was an American computer scientist. He created the C programming language and, with long-time colleague Ken Thompson, the Unix operating system. Born: September 9, 1941, Bronxville, NY Died: October 12, 2011, Berkeley Heights, NJ Education: Harvard University Fields: Computer Science Books: The C Programming Language, More Awards: Turing Award, National Medal of Technology and Innovation, IEEE Emanuel R. Piore Award

34. UNIX Operating System Authors of UNIX Ken Thompson Computer Designer
Kenneth Lane "Ken" Thompson, commonly referred to as ken in hacker circles, is an American pioneer of computer science. Having worked at Bell Labs for most of his career, Thompson designed and implemented the original Unix operating system. 
Born: February 4, 1943 (age 72), New Orleans, LA
Fields: Computer Science
Awards: Turing Award, National Medal of Technology and Innovation,Tsutomu Kanai Award, IEEE Emanuel R. Piore Award
Education: University of California, Berkeley (1966), University of California, Berkeley (1965)

35. UNIX Operating System
What is the next user interface going to be? The textual (command line) and the visual (graphical user) interfaces are the two most common modalities used to support engineers in network and system administration positions. The command line interface is recognized as the first generation and the graphical user interface is considered the second generation. Currently, research is trying to determine the next best interface. The command line interface is known as, “under the hood” method of interacting with the operating system.

36. UNIX Operating System
CLI Benefits Manipulate textual data Quick customization of data allows engineers the ability to change data to another form Excellent for filtering data on systems Commands are rich, expressive, flexible, and powerful GUI Benefits Reduces data overload Simple filtering and manipulation of the data Excellent for displaying trends in data

37. UNIX Operating System
Changing shells It is possible to invoke a different shell, while within another shell, you just type the name of the shell you want to execute after the terminal prompt, ksh, csh, or sh. Command renaming Both ksh and csh provide command name aliasing, which allows you to change a command name. Command aliasing can save you time and keystrokes by renaming frequently issued long commands. Two pieces of information are needed. The command you want to alias, and the alias you want to use to refer to it. alias g=‘grep’

38. UNIX Operating System
File name completion If you dislike lengthy typing, csh and ksh provide a feature the allows you to type partial file names. By typing a partial file name and pressing the ESCAPE key once for csh and twice for ksh, you can retrieve the file. If the beginning characters for a file are entered and the file does not exist, the shell will beep at you. Command history substitution Both ksh and csh maintain an ordered list of commands that you issued, allowing them to be retrieved from the list. Typing the history command will displayed the list of previous commands issued.

39. UNIX Operating System
set ksh history command size HISTSIZE=60; export HISTSIZE set csh history command size set history=70

40. UNIX Operating System
UNIX File System A disk drive is a device that stores data by making electrical imprints on a magnetic surface, optical, or mechanical changes to a surface layer of one or more rotating disks. A hard disk consists of one or more circular aluminum platters of which either or both surfaces are coated with a magnetic substance used for recording the data. For each surface, there is a read-write head that examines or alters the recorded data. The platters rotate on a common axis; typical rotation speed is 5400 or 7200 rotations per minute, although high-performance hard disks have higher speeds and older disks may have lower speeds. The heads move along the radius of the platters; this movement combined with the rotation of the platters allows the head to access all parts of the surfaces.

41. UNIX Operating System

42. UNIX Operating System

43. UNIX Operating System
The Power of Memory Keeping data in memory is the current direction technology is headed instead of pulling data in from a disk during processing. Database companies are interested in getting their share of the in-memory database market. Industries actively involved are Aerospike, Microsoft Research, IBM, Oracle, VoltDB, and MemSQL.

44. UNIX Operating System
The Power of Memory
In-memory databases is all so know as main memory databases. The in- memory databases speed up processing in two basic areas.
Data available in memory
The time lag caused by fetching the data from a disk

45. UNIX Operating System
The Power of Memory A more basic delay retrieving data from disk is the way data is stored on disk in blocks. To get to a particular piece of data, the computer must import the whole block from disk, decode it, run the process on the piece it wants, re-encode the blocks, and send the block back where it came from.

46. UNIX Operating System
The Power of Memory When you store data in memory, you eliminate the overhead associated with block data. When data resides in- memory, you can be much more efficient. One database vendor sees in-memory databases as the wave of the future for online transactions. They envision the in- memory database handling a third of the database world.

47. UNIX Operating System
How a UNIX file system functions
Every item in a UNIX file system can be identified to belong to one of the four types:
Ordinary files contain text, data, or program information. They cannot contain another file or directory. They can be viewed as a one- dimensional array.
Directory contain files, and other directories. A directory is a file that has one line for each item contained within the directory. Each line in a directory file contains the name of the item, and a numeric reference to the location of the item. The numerical reference is called, i-node, which will be discussed at a later time.

48. UNIX Operating System How a UNIX file system functions
Every item in a UNIX file system can be identified to belong to one of the four types:
Link is a pointer to another file. As a review, a directory is a list of the names and i-numbers of files. A directory entry can be a Hard Link or a Symbolic link. Both of these topics will be addressed at a later time. Special files are files representing input/output (I/O) devices. A special file can be a printer, a disk drive, or a terminal. Because UNIX treats all such devices as a file, it achieves a greater degree of compatibility between I/O devices and I/O of ordinary files, providing more efficient use of software.

49. UNIX Operating System
Types of File and Directory Access Access File Meaning Directory Meaning r View file contents Search directory contents w Alter file contents Alter directory contents x Run executable file Make your current directory -rwx Owner (columns 2-4) 700 ( ) ----rwx--- Group (columns 5-7) 070 ( ) rwx Other (columns 8-10) 007 ( )

50. UNIX Operating System
File Descriptors A file descriptor is generally an index for an entry in a kernel-resident data structure that contains information on all open files. Each process on the system has its own file descriptor table. A user application passes the abstract key to the kernel through a system call, and the kernel accesses the file for the application.

51. UNIX Operating System

52. UNIX Operating System

53. UNIX Operating System
wc `ls` commands The word count (wc) command reads either standard input or a list of files and generates one or more of the following statistics: newline count, word count, and byte count. A snippet of output from the wc –l `ls` command: 19 xyz.sh 2 zombie 24 zombie.c 9890 total

54. UNIX Operating System
Shell and Programs Access To run a shell script, you will need read (r) and execute (x) access (r-x). The read access mode is a binary 4. The execute access mode is a binary 1. To run a binary executable program, you will need execute (x) access (--x). The execute access mode is a binary 1.

55. UNIX Operating System
Character Action > Redirect standard output >& Redirect standard output and standard error >>& Append standard output and standard error < Redirect standard input

56. UNIX Operating System
Shell redirect ls –a > /tmp/output 2>&1 (stderr redirected to stdout) > is equivalent to 1> (stdout (write) redirect symbol)) < is equivalent to <0 (stdin (read) redirect symbol)

57. UNIX Operating System
Pipe command (command1 | command2) Command2 does not start executing until command1 has finished, and a sufficiently large scratch file is required to hold the intermediate results as well as whatever work space each task required. command1 > tempfile command2 < tempfile rm tempfile

58. UNIX Operating System
Pipe command

59. UNIX Operating System
What is a data structure? A data structure is a specific way of storing and organizing data in a computer so that it can be accessed with high efficiently. Data structures can be used as a single place for storing unrelated information. A data structure is a particular way of storing and organizing data in a computer so that it can be used efficiently. Different kinds of data structures are suited to different kinds of applications, and some are highly specialized to specific tasks. Some common data structures: array, hash table, linked list, queue, and stack.

60. UNIX Operating System
What is a shared memory? Shared memory is memory that may be simultaneously accessed by multiple programs with an intent to provide communication among them or avoid redundant copies. Shared memory is an efficient means of passing data between programs. Depending on context, programs may run on a single processor or on multiple separate processors.

61. UNIX Operating System

62. UNIX Operating System
What shared memory looks like?

63. UNIX Operating System
What shared memory looks like? A shared memory arena file is a regular or shared memory device file /dev/shm is mapped and shared by one or more processes.

64. UNIX Operating System
What is a regular expression? A regular expression (abbreviated regex or regexp) is a sequence of characters that forms a search pattern, mainly for use in pattern matching with strings, or string matching, i.e. "find and replace"-like operations. The concept arose in the 1950s, when the American mathematician Stephen Kleene formalized the description of a regular language, and came into common use with the Unix text processing utilities ed, an editor, and grep (global regular expression print), a filter.

65. UNIX Operating System
Introduction to Perl Perl - Practical Extraction and Report Language Originally developed by Larry Wall, a linguist. Perl is 21 years old and Perl 5 is 14 years old. Perl is a simple language - Compiles and executes like a shell script or a batch file - Perl doesn’t impose special growth limitations on arrays and data strings - Perl is a composite of C, AWK, and Basic - Originally developed to process text and automating tasks

66. UNIX Operating System System administration Web development
Perl’s range of flexibility
System administration
Web development
Network programming
GUI development
Major features
Procedural Programming
Sequence or unstructured statements
Includes routines, subroutines, methods, or functions
Object Oriented Programming
Module uses “objects” and their interactions to design applications and computer programs.

67. UNIX Operating System
Major features (continued) Powerful built-in support for text processing Large collection of third-party modules. Variable A variable is a storage location that can hold any of various kinds of value, as a program sees fit. This storage location is an abstract area known as namespaces.

68. UNIX Operating System
Perl language Perl is a free-form language, but is not free of form. Perl is a free-form language in which you can put spaces, tabs, and new lines anywhere you like, except where they aren’t allowed. Whitespace cannot be placed within a token. A token is a sequence of characters with a unit of meaning, like a word in the English language. But unlike the typical word, a token can contain other letters and characters as long as they are kept together.

69. UNIX Operating System Scope of a Perl variable
The scope of a variable means how far away you can see a variable, looking through one. Perl has two visible mechanisms.
Dynamic scoping of local variables, meaning that the rest of the block, and any subroutines that are called by the rest of the block, can see the variables that are local to the block.
Lexical scoping of “my” variables, meaning that the rest of the block can see the variable, but other subroutines called by the block cannot see the variable.

70. UNIX Operating System
Variable and constant A variable is a named object that resides in RAM memory and is capable of being examined and modified. A variable is used to hold information critical to the operation in an embedded system. A constant is a named object that resides in memory (usually in ROM) and is only capable of being examined.

71. UNIX Operating System
AWK and Perl The AWK utility is an interpreted programming language typically used as a data extraction and reporting tool. It is a standard feature of most Unix-like operating systems. AWK was created at Bell Labs in the 1970s, and its name is derived from the family names of authors – Alfred Aho, Peter Weinberger, and Brian Kernighan. The power, terseness, and limits of early AWK programs inspired Larry Wall to write Perl just as a new, more powerful POSIX AWK and gawk (GNU AWK) were being defined.

72. UNIX Operating System
AWK Alfred Aho Computer Scientist Alfred Vaino Aho is a Canadian computer scientist best known for his work on programming languages, compilers, and related algorithms, and his textbooks on the art and science of computer programming. Born: August 9, 1941 (age 74), Timmins, Canada Doctoral advisor: John Hopcroft Residence: United States of America Education: University of Toronto, Princeton University Awards: IEEE John von Neumann Medal

73. UNIX Operating System
AWK Brian Kernighan Computer Scientist Brian Wilson Kernighan is a Canadian computer scientist who worked at Bell Labs alongside Unix creators Ken Thompson and Dennis Ritchie and contributed to the development of Unix. He is also coauthor of the AWK and AMPL programming languages. Born: January 1, 1942 (age 73), Toronto, Canada Fields: Computer Science Education: Princeton University, University of Toronto

74. UNIX Operating System
AWK Peter J. Weinberger Computer Scientist Peter Jay Weinberger is a computer scientist best known for his early work at Bell Labs. He now works at Google. Weinberger was an undergraduate at Swarthmore College, graduating in Born: August 6, 1942 (age 73), New York City, NY Known for: AWK Doctoral advisor: Derrick Henry Lehmer Education: Swarthmore College, University of California, Berkeley Fields: Number theory, Computer Science

75. UNIX Operating System Why is awk language so important?
Awk language is an excellent filter and report writer. Many UNIX utilities generate rows and columns of information. Awk is an excellent tool for processing rows and columns, and it is easier to use awk than other conventional programming languages. Perl recognized the importance of awk, so it was included and enhanced in Perl.
Example: awk '{print $1}'

76. UNIX Operating System
Points of interest who | sort > /tmp/test_file.txt The output of the who command is piped to the sort function and written in ascending order in the /tmp/test_file.txt file. The “who” and “sort” commands both write and read an intermediate file. Test one liners: who who | sort who | sort | /tmp/test_file.txt

77. UNIX Operating System
Moving Around in UNIX ls –a ps –ef whoami rmdir touch less w tail head sort man

78. Break-out Problems more command Program statement (#!/bin/ksh)
less command
ps –ef | wc –l
who | awk ‘{print $1}’ | sort –u | wc –l
ps –ef | awk ‘{print $1}’ |sort –u | wc –l
find / ex
ps –ef | awk ‘{print $9, $1}’
Shell language syntax: $#
Shell language command: for name in *
Shell language command:
NUMBER=$(($NUMBER - 1))
Shell variable: PATH

79. Break-out Problems ps | wc –l who | awk ‘{print $1}‘ | sort –u | wc –l
ps –ef | awk ‘{print $1}’ | sort –u | wc –l
sort –r names.txt
ps –ef | awk ‘{print $8, $1}’
find /bin -name gzip
find /etc -name motd
> newfile
rm file_name
date | cut –c 12-19
cp test_data.txt ~dandrea/temp
mv test_data.txt ~dandrear/temp
printf $NUMBER
touch newfile

80. Hands-On-Information
Read Chapters 4, 5, and 6 in Essential System Administration text.
Read Module 3 and 4 listed under the course Web site.

82. After Class Assistance
Questions? Comments? Concerns? After each Franklin Live session, I will remain on the session to provide assistance unless otherwise indicated.

83. Lab assistance, questions, and chat time