sed

Class Issues vSphere Issues – root only until lab 3

sed sed = stream editor Allows you make changes to text files on the fly on a line by line basis Basic syntax is: sed ‘s/pattern/replacement/’ filename Works on files, default output is to the screen

Basic Examples Replace: –sed ‘s/1/ONE/g’ file.txt (would replace the first 1 with ONE in file.txt) Swap: – sed 'y/abc/ABC' file.txt (replace a with A, b with B, c with C) Show line numbers then lines: –sed ‘=‘ file.txt Lowercase: sed 's/[A-Z]/\l&/‘ file.txt

Special Replacements \L - turn the replacement to lowercase until a \U or \E is found, \l - turn the next character to lowercase \U - turn the replacement to uppercase until a \L or \E is found \u - turn the next character to uppercase, \E - stop case conversion started by \L or \U.

Remember the g? First Character Lower Case: –sed 's/[A-Z]/\l&/' addresses.txt All Characters Lower Case: –sed 's/[A-Z]/\l&/g' addresses.txt 5th Character Lower Case: –sed 's/[A-Z]/\l&/5' addresses.txt

Useful sed command line -f means read sed commands from file –sed –f mysed.txt

Gotchas Don’t do: –sed 's/[A-Z]/\l&/[3-5]' addresses.txt > addresses.txt Create a new file and mv it back.

awk

Programming language for data streams; usually files or subsets of files Developed at Bell Labs in the ’70’s by: Alfred Aho, Peter Weinberger, and Brian Kernighan Like sed but does the following as well: –specify particular fields of a matching line to manipulate and/or output –use variables to store values temporarily –perform calculations on those variables.

How to run? Usually: awk ‘ ’ files to parse Example (a cat replacement): awk ‘{print $0}’ *

Fields in awk By default lines are delimited by white space: space characters or tabs $0 – whole line $1 – is first field in line $2 – is 2 nd field in line NF – number of fields on line No error reported if field doesn’t exist, simply outputs a blank line

Basic Pattern Matching /pattern/ { do something } Example: /A/ { print “This line has an A”} No pattern means do the work on every line

Multiple patterns Just list them: awk ‘ /A/ { print “this line has an A”} /B/ { print “this line has a B”} { print “this gets printed always” } ‘ my_file Don’t forget the closing ‘

Forming Patterns Can use most of the extended regular expressions, except {} Can do the relational expressions from other languages:, ==, !=, =

Comparisons Numeric example: ls -l | awk '$5 >= 512 {print $0}' String example: ls -l | awk '$6 == “Sep” {print $0}'

Variables Variables are allowed: ls -l | awk '$5 >= 512 {size += $5}‘ Variables don’t have to be initialized However, we won’t ever see size output!

BEGIN/END BEGIN – run before any lines are read END – run after every line has been read You can have one, both, neither Example: ls -l | awk '$5 >= 512 {size += $5} END { print size}‘

Begin Example awk ‘BEGIN { printf(“I’m pretending to be cat\n”) } { print $0}’ my_file printf is C/Java like

Useful BEGIN command Remember BEGIN goes before reading of lines begins Want the first line of your file as a header? Use getline, but no pattern matching or processing of the lines read awk ‘BEGIN { getline; print $0}’ my_file

Useful awk variables FILENAME – current file being parsed NF – number of fields on the current line FNR – current record in the file (usually means current line) NR – current record in the stream There are many, many others as well, read the man page

Standalone awk scripts Add the following to top of file of awk commands: #!/usr/bin/awk –f Don’t forget to chmod +x Example: #!/usr/bin/awk -f /^.\// { dots += 1 } /^$/ { blank += 1 } /^l/ { sym += 1 } /^d/ { dir += 1 } /^-/ { file += 1 } { total += 1 } END { printf("Dots=%d\nBlank Lines=%d\nSym Links=%d\nDirectories=%d\nFiles=%d\nTotal=%d\n",do ts, blank, sym, dir, file, total)}

Intro to Shell Scripting

Shell Script 101 List of commands interpreted by a shell cd /var/tmp mkdir boks cd boks gzip -dc /var/tmp/boks_install.tar.gz | tar xf -./install.jvss0001 cd /var/tmp rm -r boks* cd /etc/rc2.d mv S99boksm _S99boksm

Why need? Add users List bad perms Parsing log files A menu system for users Startup of system (.bashrc is a script) etc

Tests (if then else) if [ $count –ne 0 ] then avg=$((sum/count)) else avg=0 fi

Looping (for) for var in do echo $var done

Variables in Scripts I=0 J=$I E=$(($C+$D)) X=$(($X+1)) Notice the lack of spaces!

Input/Output/Assignment Input –read Output –echo – printf

Comparisions Numeric –-eq, -ne, -lt, -le, -gt, -ge Strings ==, != And is && Or is ||

Comparisions II File testing: –-d – is the item a directory? –-e – does the file already exist? –-f – does the file exist and is it a regular file? –-h (or –L) – is the item a symbolic link? (current user) –-r – does the file exist and is it readable? –-w – does the file exist and is it writable? –-x – does the file exist and is it executable?

Sample Comparisons [ -f my_file ] (don’t forget spaces) [ $x –ge 0 ] [ $1 = start ] [ “$1” = “” ] – empty string test! [ “$1x” = “x” ] – better empty string test!

Command Line Parameters Accessed via $# parameters –$0 is script name –$1, $2 … $9 = parameters on the command line or $* are all parameters –$# is total number passed Assignments say receive as parameter, assume it’s this one!

Accessing Parameters #!/bin/bash echo The command name is: $0. echo The number of command line arguments passed as parameters are $#. echo Yet another way to show total number: $*. echo The value of the command line arguments are: $1 $2 $3 $4 $5 $6 $7 $8 $9. echo Another way to display values of all of the arguments: exit 0

Debugging? Set -vx shows info -v = verbose -x = lines after run