1. Intro to Linux (fun with UNIX)

2. Synopsis Why Linux? Your computing environment
The command line interpreter (CLI/terminal/shell)
Linux directory structure
Working with files
Pipes and redirections
Where to find help?
SSH + FTP
If we have time…
Basics of (shell) scripting
Working with multiple processors
User groups and permissions

3. Why use Linux? Learning the basics of Linux allows you to:
The best tools available in the field of bioinformatics are largely non-portable, open source tools that are written for POSIX systems (such as Linux and UNIX).
Learning the basics of Linux allows you to:
develop a skill set that sets you apart from a wet-only biologist
run bioinformatics resources on your own machine
access ready-made bioinformatics computing environments
do reproducible research
perform analyses on computer clusters – important for long computational jobs
access cloud computing and High Performance Computing (HPC) resources
Linux is a multi-access, multi-tasking system: multiple users may be logged in and run multiple tasks on one machine at the same time, sharing resources (CPUs, memory, disk space) -> this is what we will be doing during the course

4. Your computing environment
In this class you will be using a thin client.
A thin client refers to either a software program or to an actual computer that relies heavily on another computer to do most of its work.
Think of it as a dumb computer that allows you to access our server (much bigger computer) stored in the data center.
You get access to computational resources that would not be available in a “normal” computer, such as the one you use at work or at home.
The server we will be using is a HP DL 580 G7
4 cores x 8 threads x hyperthreading = 64 threads
1 TB RAM
access to 100 TB storage

5. Logging in to our server
It’s easy! You will start with a fairly familiar-looking desktop. Double click on highlighted icon.

6. Enter your TU Graz credentials – the credentials you use to login to
online.tugraz.at

7. After you’ve entered your credentials, click “Connect”.

9. Important!

10. On this screen you can select how your desktop will look like.
All further instructions will assume that you selected the default XFCE Desktop.
Click “Ok” to continue.

12. Welcome to your Linux desktop

13. Logging out of your session
At the end of each class please log out of your session.
I will later cover the “screen” command which will allow you to leave your programs running even if your session has ended.
To log out simply click this icon at the top left of your screen.
A dialog like the one on the right will appear, make sure
you have selected “Save this session for future logins”
and then click the “Log Out” button.
You will be returned to the initial desktop of the thin client.
Try it now!

14. Before we begin

15. The command line interface (CLI)
The CLI is a text-based interface that allows computer users to type commands that get executed by the operating system. 
You call programs, modify files and control your system just by writing a bit of text.
It is an extremely powerful tool, because it centralizes managing your system and, crucially, makes automation easier, saving you a lot of work (more on this later, if we have time).
We will be using the CLI to do most of the lab tasks in this class.
There are different types of Unix shells or CLIs available, the one we are using is called Bash. Other good examples are tcsh, zsh or ksh.

16. How to open a shell window

18. Linux directory structure
You won’t find any Windows, Program Files, or Users folders if you start browsing around the file system on your Linux computer. 
On Windows, an application might store all its files in C:\Program Files\Application. On Linux, its files would be split between multiple locations – its binaries in /usr/bin, its libraries in /usr/lib, and its configuration files in /etc/.
Notice the use of slashes: in Windows you use “\” C:\Users, while in Linux you will use “/” /export/home.
3 most important folders we will be using in this course
/export/home/username – your user home directory, similar to the Windows C:\Users\User
/export/programs – contains all the bioinformatics software used in the course

19. We will browse to /export/home/sensencw – the path to my home directory
Windows directories in C:\
Linux directories in / (root)

20. Linux directory structure at the root - /
GUI
CLI
Linux directory structure at the root - /
Graphical User Interface (GUI) versus command line view

21. GUI
CLI
/export

22. GUI
CLI
/export/home

23. GUI
CLI
/export/home/amgrecu

24. It’s a tree!

25. Paths A Windows path: C:\Program Files\ Linux path types:
Full: /export/home/amgrecu/home.png
Relative path to /export/home: amgrecu/home.png
Relative path to /export/home/amgrecu: home.png
Your home directory: /export/home/amgrecu
OR ~

26. Working with directories
After you log in to the terminal window, you are in your
home directory, i.e. /export/home/amgrecu.
To see in what directory you are, type pwd (print working
directory)

27. Working with directories
cd - Change (current) directory
Without additional arguments, this command will take
you to your home directory
cd ./ - Change (current) directory to the same directory.
cd /export/home/amgrecu/Documents - Change
directory from wherever to /export/home/amgrecu/Documents.
cd Desktop - Change (current) directory to Desktop
(will work if the current directory contains “Desktop”)
cd ../ - Change (current) directory one level back
(closer to the root)
cd ../../../ - Change (current) directory three levels
back (closer to the root)

28. Working with directories
Creating directories
mkdir /export/home/amgrecu/new_dir - Make a new directory called “new_dir” in /export/home/amgrecu
mkdir new_dir - Make a new directory called “new_dir” in the current directory
Removing directories
rmdir /export/home/amgrecu/new_dir - Remove directory called “new_dir” in /export/home/amgrecu – will fail if the directory is not empty
rm –rf /export/home/amgrecu/new_dir - Remove directory called “new_dir” in /export/home/amgrecu with all its content (i.e. all files and subdirectories will be gone)
rm –rf new_dir - Remove directory called “new_dir” in current directory with all its content (i.e. all files and subdirectories will be gone)
rm –rf is a very dangerous command! Once you have deleted something it is GONE!
There is no “Recycle Bin”! There are no backups!
Check carefully what you are deleting!

29. Listing directory content
Use ls –alh in the current directory. For even more options, man ls.
file last
accessed
name
blue - directory
permissions
Owner user and group
size

30. May not work with all remote clients!
Useful things to know
May not work with all remote clients!
Use Up/Down arrow keys – this will cycle through recently executed commands.
Use the TAB key – this will often present you with a list of choices after typing a part of a command (more on this in a moment)
history command: list all recently used commands
history - list all remembered commands
history | more - list all remembered commands page by page
history | grep –i example (list all remembered commands containing string “example”, case-insensitive)

31. Tab Completion
Writing every single character of a command is tedious. But your shell can help you!
Press the Tab key while writing commands, directories or file names.
Press once to complete until the last unambiguous character (you will hear a beep if there are no suggestions or you have ambiguous options).
Press twice to show all completion options.

32. Examples: Commands: Directories: Files: Tab Completion
cuffl<tab> -> cufflinks
Directories:
ls /ex<tab> -> ls /export
Files:
cp /export/data/mol923/c<tab> -> cp /export/data/mol923/completion-example.txt

33. More examples: Tab Completion
bow<tab> -> bowtie and you hear a beep
bowtie<tab><tab> :

34. Real World Example: is simply Tab Completion
cd /export/data/mol923/Bowtie-example
is simply
cd /ex<tab>da<tab>m<tab>B<tab>

35. Working with files – Types of files
There are many types of files. Here are the most important:
Text files (human-readable; can be viewed and modified using a text editor)
Text documents (e.g., README files)
Data in text format (e.g., FASTA, FASTQ, …)
Scripts:
Shell scripts (*.sh)
Perl scripts (*.pl)
Python scripts (*.py) and many others
Binary files (not human-readable; cannot be viewed using a text editor)
Executables (e.g., samtools, blastn, bowtie)
Data in binary format (e.g, BAM files, index files for Bowtie, formatted BLAST databases)
Compressed files - archives (*.gz, *.zip, *.bz2) – often text files re-formatted to save space on disk or packaged directory trees
How do you know what type of file you are looking at?

36. The “file” command
UNIX_examples]$ file example.txt example.txt: UTF-8 Unicode English text, with very long lines UNIX_examples]$ file read read: ASCII text UNIX_examples]$ file first-script.sh first-script.sh: Bourne-Again shell script text executable UNIX_examples]$ file blastn-example.fasta blastn-example.fasta: ASCII text velvet]$ file velvetg velvetg: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux , not stripped

37. Naming – Best Practices
Names should be meaningful, e.g. sample19_blastn_out.fasta, NOT blastn.fasta
Names are case-sensitive - MyFile, myfile, myFile are all different
Use only letters (upper- and lower-case), numbers from 0 to 9, a dot (.), and an underscore (_) – e.g this_is_an_example.ext, NOT this &ismy example^.ext
Avoid other characters! They may have special meaning in Linux
While working in command line terminal you always explicitly specify a program which is supposed to open this file.

38. Where to find help?
Every UNIX and Linux machine comes with built-in help for most of its commands and programs.
Type “man” followed by the command name
To leave: q
For Scrolling:
Arrow Up or Down for one line up or down
Page Down or Ctrl-F for one page down
Page Up or Ctrl-B for one page up
To search
A slash, then your search term, then enter. e.g. /list
n to go to the next occurrence of the search term
Even “man” itself has a manpage. Type man man into the terminal.

39. System commands on files – Copy (cp)
Usage: cp <source_file> <destination_file>
cp sample_data.fa /home/amgrecu/sample.fa - copy file sample_data.fa from the current directory to /home/amgrecu and give the copy a name sample.fa; destination directory must exist
cp /home/amgrecu/my_script.sh . - copy file myscript.sh from /home/amgrecu to the current directory under the same file name
cp /home/amgrecu/*.fastq /workdir/amgrecu - copy all files with file names ending with “.fastq” from /home/amgrecu to /workdir/amgrecu; destination directory must exist
cp –r /workdir/amgrecu/test /home/amgrecu - if test is a directory, it will be copied with all its files and subdirectories to directory /home/amgrecu/test; if /home/amgrecu/test did not exist, it will be created
man cp for all options to the cp command

40. Moving, renaming and removing files
Moving and renaming files - has the same command line structure as cp
Usage: mv <source_file> <destination_file>
man mv for all options to the mv command
Removing (deleting) files
Usage: rm <file>
man rm for all options to the rm command

41. The chmod command is used to change file permissions. 4= read 2= write
1= execute
Myself, Group, World
chmod 700 filename

42. Working with files - compression
To save disk space, we can compress large files if we do not intend to use them for a while. A lot of files downloaded from the web are compressed and usually need to be uncompressed before processing can take place. Compression works best on text files, such as FASTA or FASTQ.
Common compressed formats:
gzip (gz)
gzip my_file - compresses file my_file, producing my_file.gz
gzip –d my_file.gz - decompress my_file.gz, producing my_file
bzip2
bzip2 my_file - compresses to my_file.bz2
bzip2 my_file.bz2 – decompress to my_file

43. Working with files - compression
zip
zip my_file.zip my_file1 my_file2 my_file3 - create a compressed archive called my_files.zip, containing three files: my_file1, my_file2, my_file3
unzip my_files.zip - decompress the archive into the constituent files and directories
tar
tar -cvf my_file.tar my_file1 my_file2 my_dir - create a compressed archive called my_files.tar, containing files my_file1, my_file2 and the directory my_dir with all its content
tar -xvf my_files.tar - decompress the archive into the constituent files and directories
Help for compression tools:
gzip -h
bzip2 --help
zip --help
tar --help

44. gedit Kate nano Editing text files User-friendly options:
More advanced suggestions:
nano
You can call them from CLI:
nano filename
gedit filename
kate filename
In the course we will generally be using gedit

45. Working with text files
more example.txt - display the content of the file README.txt in the current directory dividing the file into pages; press SPACE bar to go to the next page
head -5 big.fasta - display first 100 lines of the file big.fasta in the current directory
tail -5 big.fasta - display last 100 lines of the file big.fasta in the current directory
tail -10 big.fasta | more - extract the last 1000 lines of the file big.fasta and display them page by page
head big.fasta | tail display lines 901 through 1000 of the file big.fasta
cat big.fasta - print the file on screen. Only use for small files!
wc big.fasta - display the number of lines, words, and characters in a file
Note the character | - it pipes the output from one command as input to another.

46. Working with text files
Looking for a string in a text file:
grep “Error: lane” error.log - display all lines of the file error.log in the current directory which contain the string “Error: lane”
Looking for a string in a group of text files:
grep “Error: lane” *.out - display all files *.out in the current directory which contain the string “Error: lane”; also display the lines containing that string

47. Pipes and Redirections
Every command line program has three streams:
STDIN: Standard input
STDOUT: Standard output
STDERR: Standard error
The output you see in the console is either from STDOUT or STDERR. If the command is interactive, it waits for input on STDIN.
| A pipe character redirects the STDOUT from the left program into the STDIN of the right program
> Redirects the STDOUT from the left program into the file on the right
2> Redirects the STDERR from the left program into the file on the right
< Program on the left receives STDIN input from the file on the right

48. Pipes and Redirections
Show files with a specific extension in a directory ls /export/data/mol923/Bowtie_example/reads | grep fa
Capture output of program to a file blastn -db nt -query blastn-example.fasta > hits.txt
Redirect error channel into a log file cuffmerge -g genes.gtf -s genome.fa assemblies.txt 2> cuffmerge_errors.txt

49. Where to find help No.2?
Not every command has a man page.
However most commands have built-in help output:
cp --help
blastn -help
cufflinks -h
It’s usually one of “-h”, “--help“, “-help” or similar
You can pipe the output into a pager (use q to leave; up/down arrow Ctrl-F/Ctrl-B to navigate)
cp --help | less

50. Where to find help No.3? Also, there is plenty of help on the Internet
Most problems can be solved by Googling. Basically, just type all related words into the search field.
Ask specific questions here. General questions are usually already answered. You can find them through the search field, but usually they’re the top hits on Google anyway.
“Stackoverflow for Biologists”
Forum about Bioinformatics tools and –omics data processing tools

51. UNIX Tutorials

52. https://filezilla-project.org/ Connect to „cbt01.cbt.tugraz.at“
Getting files in and out of my home computer
Filezilla
Connect to „cbt01.cbt.tugraz.at“
Use Port 22

54. Connect to „cbt01.cbt.tugraz.at“
Connecting from Home to the UNIX machines at TUGraz
ThinLinc Client
Connect to „cbt01.cbt.tugraz.at“
We have 50 licenses, if you aren‘t allowed in, please try again later.

55. Book Recommendation Only available as an ebook: The Linux Command Line
A Complete Introduction
By William E. Shotts Jr.
An alternative:
Learning the bash Shell, 3rd Edition
Unix Shell Programming
By Cameron Newham

56. Your assignment for the weekend
Download Filezilla and ThinLinc client and install them on your home computer.
Connect to cbt01.cbt.tugraz.at and play through a UNIX tutorial some more.

57. Max Malek malek@tugraz.at
Emergency Assistance
Max Malek