Intro to Linux (fun with UNIX)

Synopsis Why Linux? Your computing environment The command line interpreter (CLI/terminal/shell) Linux directory structure Working with files Pipes and redirections Where to find help? SSH + FTP If we have time… Basics of (shell) scripting Working with multiple processors User groups and permissions

Why use Linux? Learning the basics of Linux allows you to: The best tools available in the field of bioinformatics are largely non-portable, open source tools that are written for POSIX systems (such as Linux and UNIX). Learning the basics of Linux allows you to: develop a skill set that sets you apart from a wet-only biologist run bioinformatics resources on your own machine access ready-made bioinformatics computing environments do reproducible research perform analyses on computer clusters – important for long computational jobs access cloud computing and High Performance Computing (HPC) resources Linux is a multi-access, multi-tasking system: multiple users may be logged in and run multiple tasks on one machine at the same time, sharing resources (CPUs, memory, disk space) -> this is what we will be doing during the course

Your computing environment In this class you will be using a thin client. A thin client refers to either a software program or to an actual computer that relies heavily on another computer to do most of its work. Think of it as a dumb computer that allows you to access our server (much bigger computer) stored in the data center. You get access to computational resources that would not be available in a “normal” computer, such as the one you use at work or at home. The server we will be using is a HP DL 580 G7 4 cores x 8 threads x hyperthreading = 64 threads 1 TB RAM access to 100 TB storage

Logging in to our server It’s easy! You will start with a fairly familiar-looking desktop. Double click on highlighted icon.

Enter your TU Graz credentials – the credentials you use to login to online.tugraz.at

After you’ve entered your credentials, click “Connect”.

Important!

On this screen you can select how your desktop will look like. All further instructions will assume that you selected the default XFCE Desktop. Click “Ok” to continue.

Welcome to your Linux desktop

Logging out of your session At the end of each class please log out of your session. I will later cover the “screen” command which will allow you to leave your programs running even if your session has ended. To log out simply click this icon at the top left of your screen. A dialog like the one on the right will appear, make sure you have selected “Save this session for future logins” and then click the “Log Out” button. You will be returned to the initial desktop of the thin client. Try it now!

Before we begin

The command line interface (CLI) The CLI is a text-based interface that allows computer users to type commands that get executed by the operating system.  You call programs, modify files and control your system just by writing a bit of text. It is an extremely powerful tool, because it centralizes managing your system and, crucially, makes automation easier, saving you a lot of work (more on this later, if we have time). We will be using the CLI to do most of the lab tasks in this class. There are different types of Unix shells or CLIs available, the one we are using is called Bash. Other good examples are tcsh, zsh or ksh.

How to open a shell window

Linux directory structure You won’t find any Windows, Program Files, or Users folders if you start browsing around the file system on your Linux computer.  On Windows, an application might store all its files in C:\Program Files\Application. On Linux, its files would be split between multiple locations – its binaries in /usr/bin, its libraries in /usr/lib, and its configuration files in /etc/. Notice the use of slashes: in Windows you use “\” C:\Users, while in Linux you will use “/” /export/home. 3 most important folders we will be using in this course /export/home/username – your user home directory, similar to the Windows C:\Users\User /export/programs – contains all the bioinformatics software used in the course

We will browse to /export/home/sensencw – the path to my home directory Windows directories in C:\ Linux directories in / (root)

Linux directory structure at the root - / GUI CLI Linux directory structure at the root - / Graphical User Interface (GUI) versus command line view

GUI CLI /export

GUI CLI /export/home

GUI CLI /export/home/amgrecu

It’s a tree!

Paths A Windows path: C:\Program Files\ Linux path types: Full: /export/home/amgrecu/home.png Relative path to /export/home: amgrecu/home.png Relative path to /export/home/amgrecu: home.png Your home directory: /export/home/amgrecu OR ~

Working with directories After you log in to the terminal window, you are in your home directory, i.e. /export/home/amgrecu. To see in what directory you are, type pwd (print working directory)

Working with directories cd - Change (current) directory Without additional arguments, this command will take you to your home directory cd ./ - Change (current) directory to the same directory. cd /export/home/amgrecu/Documents - Change directory from wherever to /export/home/amgrecu/Documents. cd Desktop - Change (current) directory to Desktop (will work if the current directory contains “Desktop”) cd ../ - Change (current) directory one level back (closer to the root) cd ../../../ - Change (current) directory three levels back (closer to the root)

Working with directories Creating directories mkdir /export/home/amgrecu/new_dir - Make a new directory called “new_dir” in /export/home/amgrecu mkdir new_dir - Make a new directory called “new_dir” in the current directory Removing directories rmdir /export/home/amgrecu/new_dir - Remove directory called “new_dir” in /export/home/amgrecu – will fail if the directory is not empty rm –rf /export/home/amgrecu/new_dir - Remove directory called “new_dir” in /export/home/amgrecu with all its content (i.e. all files and subdirectories will be gone) rm –rf new_dir - Remove directory called “new_dir” in current directory with all its content (i.e. all files and subdirectories will be gone) rm –rf is a very dangerous command! Once you have deleted something it is GONE! There is no “Recycle Bin”! There are no backups! Check carefully what you are deleting!

Listing directory content Use ls –alh in the current directory. For even more options, man ls. file last accessed name blue - directory permissions Owner user and group size

May not work with all remote clients! Useful things to know May not work with all remote clients! Use Up/Down arrow keys – this will cycle through recently executed commands. Use the TAB key – this will often present you with a list of choices after typing a part of a command (more on this in a moment) history command: list all recently used commands history - list all remembered commands history | more - list all remembered commands page by page history | grep –i example (list all remembered commands containing string “example”, case-insensitive)

Tab Completion Writing every single character of a command is tedious. But your shell can help you! Press the Tab key while writing commands, directories or file names. Press once to complete until the last unambiguous character (you will hear a beep if there are no suggestions or you have ambiguous options). Press twice to show all completion options.

Examples: Commands: Directories: Files: Tab Completion cuffl<tab> -> cufflinks Directories: ls /ex<tab> -> ls /export Files: cp /export/data/mol923/c<tab> -> cp /export/data/mol923/completion-example.txt

More examples: Tab Completion bow<tab> -> bowtie and you hear a beep bowtie<tab><tab> :

Real World Example: is simply Tab Completion cd /export/data/mol923/Bowtie-example is simply cd /ex<tab>da<tab>m<tab>B<tab>

Working with files – Types of files There are many types of files. Here are the most important: Text files (human-readable; can be viewed and modified using a text editor) Text documents (e.g., README files) Data in text format (e.g., FASTA, FASTQ, …) Scripts: Shell scripts (*.sh) Perl scripts (*.pl) Python scripts (*.py) and many others Binary files (not human-readable; cannot be viewed using a text editor) Executables (e.g., samtools, blastn, bowtie) Data in binary format (e.g, BAM files, index files for Bowtie, formatted BLAST databases) Compressed files - archives (*.gz, *.zip, *.bz2) – often text files re-formatted to save space on disk or packaged directory trees How do you know what type of file you are looking at?

The “file” command [amgrecu@cbt01 UNIX_examples]$ file example.txt example.txt: UTF-8 Unicode English text, with very long lines [amgrecu@cbt01 UNIX_examples]$ file read read: ASCII text [amgrecu@cbt01 UNIX_examples]$ file first-script.sh first-script.sh: Bourne-Again shell script text executable [amgrecu@cbt01 UNIX_examples]$ file blastn-example.fasta blastn-example.fasta: ASCII text [amgrecu@cbt01 velvet]$ file velvetg velvetg: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped

Naming – Best Practices Names should be meaningful, e.g. sample19_blastn_out.fasta, NOT blastn.fasta Names are case-sensitive - MyFile, myfile, myFile are all different Use only letters (upper- and lower-case), numbers from 0 to 9, a dot (.), and an underscore (_) – e.g this_is_an_example.ext, NOT this &ismy example^.ext Avoid other characters! They may have special meaning in Linux While working in command line terminal you always explicitly specify a program which is supposed to open this file.

Where to find help? Every UNIX and Linux machine comes with built-in help for most of its commands and programs. Type “man” followed by the command name To leave: q For Scrolling: Arrow Up or Down for one line up or down Page Down or Ctrl-F for one page down Page Up or Ctrl-B for one page up To search A slash, then your search term, then enter. e.g. /list n to go to the next occurrence of the search term Even “man” itself has a manpage. Type man man into the terminal.

System commands on files – Copy (cp) Usage: cp <source_file> <destination_file> cp sample_data.fa /home/amgrecu/sample.fa - copy file sample_data.fa from the current directory to /home/amgrecu and give the copy a name sample.fa; destination directory must exist cp /home/amgrecu/my_script.sh . - copy file myscript.sh from /home/amgrecu to the current directory under the same file name cp /home/amgrecu/*.fastq /workdir/amgrecu - copy all files with file names ending with “.fastq” from /home/amgrecu to /workdir/amgrecu; destination directory must exist cp –r /workdir/amgrecu/test /home/amgrecu - if test is a directory, it will be copied with all its files and subdirectories to directory /home/amgrecu/test; if /home/amgrecu/test did not exist, it will be created man cp for all options to the cp command

Moving, renaming and removing files Moving and renaming files - has the same command line structure as cp Usage: mv <source_file> <destination_file> man mv for all options to the mv command Removing (deleting) files Usage: rm <file> man rm for all options to the rm command

The chmod command is used to change file permissions. 4= read 2= write 1= execute Myself, Group, World chmod 700 filename

Working with files - compression To save disk space, we can compress large files if we do not intend to use them for a while. A lot of files downloaded from the web are compressed and usually need to be uncompressed before processing can take place. Compression works best on text files, such as FASTA or FASTQ. Common compressed formats: gzip (gz) gzip my_file - compresses file my_file, producing my_file.gz gzip –d my_file.gz - decompress my_file.gz, producing my_file bzip2 bzip2 my_file - compresses to my_file.bz2 bzip2 my_file.bz2 – decompress to my_file

Working with files - compression zip zip my_file.zip my_file1 my_file2 my_file3 - create a compressed archive called my_files.zip, containing three files: my_file1, my_file2, my_file3 unzip my_files.zip - decompress the archive into the constituent files and directories tar tar -cvf my_file.tar my_file1 my_file2 my_dir - create a compressed archive called my_files.tar, containing files my_file1, my_file2 and the directory my_dir with all its content tar -xvf my_files.tar - decompress the archive into the constituent files and directories Help for compression tools: gzip -h bzip2 --help zip --help tar --help

gedit Kate nano Editing text files User-friendly options: More advanced suggestions: nano You can call them from CLI: nano filename gedit filename kate filename In the course we will generally be using gedit

Working with text files more example.txt - display the content of the file README.txt in the current directory dividing the file into pages; press SPACE bar to go to the next page head -5 big.fasta - display first 100 lines of the file big.fasta in the current directory tail -5 big.fasta - display last 100 lines of the file big.fasta in the current directory tail -10 big.fasta | more - extract the last 1000 lines of the file big.fasta and display them page by page head -1000 big.fasta | tail -100 - display lines 901 through 1000 of the file big.fasta cat big.fasta - print the file on screen. Only use for small files! wc big.fasta - display the number of lines, words, and characters in a file Note the character | - it pipes the output from one command as input to another.

Working with text files Looking for a string in a text file: grep “Error: lane” error.log - display all lines of the file error.log in the current directory which contain the string “Error: lane” Looking for a string in a group of text files: grep “Error: lane” *.out - display all files *.out in the current directory which contain the string “Error: lane”; also display the lines containing that string

Pipes and Redirections Every command line program has three streams: STDIN: Standard input STDOUT: Standard output STDERR: Standard error The output you see in the console is either from STDOUT or STDERR. If the command is interactive, it waits for input on STDIN. | A pipe character redirects the STDOUT from the left program into the STDIN of the right program > Redirects the STDOUT from the left program into the file on the right 2> Redirects the STDERR from the left program into the file on the right < Program on the left receives STDIN input from the file on the right

Pipes and Redirections Show files with a specific extension in a directory ls /export/data/mol923/Bowtie_example/reads | grep fa Capture output of program to a file blastn -db nt -query blastn-example.fasta > hits.txt Redirect error channel into a log file cuffmerge -g genes.gtf -s genome.fa assemblies.txt 2> cuffmerge_errors.txt

Where to find help No.2? Not every command has a man page. However most commands have built-in help output: cp --help blastn -help cufflinks -h It’s usually one of “-h”, “--help“, “-help” or similar You can pipe the output into a pager (use q to leave; up/down arrow Ctrl-F/Ctrl-B to navigate) cp --help | less

Where to find help No.3? Also, there is plenty of help on the Internet http://www.google.com Most problems can be solved by Googling. Basically, just type all related words into the search field. http://stackoverflow.com/ Ask specific questions here. General questions are usually already answered. You can find them through the search field, but usually they’re the top hits on Google anyway. https://www.biostars.org/ “Stackoverflow for Biologists” http://seqanswers.com/ Forum about Bioinformatics tools and –omics data processing tools

UNIX Tutorials http://people.ischool.berkeley.edu/~kevin/unix-tutorial/toc.html http://www.ee.surrey.ac.uk/Teaching/Unix/unixintro.html

https://filezilla-project.org/ Connect to „cbt01.cbt.tugraz.at“ Getting files in and out of my home computer Filezilla https://filezilla-project.org/ Connect to „cbt01.cbt.tugraz.at“ Use Port 22

Connect to „cbt01.cbt.tugraz.at“ Connecting from Home to the UNIX machines at TUGraz ThinLinc Client https://www.cendio.com/thinlinc/download Connect to „cbt01.cbt.tugraz.at“ We have 50 licenses, if you aren‘t allowed in, please try again later.

Book Recommendation Only available as an ebook: The Linux Command Line A Complete Introduction By William E. Shotts Jr. http://shop.oreilly.com/product/9781593273897.do# An alternative: Learning the bash Shell, 3rd Edition Unix Shell Programming By Cameron Newham

Your assignment for the weekend Download Filezilla and ThinLinc client and install them on your home computer. Connect to cbt01.cbt.tugraz.at and play through a UNIX tutorial some more.

Max Malek malek@tugraz.at Emergency Assistance Max Malek malek@tugraz.at