1. Hackinars in Bioinformatics
Unix basics and usage of Computerome
Erland Hochheim
DTU Systems Biology
Center for Biological Sequence Analysis

2. Log on the Danish National Life Sciences Supercomputer – aka
Log on the Danish National Life Sciences Supercomputer – aka. Computerome
Prerequisites
Some sort of personal computer
SSH client
Linux and Mac have built-in terminal programs which support SSH
Terminal, Xterm, etc.
Windows does not…
PuTTY is a good choice for most people
MobaXterm is widely used, but 2-factor authentication can be a bit tricky
How to do
Use User-ID and Password provided from DTU
Login to
homesystem$ ssh
~]$

3. Login to Computerome

4. Be familiar with the strengths and weaknesses of each computer including their own laptop.
Easy to carry
Provides access to resources elsewhere
Computerome
500+ nodes, cores
3PB available storage
Somewhat difficult to carry
Provides massive resources from (almost) everywhere

5. Exercise: Open terminal and login to Computerome
Find the terminal program on your computer
Linux and Mac should be built-in
Windows might want to download PuTTY
Login to Computerome

6. Basic Linux/UNIX commands
Manual pages: man
Help flag: --help, -h
List files and directories: ls
List, long form: ls -l List all: ls -a
Change directory: cd
Move files and directories: mv
Copy file and directories: cp
Copy recursively: cp -r
Copy, and protect permissions: cp -p
Remove files: rm
Remove recursively: rm -r
Force remove: rm -f
Create directory: mkdir
Create structure directory with parents: mkdir -p

7. man command – example ‘man ls’

8. --help flag

9. ls

10. cd, cp, mv, rm

11. Exercise: Experiment with commands
‘man ls’ - see what ls command can do
Try some different flags to ls: ls -l ls -a others…
Change directory: cd
Get help on mv command mv --help
mv -h
Copy file and directories: cp
Copy recursively: cp -r
Copy, and protect permissions: cp -p
Remove files: rm
Remove recursively: rm -r
Force remove: rm -f
Create directory: mkdir
Create structure directory with parents: mkdir -p

12. Commands for viewing files
Concatenate files: cat
Output first part of file: head
Output last part of file: tail
View contents of file: more
“Opposite of more”: less

13. Viewing - cat

14. Viewing - head, tail

15. Viewing - more, less, example ‘less outfile’

16. Pipes and redirects | Pipe, example: date | awk '{print $3,$2,$6}'
> [FILE]
Overwrite file, example: echo Hello World > hello.txt
>> [FILE]
Append to file, example: echo Hello CBS >> hello.txt

17. Pipes and redirects - |, >, >>

18. Editors on Linux/UNIX machines
vi and vim Programmers text editors
nedit NEdit is a standard GUI (Graphical User Interface) style text editor
gedit Graphical text editor for Gnome

19. Editors

20. Editing - vim hello.txt

21. <use awk/nawk/gawk, sed, sort, count for formatting of text>

22. Basic file analysis Report lines, words and bytes: wc
Sort contents: sort

23. Basic analysis - wc

24. Basic analysis - sort

25. File and directory permissions
ls –l to see permissions on files
drwxrwxr-x 3 erhh erhh Mar 12 09:00 course
-rw erhh erhh Mar 10 12:48 course/STDIN.o3929
-rw-rw-r-- 1 erhh erhh Mar 10 12:55 course/tmpdir/Graph
ugo
User
Group
Owner
rwx
Read
Write
eXecute
Change owner: chown
Change group: chgrp
Change permissions: chmod

26. Permissions and their consequences

27. File transfer between computers
Create a portable archive: tar
Secure copy: scp
Get stuff from the Web
wget
curl

28. Transfer - tar, scp

29. Transfer - wget, curl

30. <Knowing about good coding practices (Leon’s 3papers)>

31. Computerome Wiki
Main information source for all things Computerome
A living document
Corrections to be mailed to
Special page for Tips and Tricks

32. Module environment module avail module load module initadd
Reference:

33. module avail, module load, module initadd

34. Running jobs on Computerome
Jobs are run in a Batch environment
Moab scheduler
Torque resource manager
Reference:

35. Job submission qsub msub xqsub xmsub
Reference:

36. Monitoring jobs qstat showq checkjob pestat
Reference:

37. Simple job submission with qsub

38. Monitoring - watch showq

39. Monitoring - showq -c

40. Monitoring - qstat

41. Monitoring - checkjob

42. Monitoring - pestat

43. Job control canceljob showstart tracejob
Reference:

44. Job control – canceljob, showstart

45. Job control - tracejob

46. Job control – checkjob –v –v

47. Exercise with Velvet Login to Computerome
Go to your subdirectory in hackinars project directory /home/projects/pr_hackinars/people/<userdir>
Copy data/Strain_H fastq to you own directory $ cp /home/projects/pr_hackinars/data/Strain_H fastq .

48. Exercise with Velvet – cont.
Create a basic jobfile (script) $ vi velvet.sh #!/bin/bash velveth tmpdir 21 -fastq Strain_H fastq velvetg tmpdir
Make it executable $ chmod +x velvet.sh

49. Exercise with Velvet – cont.
Load modules $ module load tools $ module load moab torque $ module load velvet/1.2.10
Run velvet job using xmsub $ xmsub -W group_list=pr_hackinars -A pr_hackinars \ > -l nodes=1:ppn=2,mem=100m,walltime=3600 \ > -V -d $PWD -ro outfile -re errorfile -de ./velvet.sh
Watch job $ showq -u <userid> or watch showq -u <userid> $ qstat -u <userid> or watch qstat -u <userid> $ checkjob <jobid> $ tracejob <jobid> $ cat outfile $ cat errorfile $ cat tmpdir/Log

50. Exercise with Velvet – cont.
Extend existing jobfile (script) $ vi velvet.sh #!/bin/sh ### Account information #PBS -W group_list=pr_hackinars -A pr_hackinars ### Job name #PBS -N velvet_test ### Output files #PBS -e errorfile #PBS -o outfile ### Number of nodes #PBS -l nodes=1:ppn=2 ### Memory #PBS -l mem=100m ### Requesting time (format dd:hh:mm:ss or just number of seconds) #PBS -l walltime=1:00:00 ### Script goes below here # Go to the directory from where the job was submitted (initial directory is $HOME) echo Working directory is $PBS_O_WORKDIR cd $PBS_O_WORKDIR module load tools module load velvet/ velveth tmpdir 21 -fastq Strain_H fastq velvetg tmpdir
Run job $ qsub ./velvet.sh

51. BONUS: Estimating number of cores
gzip
gunzip
gzip
PERL Script
gzip