1. Editing, Transferring, and Running Files on Vieques Daniel Malmer Dowell Lab Short Reads Course 6/9/15

2. …We will use secure File Transfer Protocols (SFTP and SCP)

3. Today Edit files in the terminal (emacs, nano) Transfer files to/from vieques (scp, FileZilla) Run jobs on compute nodes (qsub)

4. First, a quick review (plus setup some stuff we need) Login to vieques ssh username@vieques.colorado.edu Make a new directory, Day3 mkdir Day3 Logout exit

5. Editing files in the terminal (emacs) (nano)

6. Editing files in the terminal Open a terminal, do not login to vieques yet! Create a new file: nano hello_world.py Write the line: print “hello world!” Save and exit: – In nano: Ctrl-o,, Ctrl-x – In emacs: Ctrl-x, Ctrl-s, Ctrl-x, Ctrl-c Run ls to see the file you just created ls

7. Transferring to vieques FileZilla vs scp / sftp

8. Transferring to vieques FileZilla: Then simply drag the file over to the Day3 directory vieques.colorado.edu Important!

9. Transferring to vieques scp (Mac/Linux only): scp hello_world.py username@vieques.colorado.edu:/Users/username/Day3/ PuTTY (Windows): – In PuTTY, open PSFTP – Type: open username@vieques.colorado.edu – Then: put hello_world.py no return here, this command is all one line

10. Transferring to vieques Log back into vieques Change into Day3 directory cd Day3 Run your script! python hello_world.py

11. Grabbing files from an FTP In a browser (chrome, safari, firefox), go to google and search for “UCSC downloads” Scroll down, click “C. elegans” Click “Full data set” Scan readme, scroll down to the blue links Right-click on chromFa.tar.gz and click “Copy link address”

12. Grabbing files from an FTP In the terminal type: wget (put a space after wget) Then paste your URL: Press enter and it will download the file:

13. gzip/tar ls to see the new file and its size: ls -lh Unzip (chromFa.tar.gz -> chromFa.tar): gunzip chromFa.tar.gz ls -lh (how big is the unzipped file?) Untar (chromFa.tar -> indv.fa files): tar -xvf chromFa.tar (google is still your best friend!) ls -lh (what do you see now?)

14. gzip/tar Now lets compress the fasta files back into a new.tar.gz: tar -cvf C_elegans_chrs.tar *.fa ls -lh Tar can create (and unzip).gz files as well: tar -cvzf C_elegans_chrs.tar.gz *.fa ls -lh Tarred/zipped files are much easier to transfer and store!

15. Coffee break?

16. Before moving on, let’s copy some more files cd /projects/sreadgrp/homeworkfiles/Day3 ls cp -r BowtieExample /Users/username/Day3 cd /Users/username/Day3/BowtieExample ls

17. Cluster architecture

18. Job queue Jobs running on compute nodes are kept in a queue To view currently running jobs, type: qstat To view only your currently running jobs: qstat -u username To stop one of your running jobs, type: qdel ###### (job_id number, do not include.vieques) To kick off a job: qsub PBS_script_file

19. Job queue After the job finishes, output files are printed in the directory you kicked the job off in job_name.o###### job_name.e###### If set in the PBS script, you’ll also receive an email after the job finishes:

20. PBS scripts Go to your BowtieExample directory: cd /Users/username/Day3/BowtieExample/ Open up first script nano run_bowtie_1core.pbs – Change email – Change the word “username” to your username Ctrl \ (to start a search and replace) Search (to replace): username (enter in the word “username”) Replace with: your_username (enter in your username, eg. dama9282) Replace this instance? A (replace all instances) Do the same for the other script nano run_bowtie_8core.pbs

21. PBS scripts You can re-use PBS scripts every time! – Copy an existing script with cp – Edit with nano/emacs/vim

22. Kicking off jobs To run a job from a PBS script, type: qsub PBS_script_file Before you do it yourselves, let me go through a quick example Always check the error output! Now go ahead and kick off the two scripts with: qsub run_bowtie_1core.pbs qsub run_bowtie_8core.pbs Check that your jobs are running: qstat -u username

23. Kicking off jobs After the jobs finish, check the output files: ls (check the filenames of the output files) less bowtie_1c.e###### (press q to exit) less bowtie_8c.e###### Check email Exit_status=0 means the job succeeded Exit_status=(anything else) usually means the job failed cput = total amount of computation that was done (aggregate time of each cpu) mem = max memory used by the job walltime = total time from the start of the job to the end of the job