1. 1 Perl Programming for Biology The Bioinformatics Unit G.S. Wise Faculty of Life Science Tel Aviv University, Israel October 2009 By Eyal Privman and Dudu Burstein http://ibis.tau.ac.il/twiki/bin/view/Bioinformatics/PERL2009

2. 2 Why biologists need computers? Collecting and managing data http://www.ncbi.nlm.nih.gov/ Searching databases http://www.ncbi.nlm.nih.gov/BLAST/ Interpreting data Browsing genomes - http://genome.ucsc.edu/http://genome.ucsc.edu/ Protein function prediction - http://smart.embl- heidelberg.de/http://smart.embl- heidelberg.de/ Gene expression - http://www.bioconductor.org/http://www.bioconductor.org/ Protein-protein interactions - http://string.embl.de/http://string.embl.de/

3. 3 Why biologists need to program? (or: why are you here?)

4. 4 Why biologists need to program? A real life example Proto-oncogene activation by retrovirus insertion c-Myc: an example for transformation caused by over- or misexpression (In w.t. cells c-Myc is expressed only during the G 1 phase).

5. 5 A real life example Shmulik >tumor1 TAGGAAGACTGCGGTAAGTCGTGATCTGAGCGGTTCCGTTACAGCTGCTA CCCTCGGCGGGGAGAGGGAAGACGCCCTGCACCCAGTGCTG... >tumor157 Run BLAST: http://www.ncbi.nlm.nih.gov/BLAST/ Click “Reformat these results”, choose “Show alignment as plain text”, click “view report” and save it to a text file: http://www.ncbi.nlm.nih.gov/BLAST/ Score E Sequences producing significant alignments: (bits) Value ref|NT_039621.4|Mm15_39661_34 Mus musculus chromosome 15 genomic... 186 1e-45 ref|NT_039353.4|Mm6_39393_34 Mus musculus chromosome 6 genomic c... 38 0.71 ref|NT_039477.4|Mm9_39517_34 Mus musculus chromosome 9 genomic c... 36 2.8 ref|NT_039462.4|Mm8_39502_34 Mus musculus chromosome 8 genomic c... 36 2.8 ref|NT_039234.4|Mm3_39274_34 Mus musculus chromosome 3 genomic c... 36 2.8 ref|NT_039207.4|Mm2_39247_34 Mus musculus chromosome 2 genomic c... 36 2.8 >ref|NT_039621.4|Mm15_39661_34 Mus musculus chromosome 15 genomic contig, strain C57BL/6J Length = 64849916 Score = 186 bits (94), Expect = 1e-45 Identities = 100/102 (98%) Strand = Plus / Plus Query: 1 taggaagactgcggtaagtcgtgatctgagcggttccgttacagctgctaccctcggcgg 60 ||||||||||||||| ||||||||||||||||||||||| |||||||||||||||||||| Sbjct: 23209391 taggaagactgcggtgagtcgtgatctgagcggttccgtaacagctgctaccctcggcgg 23209450...

6. 6 A Perl script can do it for you Shmulik writes a simple Perl script to parse blast results and find all hits that are in the myc locus, or up to 10kb from it: Use the BioPerl package SearchIO Open and read file “mice.blast” Iteration – for each blast result: If we hit the genomic sequence “Mm15_39661_34” in the coordinates of the Myc locus (23,198,120.. 23,223,004) then print this hit (hit number and position in locus) We’ll get back to this later…

7. 7 What is Perl ? Perl was created by Larry Wall. (read his forward to the book “Learning Perl”) Perl = Practical Extraction and Report Language (or: Pathologically Eclectic Rubbish Lister)‏forward to the book “Learning Perl” Perl is an Open Source project Perl is a cross-platform programming language.

8. 8 Why Perl ? Perl is a popular programming language, especially for bioinformatics Perl allows a rapid development cycle Perl is strong in text manipulation Perl can easily handle files and directories Perl can easily run other programs Perl doesn’t impose arbitrary limitations (e.g. memory)‏

9. 9 Perl & biology BioPerl: “An international association of developers of open source Perl tools for bioinformatics, genomics and other fields in life science research” http://bioperl.org/ http://bioperl.org/ Many smaller projects, and millions of little pieces of biological Perl code (which you should use as references – google and find them!)‏

10. 10 This workshop No experience in programming is assumed Hands-on practice Programming tasks for molecular biology Read and manipulate sequence files Extract and analyze desired information from large files For your convenience, download this presentation from: http://ibis.tau.ac.il/perluser/2010/workshop/workshop.ppt http://ibis.tau.ac.il/perluser/2010/workshop/workshop.ppt Save it on your computer (choose “Save”, not “Open”) It will be useful to copy-paste lines from my slides to your scripts…

11. 11 Further study... I cannot teach a full Perl course in 3 hours You could read a book: Beginning Perl for BioinformaticsBeginning Perl for Bioinformatics Or some of the great Perl tutorials on the internet … (Google!) Or take the full Perl course! (this semester) Text file handling Using complex data structure Using BioPerl tools for common tasks such as: Reading/writing sequence files in different formats Reverse-complementing and translating DNA sequences Analyzing BLAST results, Genbank records, Swiss-Prot And more …

12. 12 Free Perl software (for Windows) Getting Perl: http://www.activestate.com/Products/ActivePerl/ http://www.activestate.com/Products/ActivePerl/ (Follow the links to download, and choose “MSI” for windows) Editor & debugger: http://www.perl-express.com

13. 13 Perl documentation There ’ s lots of Perl materials on the web: Use the central Perl web site: http://www.perl.org/ Look in “ Online Documentation ”, “ Manual Pages ”, “ Functions ”, etc.http://www.perl.org/ Perl-Express: In the “ Directory Window ” click the “ Perl Function ” button (it looks like a purple book), and type the name of a Perl function Or – Google what you ’ re looking for! e.g. “Perl”, “reverse” and “complement”

14. 14 The Perl-Express editor

15. 15 A first Perl script print "Hello world!"; A Perl statement must end with a semicolon “ ; ” The print function outputs some information to the terminal screen Try it yourself! Use Perl Express to write the script in a file named “ hello.pl ” (Save it in D:\perl_workshop) Run it!

16. 16 Output tab Output of run Perl Express – running a script Run the script Warnings and errors

17. 17 Data TypeDescription scalarA single number or string value 9 -17 3.1415 "hello" arrayAn ordered list of scalar values (9,-15,3.5) Data types

18. 18 1. Scalar Data

19. 19 Scalar values A scalar is either a number: 3 -20 3.1415 1.3e4 (= 1.3 × 10 4 ) or a string: print "hello world"; hello world print "hello\nworld"; hello world

20. 20 Variables Variable declaration: my $priority; Note: Everything in Perl is case sensitive! i.e. $priority is different from $Priority Scalar variables can store scalar values: Numerical assignment: $priority = 1; String assignment: $priority = "high"; Copy the value of variable $b into $a: $a = $b; Important: To make Perl check the correctness of your variable names – always include: use strict; as the first line of all scripts!

21. 21 Interpolating variables into strings $a = 9.5; print "a is $a!\n"; a is 9.5!

22. 22 Built-in Perl functions: The length function The length function returns the length of a string: print length("length"); 6

23. 23 The substr function The substr function extracts a substring out of a string. It receives 3 arguments: substr(EXPR,OFFSET,LENGTH) For example: $str = "university"; $sub = substr ($str, 3, 5); $sub is now "versi", and $str remains unchanged. Note: If length is omitted, everything to the end of the string is returned. You can use variables as the offset and length parameters. The substr function can do a lot more, google it and you will see…

24. 24 Reading input <STDIN> allows us to get input from the user: print "What is your name?\n"; my $name = <STDIN>; print "Hello $name!"; Here is a test run: What is your name? Shmulik Hello Shmulik ! $name: "Shmulik\n"

25. 25 $name: "Shmulik\n" Reading input Use the chomp function to remove the “new-line” from the end of the string (if there is any): print "What is your name?\n"; my $name = <STDIN>; chomp $name; # Remove the new-line print "Hello $name!"; Here is a test run: What is your name? Shmulik Hello Shmulik! $name: "Shmulik"

26. 26 Perl Express – entering input Click “ Std. Input ”

27. 27 Click “ i/o ” Perl Express – entering input

28. 28 Go back to “ Std. Output ” Perl Express – entering input Enter input

29. 29 Exercise 1 1.Write a script that prints "goodbye world!" 2.Assign your name into a variable and then print this variable 3.Read an input line and print it three times, and then print its length

30. 30 2. Controls: Ifs and Loops

31. 31 Controls: if ? Controls allow non-sequential execution of commands, and responding to different conditions else { print "Here is your beer!\n"; } print "How old are you?\n"; my $age = ; # Read number if ($age < 18) { print "How about some orange juice?\n"; }

32. 32 Comparison operators StringNumericComparison eq==Equal ne!=Not equal lt<Less than gt>Greater than le<= Less than or equal to ge>= Greater than or equal to if ($age == 18)... if ($name eq "Yossi")... if ($name ne "Yossi")... if ($name lt "n")...

33. 33 Boolean operators if (($age==18) or ($name eq "Yossi")){... } if (($age==18) and ($name eq "Yossi")){... } if (not ($name eq "Yossi")){... } and or not

34. 34 Controls: Loops Loops allow iterating over a list of inputs, and performing some actions for each input line. We can repeat a loop until something happens: while (length $name > 1) { $name = ; chomp $name; print "Hello $name!\n"; }

35. 35 Class exercise 2 1.Read several protein sequences in FASTA format (see for example the file “EHD.fasta” in the zip file from the workshop webpage), and print only their header lines (lines that start with “>”). Quit the program when you encounter an empty line. 2*.Now print the last 20 amino acid of each sequence

36. 36 3.

37. 37 The BioPerl project is an international association of developers of open source Perl tools for bioinformatics, genomics and life science research. Things you can do with BioPerl: Read and write sequence files of different format, including: Fasta, GenBank, EMBL, SwissProt and more … Extract gene annotation from GenBank, EMBL, SwissProt files Read and analyze BLAST results. Read and process phylogenetic trees and multiple sequence alignments. Analysing SNP data. And more … BioPerl

38. 38 A module or a package is a collection of functions. e.g. The module Bio::SearchIO contains functions that read sequence search results, such as BLAST output files. In order to write a script that uses a module add a “ use ” line at the beginning of the script. For example: use Bio::SearchIO; Using modules

39. 39 Installing modules from the internet The best place to search for Perl modules that can make your life easier is: http://www.cpan.org/ http://www.cpan.org/ The easiest way to download and install a module is to use the Perl Package Manager (part of the ActivePerl installation) Note: ppm installs the packages under the directory “site\lib\” in the ActivePerl directory. You can put packages there manually if you would like to download them yourself from the net, instead of using ppm. Choose “ View all packages ” Enter module name

40. 40 BioPerl modules are called Bio::XXX You can use the BioPerl wiki: http://bio.perl.org/ with documentation and examples for how to use them – which is the best way to learn BioPerl. We recommend beginning with the "how-tos": http://www.bioperl.org/wiki/HOWTOs http://www.bioperl.org/wiki/HOWTOs For a more hard-core inspection of BioPerl modules: BioPerl 1.5.2 Module Documentation BioPerl

41. 41 First we need to have the BLAST results in a text file BioPerl can read. Here is one way to achieve this: BioPerl: reading BLAST output Text Download

42. 42 BioPerl: reading BLAST output

43. 43 BioPerl: reading BLAST output

44. 44 Why biologists need to program? A real life example Proto-oncogene activation by retrovirus insertion c-Myc: an example for transformation caused by over- or misexpression (In w.t. cells c-Myc is expressed only during the G 1 phase).

45. 45 A real life example Shmulik >tumor1 TAGGAAGACTGCGGTAAGTCGTGATCTGAGCGGTTCCGTTACAGCTGCTA CCCTCGGCGGGGAGAGGGAAGACGCCCTGCACCCAGTGCTG... >tumor157 Run BLAST: http://www.ncbi.nlm.nih.gov/BLAST/ Click “Reformat these results”, choose “Show alignment as plain text”, click “view report” and save it to a text file: http://www.ncbi.nlm.nih.gov/BLAST/ Score E Sequences producing significant alignments: (bits) Value ref|NT_039621.4|Mm15_39661_34 Mus musculus chromosome 15 genomic... 186 1e-45 ref|NT_039353.4|Mm6_39393_34 Mus musculus chromosome 6 genomic c... 38 0.71 ref|NT_039477.4|Mm9_39517_34 Mus musculus chromosome 9 genomic c... 36 2.8 ref|NT_039462.4|Mm8_39502_34 Mus musculus chromosome 8 genomic c... 36 2.8 ref|NT_039234.4|Mm3_39274_34 Mus musculus chromosome 3 genomic c... 36 2.8 ref|NT_039207.4|Mm2_39247_34 Mus musculus chromosome 2 genomic c... 36 2.8 >ref|NT_039621.4|Mm15_39661_34 Mus musculus chromosome 15 genomic contig, strain C57BL/6J Length = 64849916 Score = 186 bits (94), Expect = 1e-45 Identities = 100/102 (98%) Strand = Plus / Plus Query: 1 taggaagactgcggtaagtcgtgatctgagcggttccgttacagctgctaccctcggcgg 60 ||||||||||||||| ||||||||||||||||||||||| |||||||||||||||||||| Sbjct: 23209391 taggaagactgcggtgagtcgtgatctgagcggttccgtaacagctgctaccctcggcgg 23209450...

46. 46 A Perl script can do it for you Shmulik writes a simple Perl script to parse blast results and find all hits that are in the myc locus, or up to 10kb from it: Use the BioPerl package SearchIO Open and read file “ mice.blast ” Iteration – for each blast result: If we hit the genomic sequence “ Mm15_39661_34 ” in the coordinates of the Myc locus (23,198,120.. 23,223,004) then print this hit (hit number and position in locus)

47. 47 We can use the module Bio::SearchIO to read a text file with blast results: use Bio::SearchIO; Use the new command to open the results file: (using Bio::SearchIO) my $blast_report = new Bio::SearchIO ('-format' => 'blast', '-file' => 'mice.blast'); There are three levels to blast results: $result = $blast_report->next_result (a blast query) $hit = $result->next_hit (a blast hit) $hsp = $hit->next_hsp (a “ high scoring pair ” – an alignment of a certain region) BioPerl: reading blast output

48. 48 A Perl script can do it for you use Bio::SearchIO; my $blast_report = new Bio::SearchIO ('-format'=>'blast', '-file' =>'mice.blast'); while (my $result = $blast_report->next_result) { print "Checking query ", $result->query_name, "...\n"; my $hit = $result->next_hit(); my $hsp = $hit->next_hsp(); if ($hit->name() eq "ref|Mm15_39661_34" and $hsp->hit->start() > 23198120 and $hsp->hit->end() < 23223004) { print " hit ", $hit->name(); print " (at position ", $hsp->hit->start(), ")\n"; } } Use the BioPerl package SearchIO Open file “ mice.blast ” Iterate over all blast results For each blast hit – ask if we hit the genomic sequence “ Mm15_39661_34 ” in the coordinates of the Myc locus 23,198,120..23,223,004 If so – print hit name and position

49. 49 A Perl script can do it for you Checking query tumor1... hit ref|NT_039621.4|Mm15_39661_34 (at position 23209391) Checking query tumor2... Checking query tumor3... Checking query tumor4... hit ref|NT_039621.4|Mm15_39661_34 (at position 23211826) Checking query tumor5... Checking query tumor6... Checking query tumor7... hit ref|NT_039621.4|Mm15_39661_34 (at position 23210877) Checking query tumor8... Checking query tumor9... Checking query tumor10... Checking query tumor11... hit ref|NT_039621.4|Mm15_39661_34 (at position 23213713) Checking query tumor12...

50. 50 Class exercise 3 1.Change the script “ ex3.pl ” : limit the search to the region of the 2 nd and 3 rd exons, which are in coordinates: 23210377.. 23215453 2.Now print just hits with e-value smaller than 10 -40

51. 51 The Bio::SeqIO module allows reading/writing sequences from/to files, using many file formats (fasta, Genbank, EMBL … ) use Bio::SeqIO; $in = new Bio::SeqIO("-file" => "inputFileName", "-format" => "embl"); $out = new Bio::SeqIO("-file" => ">outputFileName", "-format" => "fasta"); while ( my $seqObj = $in->next_seq() ) { $out->write_seq($seqObj); } BioPerl: the SeqIO module

52. 52 The Bio::SeqIO function “ next_seq ” returns an object of the Bio::Seq module. This module provides functions like id, accession, length and subseq (read about them in the documentation!): use Bio::SeqIO; $in = new Bio::SeqIO("-file" => "inputfilename", "-format" => "fasta"); while ( my $seqObj = $in->next_seq() ) { print "Sequence ",$seqObj->id(),"\n"; print "First 10 bases "; print $seqObj->subseq(1,10),"\n"; } And other functions such as: length, revcom, translate, etc. BioPerl: the Seq module

53. 53 Class exercise 4 1.Change the script “ ex4.1.pl ” : use it to convert the file “ pp2c.gb ” from Genbank format to Fasta format (write file “ pp2c.fasta ” ) 2.Add to the script “ ex4.2.pl ” : print the sequence lengths 3.* Add to “ ex4.2.pl ” : calculated the average sequence length

54. 54 Thanks for your patience and See you in the full Perl course …