Presentation is loading. Please wait.

Presentation is loading. Please wait.

Make makefiles pipelining for the masses. Make is based on set of rules # this is a remark target : prerequisites... recipe_line1 recipe_line2...

Similar presentations


Presentation on theme: "Make makefiles pipelining for the masses. Make is based on set of rules # this is a remark target : prerequisites... recipe_line1 recipe_line2..."— Presentation transcript:

1 Make makefiles pipelining for the masses

2 Make is based on set of rules # this is a remark target : prerequisites... recipe_line1 recipe_line2...

3 Simple example Leg.pdf : Leg.tree R < plotTree.R Leg.tree Leg.tree : Leg.aln RAxML -f a -m GTRGAMMA -s Leg.aln > Leg.tree Leg.aln : Lpne.aln Ldra.aln Lisr.aln Ljor.aln perl fastaMsaConcat.pl Lpn.aln Ldra.aln … > Leg.aln Lpne.aln : Lpne.fasta prank -F Lpne.fasta > Lpne.aln Ldra.aln : Ldra.fasta prank -F Ldra.fasta > Ldra.aln Lisr.aln : Lisr.fasta prank -F Lisr.fasta > Lisr.aln Lsha.aln : Lsha.fasta prank -F Lsha.fasta > Lsha.aln Recipe will be executed if either: 1. Target does not exist 2. Target is older than prerequisite

4 Makefile is a tree Leg.pdfLeg.treeLeg.alnLpne.alnLpne.fasLdra.alnLdra.fasLisr.alnLisr.fasLsha.alnLsha.fas Recipe will be executed if either: 1. Target does not exist 2. Target is older than prerequisite

5 Simple example Leg.pdf : Leg.tree R < plotTree.R Leg.tree Leg.tree : Leg.aln RAxML -f a -m GTRGAMMA -s Leg.aln > Leg.tree Leg.aln : Lpne.aln Ldra.aln Lisr.aln Ljor.aln perl fastaMsaConcat.pl Lpn.aln Ldra.aln … > Leg.aln Lpne.aln : Lpne.fasta prank -F Lpne.fasta > Lpne.aln Ldra.aln : Ldra.fasta prank -F Ldra.fasta > Ldra.aln Lisr.aln : Lisr.fasta prank -F Lisr.fasta > Lisr.aln Lsha.aln : Lsha.fasta prank -F Lsha.fasta > Lsha.aln Makefile makes the first target (the order of the rest is not important

6 Variables alns = Lpne.aln Ldra.aln Lisr.aln Ljor.aln Leg.tree.pdf : Leg.tree R < plotTree.R Leg.tree Leg.tree : Leg.aln RAxML -f a -m GTRGAMMA -s Leg.aln > Leg.tree Leg.aln : $(alns) perl fastaMsaConcat.pl $(alns) > Leg.aln Lpne.aln : Lpne.fasta prank -F Lpne.fasta > Lpne.aln Ldra.aln : Ldra.fasta prank -F Ldra.fasta > Ldra.aln Lisr.aln : Lisr.fasta prank -F Lisr.fasta > Lisr.aln Lsha.aln : Lsha.fasta prank -F Lsha.fasta > Lsha.aln

7 Variables dir = ~/Legionella/Genomes/phase1/ alns = $(dir)Lpne.aln $(dir)Ldra.aln $(dir)Lisr.aln … $(dir)Leg.tree.pdf : $(dir)Leg.tree R < plotTree.R $(dir)Leg.tree $(dir)Leg.tree : $(dir)Leg.aln RAxML -f a -s $(dir)Leg.aln > $(dir)Leg.tree $(dir)Leg.aln : $(alns) perl fastaMsaConcat.pl $(alns) > $(dir)Leg.aln $(dir)Lpne.aln : $(dir)Lpne.fasta prank -F $(dir)Lpne.fasta > $(dir)Lpne.aln $(dir)Ldra.aln : $(dir)Ldra.fasta prank -F $(dir)Ldra.fasta > $(dir)Ldra.aln $(dir)Lisr.aln : $(dir)Lisr.fasta prank -F $(dir)Lisr.fasta > $(dir)Lisr.aln $(dir)Lsha.aln : $(dir)Lsha.fasta prank -F $(dir)Lsha.fasta > $(dir)Lsha.aln

8 Special Variables dir = ~/Legionella/Genomes/phase1/ alns = $(dir)Lpne.aln $(dir)Ldra.aln $(dir)Lisr.aln … $(dir)Leg.tree.pdf : $(dir)Leg.tree R < plotTree.R $^ $(dir)Leg.tree : $(dir)Leg.aln RAxML -f a -s $^ > $@ $(dir)Leg.aln : $(alns) perl fastaMsaConcat.pl $^ > $@ $(dir)Lpne.aln : $(dir)Lpne.fasta prank -F $^ > $@ $(dir)Ldra.aln : $(dir)Ldra.fasta prank -F $^ > $@ $(dir)Lisr.aln : $(dir)Lisr.fasta prank -F $^ > $@ $(dir)Lsha.aln : $(dir)Lsha.fasta prank -F $^ > $@ Useful special variables: $@ - the target $^ - all prerequisites $< - first prerequisite

9 Rulez dir = ~/Legionella/Genomes/phase1/ alns = $(dir)Lpne.aln $(dir)Ldra.aln $(dir)Lisr.aln … $(dir)Leg.tree.pdf : $(dir)Leg.tree R < plotTree.R $^ $(dir)Leg.tree : $(dir)Leg.aln RAxML -f a -s $^ > $@ $(dir)Leg.aln : $(alns) perl fastaMsaConcat.pl $^ > $@ %.aln : %.fasta prank -F $^ > $@ Another useful special variable is $* that matches % Rules will simplify your makefile and raise level of abstraction

10 Rulez dir = ~/Legionella/Genomes/phase1/ alns = $(dir)Lpne.aln $(dir)Ldra.aln $(dir)Lisr.aln … %.tree.pdf : %.tree R < plotTree.R $^ %.tree : %.aln RAxML -f a -s $^ > $@ $(dir)Leg.aln : $(alns) perl fastaMsaConcat.pl $^ > $@ %.aln : %.fasta prank -F $^ > $@ General rules can be transferred among makefiles Intermediate files, which are created by rules and are not specified as targets themselves, are by default deleted after no longer needed. (in a few slides we'll see how to change that behavior)

11 Some more variables dir = ~/Legionella/Genomes/phase1/ alns = $(dir)Lpne.aln $(dir)Ldra.aln $(dir)Lisr.aln … ALIGN = prank -F RECONSTRUCT = RAxML -f a -s %.tree.pdf : %.tree R < plotTree.R $^ %.tree : %.aln $(RECONSTRUCT) $^ > $@ $(dir)Leg.aln : $(alns) perl fastaMsaConcat.pl $^ > $@ %.aln : %.fasta $(ALIGN) $^ > $@ Use variables for things you think you might want to change between runs (e.g., BLAST flags)

12 Some functions (there are many many more…) $(addsuffix, ) $(addsuffix.ext,a dir/b)=> a.ext dir/b.ext $(addprefix, ) $(addprefix dir/,a b.ext)=> dir/a dir/b.ext $(dir ) $(dir tmp/a dir/b.ext c)=> temp/ dir/./ $(notdir ) $(notdir dir/b.ext c)=> b.ext c $(basename ) $(notdir a.R dir/b.ext c)=> a dir/b c $(shell )

13 Function usage dir = ~/Legionella/Genomes/phase1/ alns = $(addprefix $(dir), Lpne.aln Ldra.aln Lisr.aln …) ALIGN = prank -F RECONSTRUCT = RAxML -f a -s %.tree.pdf : %.tree R < plotTree.R $^ %.tree : %.aln $(RECONSTRUCT) $^ > $@ $(dir)Leg.aln : $(alns) perl fastaMsaConcat.pl $^ > $@ %.aln : %.fasta $(ALIGN) $^ > $@

14 Function usage dir = ~/Legionella/Genomes/phase1/ alns = $(addprefix $(dir),$(addsuffix.aln, Lpne Ldra Lisr …)) ALIGN = prank -F RECONSTRUCT = RAxML -f a -s %.tree.pdf : %.tree R < plotTree.R $^ %.tree : %.aln $(RECONSTRUCT) $^ > $@ $(dir)Leg.aln : $(alns) perl fastaMsaConcat.pl $^ > $@ %.aln : %.fasta $(ALIGN) $^ > $@

15 More functions $(shell ) Executes shell command and returns values as list (newlines are turned to spaces). $(shell cat b.ext)=> the content of b.ext (as a list) $(LIST:.fas=.aln) Changes all extensions of LIST Legs = Lpne.fas Ldra.fas Lisr.fas Ljor.fas $(Legs:.aln=.fas) => Lpne.aln Ldra.aln Lisr.aln Ljor.aln

16 Function usage dir = ~/Legionella/Genomes/phase1/ alns = $($(shell ls $(dir)*.fasta):.fasta=.aln) ALIGN = prank -F RECONSTRUCT = RAxML -f a -s %.tree.pdf : %.tree R < plotTree.R $^ %.tree : %.aln $(RECONSTRUCT) $^ > $@ $(dir)Leg.aln : $(alns) perl fastaMsaConcat.pl $^ > $@ %.aln : %.fasta $(ALIGN) $^ > $@

17 Special targets dir = ~/Legionella/Genomes/phase1/ alns = $($(shell ls $(dir)*.fasta):.fasta=.aln) ALIGN = prank -F RECONSTRUCT = RAxML -f a -s All : $(dir)Leg.tree.pdf %.tree.pdf : %.tree R < plotTree.R $^ %.tree : %.aln $(RECONSTRUCT) $^ > $@ $(dir)Leg.aln : $(alns) perl fastaMsaConcat.pl $^ > $@ %.aln : %.fasta $(ALIGN) $^ > $@ clean: rm -vf $(dir)Leg.tree.pdf $(dir)Leg.tree $(dir)*.aln.SECONDARY: Intermediate targets not to removed (no target will save all).Phony : A phony target is one that is not really the name of a file. It is just a name for some commands to be executed when you make an explicit request

18 Invoking makefile By default run make to execute the first target of the file called makefile > make You can invoke a specific target in a specific file: > make -f make -f makeTree.mk clean You can also pass values of variables (will override variables with in the makefile > make -f = make -f makeTree.mk DIR=/mydir/

19 Useful flags for make file -j : use n threads (without n - as many as possible) as makefile knows the dependencies he can run the non-dependent recipes in parallel… -n : just echo commands, do not excute -d : debug (shows you what makefile thinks)

20 REAL-LIFE EXAMPLES Let's check out some

21 Here are some makefiles I did A real makefile example: http://www.tau.ac.il/~davidbur/makePhylogeny.mk http://www.tau.ac.il/~davidbur/makePhylogeny.mk Huge (but well documented) makefile: http://www.tau.ac.il/~davidbur/makeCompileFeaturesByPtts.makefile http://www.tau.ac.il/~davidbur/makeCompileFeaturesByPtts.makefile I use such a makefile to run the one in (2): http://www.tau.ac.il/~davidbur/makefile http://www.tau.ac.il/~davidbur/makefile This makefile creates a makefile similar to (3) and executes it (getting weird, isn't it…): http://www.tau.ac.il/~davidbur/makeLearningHappen.makefile http://www.tau.ac.il/~davidbur/makeLearningHappen.makefile

22 useful tips and hints - before the commnad will tell make to continue even if the execution of that line fails When using "$", for example in a Perl oneliner, use $$ to let makefile know you mean the character '$' Beware of extra spaces at the end of lines You can use the program/script itself as part of prerequisit Some time it is useful to invoke make from within a makefile You can use include FILE to add the content of FILE to this point in the makefile

23 References GNU make tutorial, everything you want to know about make: http://www.gnu.org/software/make/manual/make.html Eyali's Blog in the post (my initial inspiration): http://lifeofadataminer.blogspot.co.il/2009/08/writing-makefile-instead-of- pipeline.html A pipeline is a make (I think his initial inspiration): http://archive.nodalpoint.org/2007/03/18/a_pipeline_is_a_makefilehttp://archive.n odalpoint.org/2007/03/18/a_pipeline_is_a_makefile


Download ppt "Make makefiles pipelining for the masses. Make is based on set of rules # this is a remark target : prerequisites... recipe_line1 recipe_line2..."

Similar presentations


Ads by Google