Presentation is loading. Please wait.

Presentation is loading. Please wait.

Example applications of e-Infrastructure: NGS UI/WMS Jonathan Churchill - STFC/RAL

Similar presentations


Presentation on theme: "Example applications of e-Infrastructure: NGS UI/WMS Jonathan Churchill - STFC/RAL"— Presentation transcript:

1 Example applications of e-Infrastructure: NGS UI/WMS Jonathan Churchill - STFC/RAL jonathan.churchill@stfc.ac.uk

2 Summary Overview Example case study. – Logging in: SSH and MyProxy. – Parametric jobs – Head node for the grid Submitting simple jobs. Submitting ‘real’ jobs. Lab Session.

3 NGS UI/WMS User Interface / Workload Management System UMD (gLite) UI and WMS Distributions NGS Useability improvements: – ssh / proxy logins – Extensive proxy checking – Gridftp on UI “Head node for the Grid” Works with NGS and gLite/GridPP sites 12,000 + CPU Cores Rapid take up since Oct ‘09 startup.

4 NGS WMS UI WMS Information Server (BDII) MyProxy RAL-NGS2, Scotgrid-Glasgow, Oxford-OERC, Manchester-NGS2 RAL-LCG....etc 12,000+ CPUs ssh Login : happyuser@ngs.ac.uk

5 Transcriptome Analysis using the NGS UI/WMS Jonathan Churchill - STFC/RAL jonathan.churchill@stfc.ac.uk Paul Wilkinson - Exeter University

6 Tobacco Hornworm Moth Manduca sexta Green Dock Beetle Gastrophysa viridula

7 mRNA Transcript genome.gov: National Human Genome Research Institute

8 Bio Databases and Applications ftp rsync Database Mirrors: EMBL UNIPROT, TREMBL, SWISSPROT PROSITE PRINTS REBASE Pre-Installed Applications: BLAST, EMBOSS, FASTA GROMACS, MrBAYES, EXONERATE, NAMD, Siesta

9 WMS Parametric Case Study What Proteins do these ‘contigs’/ transcripts code for ? NCBI BLAST Search in the EBI Uniprot database 1 x 55,000 Contigs  1 month elapsed annotation time. 1000 x 55 Contigs + NGS + WMS  < 6 hours elapsed annotation time.  Using WMS ‘Parametric’ JDL file  one JDL for 1000 Jobs  one Submission Command  One Status Command  Outputs returned to UI Automatically

10 Create your own username / password

11 Login : SSH/PuTTY

12 JDL File Type = "Job"; JobType = "Parametric"; Executable = "/usr/ngs/BLAST-NCBI"; Arguments = "blastall -p blastx -d uniprot -i contig-_PARAM_.fsa -a 1"; StdOutput = "contig-_PARAM_.out"; StdError = "contig-_PARAM_.err"; Parameters = 997; ParameterStart = 0; ParameterStep = 1; MyProxyServer = "myproxy.ngs.ac.uk"; InputSandbox = {"contig-_PARAM_.fsa"}; InputSandboxBaseURI = "gsiftp://ngsui03.ngs.ac.uk:2811/home/ngs0055/ParamBlast/inputs"; OutputSandbox = {"contig-_PARAM_.out","contig-_PARAM_.err"}; OutputSandboxBaseDestURI = "gsiftp://ngsui03.ngs.ac.uk:2811/home/ngs0055/ParamBlast/outputs"; Requirements = ( Member("NGS-UEE-BLAST-NCBI", other.GlueHostApplicationSoftwareRunTimeEnvironment) ); Rank = other.GlueCEStateFreeCPUs; ShallowRetryCount = -1;

13 Submit & Monitor glite-wms-job-submit –a –o jobIDs blast.jdl glite-wms-job-status –i jobIDs One jobID for all 1000 jobs 1000 Output files IP & OP files copied from/to UI Jobs 2-3 hours each Head node for the grid. Peak 320 Jobs in flight Avg 150 Jobs in flight

14 Summary 50,000 Contig analysis in < 6 hours vs 1 Month ssh Username/password Logins 1000 Jobs all managed as one ‘job’. Input/Output on the UI. Head node for the Grid.

15 Summary Overview Example case study. –L–Logging in: SSH and MyProxy. –P–Parametric jobs –H–Head node for the grid Submitting simple jobs. Submitting ‘real’ jobs. Lab Session.

16 Simple Example [ Create your proxy <- UI does this for you ! ] – voms-proxy-init --voms ngs.ac.uk See what’s available lcg-infosites --vo ngs.ac.uk ce Submit the job glite-job-submit -a –o jobIDs.txt my_test.jdl https://ngswms01.ngs.ac.uk:9000/LHGIagvDl701_msz0jpIg Check the status of your job glite-job-status -i jobIDs.txt Get the output glite-job-output –i jobIds.txt --dir./outputs Note: UI/WMS can retrieve outputs automatically

17 Simple JDL file Some default parameters set on the UI Simple Example jdl Type = "Job"; JobType = "Normal"; Executable = “settings.sh"; StdOutput = “output.out"; StdError = “output.err"; InputSandBox = {“settings.sh”}; OutputSandbox = {“output.err",“output.out"}; RetryCount = 1; Requirements = ( other.GlueCEUniqueID == "ngs.rl.ac.uk:2119/jobmanager-lsf-ngs“); Rank = other.GlueCEStateFreeCPUs; Requirements = other.GlueCEStateStatus == “Production”;

18 Summary Overview Example case study. –L–Logging in: SSH and MyProxy. –P–Parametric jobs –H–Head node for the grid Submitting simple jobs. Submitting ‘real’ jobs. Lab Session.

19 NGS Applications Docs

20 Input/Output Files InputSandBox lists all input files – Inc’s binaries/scripts to run – Wildcards ok OutputSandBox lists o/p files to retrieve. – Wildcards not allowed. – Tutorial shows ‘Epilogue’ script. InputSandboxBaseURI – Avoids 3 rd party transfers via WMS server. OutputSandBoxBaseDestURI – O/P’s to UI or elsewhere. – Output dir must exist. – Files arrive before job “Done”. Type = "Job"; JobType = "mpich"; Executable = "/usr/ngs/DLPOLY2"; CpuNumber = 8; StdOutput = "std.out"; StdError = "std.err"; Myproxyserver= "myproxy.ngs.ac.uk"; InputSandbox = {"CONFIG","CONTROL","FIELD","REVCON"}; InputSandboxBaseURI = "gsiftp://ngsui03.ngs.ac.uk:2811/home/ngsxxx/dlpoly"; OutputSandbox = {"OUTPUT","STATIS","CONFIG", "CONTROL","FIELD","REVCON","REVIVE", "stdout.out","stderr.out" }; OutputSandboxBaseDestURI = "gsiftp://ngsui03.ngs.ac.uk:2811/home/ngsxxx/dlpoly"; Requirements = ( member("NGS-UEE-DLPOLY2", other.GlueHostApplicationSoftwareRunTimeEnvironment) ); ShallowRetryCount = -1;

21 Key Features Summary ssh Logins Input/Output on the UI Head node for the Grid. Single Jobs and Parametric Sweeps Normal and MPI jobs Example JDLs on wwww.ngs.ac.uk Questions : support@grid-support.ac.uk

22 Further Information NGS Web site UI-WMS Page: – http://www.ngs.ac.uk/uiwms – Links to simple WMS Tutorials (2) & app specific (Gaussian, NAMD etc) – http://www.ngs.ac.uk/applications Tutorials – http://www.ngs.ac.uk/ngs-workload-management-system-and-user-interface-tutorials – http://wiki.ngs.ac.uk/index.php?title=UI-WMS_Tutorial – http://wiki.ngs.ac.uk/index.php?title=UI-WMS_Tutorial2 Parametric Case Study: – http://www.ngs.ac.uk/mrna-analysis-using-the-ngs – http://www.ngs.ac.uk/sites/default/files/file/newsletters/Dec%202009%20NGS%20news.pdf Links to Guides and JDL attributes doc. – http://wiki.ngs.ac.uk/index.php?title=UI-WMS_Tutorial#Further_Resources

23 Questions ?

24 Lab http://wiki.ngs.ac.uk/index.php?title=UI-WMS-SeIUCCR-Tutorial Login: – SSH username = “SummerUserXX” XX is on your packs – SSH Password = “2012-SSXX” – Valid until Friday afternoon

25 Inputs >whitefly_assembly.accurate.15_lrc1 GGTATCAACGCAGAGTKCGCGGGGAGTAGAACAAAGAGCGTCTGAGAGGACTTCGCGATA GTGTTACGTTAATCGATAGCTCGTGTGTTAAAAAAATCTTTCAAGTCCTTCCTGTCTTTT GACTACTTAATTAGTTAATTATTATTTTGATCGAGACAAGCAAAGAAAAATGAATTCCAT ATTATCTTTGACCGTTTTCGTAACTTTCACAATTGTCTTGGCTCAAAGTGAACAATTAGA CAAGAACTTCGGCGTGGGCGAAATCAAGACTCGCATCCAAGATAAAAAATTTGTTGAGAA GCAGTTGGGCTGTGTCCTAGGGAAAGCCGATTGCGACACCTTAGGAAATCAGTTGAAAGT TGCCATTCCAGAAGTCCTAGTTAAAGGCTGCAAGGATTGCACTCCGGAACAATCTGCAAA TGCCAATCGATTAATAGCTTTTATAAAGATGAATTATCCAGCAGAATGGAGTCAAATTGC TGCAAAATATGGTGTGAAAGGTGATGCTGTAAAGAGGCCACGACGACATATCAGAAGGTG AAAGGAGTGATGCCAAAGATGTGATAAGTTTTTATTGTTAACTTTCGAGTCTTGACTTGA TTTGATCATTGTGTACGTATGTATTTTAATTCTTCCAATTGTGAGCAGTATTTTAAGAGG GTATTCTAAATAACAGCCGTCCAAAAAGTTTTGAACTGAAATTTAAACTGTTAAGTGTTG ATGACTTTTACCAATATTTATTTTTTTATCACCGAACTGTTAGTAATACTGCGACCAATA CAAATTTATCTTTAGTCAGCTTGATTTTTTATCAAGTTGATTCTTTTTTTTGGACAATTT TTTTTTTATTATTATTCTTCCTCATTTAATGTATGTTTAAAATTGTTAATTGACCACCAT TCGCATTTAATTGATTAAGTTTTTCTTATTTTTTTTTTATATGAACCAATGTTATAATTT TGCTCTCATAAACCTACTGTAAAATATTGAGTGTCCAGTTAAAGCTTTAAACTTTATATA TTTTAACAAAAAATTAATGAGCTATTTTATAGAACCTAATAA >whitefly_assembly.accurate.15_lrc2 TCGGGGGAGTAAATTCATGAAAGATAATCTAATCGTGCAGCCTTTTTATGAGACGCGCTG AAGTTTCGGATTAGGTTTTAGTCTTTACTAATTAATTGTATTTGTTTAGCTCATTAATTT TAATTATTCCACATTTAAAGATGTCTAAGGAAGAAGCAGCAATCCCTCCTCCAATGATTT GGGCCCAGAGATCTGGTGTTGTCTTTTTAACAATTAATGTAGAGGATTGTAAAGACCCCG AAATTAAAATTGAAGAAGATAAATTTTCTTTTAAAAGTGTTGGTGGTGTTGAAAAGAAGA AATATGAAGTCACAGTAAATCTATTTAAAGAAATAGACCCAGAAAAATCTGTAAAACATG TTCGCGAACGACACATTGAGTTAGTCCTAAAAAAGAAAGAAGACAAAGCTCCTTACTGGC CACAATTGACGAAAGAAAAGACTAAGCACCATTGGTTAAAAGTGGATTTCAATAAGTGGA AGGATGAAGATGATAGCGAAGATGAAGCCGAAGGACAAGACTCAGATTTTGGTGATCTAA TGCGGTCGATGGGTCAAGGAGGCGGTATGGGTGGTATGGGCGGTATGGGTGGTATGGGAG GAATGGGAGGTATGGGTATGGGCGGTATGGGTGGTATGGGAATGGGTGGTTTAGGTGACA AGCCCTCTTTCGAAGGAATGGAAGAAGAAGATTCGGACGACGAAGATTTGCCCGACCTCG AAGAGTAATAGTGTTTTTATTACACCATATTCCATTTCCCTGTTATTGCATAAGGCCTCA GAAGAAGATGAAAAAATTGAAGCTATGAACGGACAGTCAAATCGATCACGCAGTTCACTG....55,000 more contigs Split up into ~1000 files of ~55 contigs each. Custom perl script or bioperl routines. contig-0.fsa... config-997.fsa

26 NGS WMS ngsui03.ngs.ac.uk ngswms01.ngs.ac.uk bdii.ngs.ac.ukuk myproxy.ngs.ac.uk RAL-NGS2, Scotgrid-Glasgow, Oxford-OERC, Manchester-NGS2 RAL-LCG....etc 12,000+ CPUs gridftp ssh Login MEG a.PhD@happyuser.ngs.ac.uk

27 Job types Single Job – Normal: simple batch job – MPICH: parallel jobs – Interactive: o/p streamed back to the client Parametric – Set of similar jobs whose jdl attributes are parameterised Collections – Group of jobs without dependencies DAG – Group of dependent jobs


Download ppt "Example applications of e-Infrastructure: NGS UI/WMS Jonathan Churchill - STFC/RAL"

Similar presentations


Ads by Google