1 2010-04-27 EDGI European Desktop Grid Initiative Have you ever submitted 10.000 jobs to gLite in one run? If not, I will show.

Slides:



Advertisements
Similar presentations
FP7-INFRA Enabling Grids for E-sciencE EGEE Induction Grid training for users, Institute of Physics Belgrade, Serbia Sep. 19, 2008.
Advertisements

EDGI European Desktop Grid Initiative EDGI gUSE portal user guide EDGI is supported by the FP7 Capacities Programme under contract nr RI
EDGI European Desktop Grid Initiative gLite job submission to EDGI EDGI is supported by the FP7 Capacities Programme under contract nr RI
Development of test suites for the certification of EGEE-II Grid middleware Task 2: The development of testing procedures focused on special details of.
P-GRADE and WS-PGRADE portals supporting desktop grids and clouds Peter Kacsuk MTA SZTAKI
Special Jobs Claudio Cherubino INFN - Catania. 2 MPI jobs on gLite DAG Job Collection Parametric jobs Outline.
Basic Grid Job Submission Alessandra Forti 28 March 2006.
FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Special Jobs Matias Zabaljauregui UNLP.
Riccardo Bruno INFN.CT Sevilla, Sep 2007 The GENIUS Grid portal.
EDGI European Desktop Grid Initiative EDGI Portal usage and use-case studies EDGI is supported by the FP7 Capacities Programme under contract.
Example Gridification via command-line Application Developer Training Day IV. Miklos Kozlovszky Ankara, 25. October, 2007.
The gLite API – PART I Giuseppe LA ROCCA INFN Catania ACGRID-II School 2-14 November 2009 Kuala Lumpur - Malaysia.
GRID Computing: Ifrastructure, Development and Usage in Bulgaria M. Dechev, G. Petrov, E. Atanassov.
INFSO-RI Enabling Grids for E-sciencE Logging and Bookkeeping and Job Provenance Services Ludek Matyska (CESNET) on behalf of the.
The EDGI project receives Community research funding 1 EDGI Brings Desktop Grids To Distributed Computing Interoperability Etienne URBAH
SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI Creating the Autodock gateway from WS-PGRADE/gUSE and making it cloud-enabled.
1 IDGF International Desktop Grid Federation How can you benefit from joining IDGF? Hannover, Peter Kacsuk, MTA SZTAKI, EDGI.
INFSO-RI Enabling Grids for E-sciencE Workload Management System Mike Mineter
Sharing Workflows through Coarse-Grained Workflow Interoperability : Sharing Workflows through Coarse-Grained Workflow Interoperability G. Terstyanszky,
Group 1 : Grid Computing Laboratory of Information Technology Supervisors: Alexander Ujhinsky Nikolay Kutovskiy.
The EDGeS project receives Community research funding 1 SG-DG Bridges Zoltán Farkas, MTA SZTAKI.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Using gLite API Vladimir Dimitrov IPP-BAS “gLite middleware Application Developers.
Enabling Grids for E-sciencE EGEE-III INFSO-RI Using DIANE for astrophysics applications Ladislav Hluchy, Viet Tran Institute of Informatics Slovak.
GRID. Register Fill the form. Your IP (Juanjo) signature is needed and the one from the.
EGEE-III INFSO-RI Enabling Grids for E-sciencE Feb. 06, Introduction to High Performance and Grid Computing Faculty of Sciences,
Getting started DIRAC Project. Outline  DIRAC information system  Documentation sources  DIRAC users and groups  Registration with DIRAC  Getting.
Job Management DIRAC Project. Overview  DIRAC JDL  DIRAC Commands  Tutorial Exercises  What do you have learned? KEK 10/2012DIRAC Tutorial.
Alaa El-Deen shaban El-Meshneb Assistant Lecturer Physics Department Sohag University Grid Computing Group 3 D modeling in Grid (by the example of Blender)
INFSO-RI Enabling Grids for E-sciencE Claudio Cherubino, INFN Catania Grid Tutorial for users Merida, April 2006 Special jobs.
INFSO-RI Enabling Grids for E-sciencE Αthanasia Asiki Computing Systems Laboratory, National Technical.
Enabling Grids for E-sciencE Workload Management System on gLite middleware - commands Matthieu Reichstadt CNRS/IN2P3 ACGRID School, Hanoi.
High-Performance Computing Lab Overview: Job Submission in EDG & Globus November 2002 Wei Xing.
EGEE-0 / LCG-2 middleware Practical.
1 DIRAC Job submission A.Tsaregorodtsev, CPPM, Marseille LHCb-ATLAS GANGA Workshop, 21 April 2004.
SHIWA and Coarse-grained Workflow Interoperability Gabor Terstyanszky, University of Westminster Summer School Budapest July 2012 SHIWA is supported.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”
INFSO-RI Enabling Grids for E-sciencE EGEE is a project funded by the European Union under contract IST Job sandboxes.
11 Introduction to EDGI Peter Kacsuk, MTA SZTAKI Start date: Duration: 27 months EDGI.
INFSO-RI Enabling Grids for E-sciencE GILDA Praticals Giuseppe La Rocca INFN – Catania gLite Tutorial at the EGEE User Forum CERN.
EGEE is a project funded by the European Union under contract IST Enabling bioinformatics applications to.
User Interface UI TP: UI User Interface installation & configuration.
SHIWA Simulation Platform (SSP) Gabor Terstyanszky, University of Westminster EGI Community Forum Munnich March 2012 SHIWA is supported by the FP7.
11 The European Desktop Grid Federation: status of the infrastructure and integration plans Peter Kacsuk, Jozsef Kovacs, and Robert Lovas (MTA SZTAKI)
1 International Desktop Grid Federation at the fingertips of EGI scientists Revised version for ‘Promoting Desktop Grids’ virtual team status report Robert.
RI EGI-TF 2010, Tutorial Managing an EGEE/EGI Virtual Organisation (VO) with EDGES bridged Desktop Resources Tutorial Robert Lovas, MTA SZTAKI.
11 Extending EMI middleware with DGs Peter Kacsuk, MTA SZTAKI Start date: Duration:
1 Globe adapted from wikipedia/commons/f/fa/ Globe.svg IDGF-SP International Desktop Grid Federation - Support Project SZTAKI.
Job Management Beijing, 13-15/11/2013. Overview Beijing, /11/2013 DIRAC Tutorial2  DIRAC JDL  DIRAC Commands  Tutorial Exercises  What do you.
InSilicoLab – Grid Environment for Supporting Numerical Experiments in Chemistry Joanna Kocot, Daniel Harężlak, Klemens Noga, Mariusz Sterzel, Tomasz Szepieniec.
Introduction to Computing Element HsiKai Wang Academia Sinica Grid Computing Center, Taiwan.
Antonio Fuentes RedIRIS Barcelona, 15 Abril 2008 The GENIUS Grid portal.
Enabling Grids for E-sciencE Work Load Management & Simple Job Submission Practical Shu-Ting Liao APROC, ASGC EGEE Tutorial.
SEE-GRID-SCI WRF-ARW model: Grid usage The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI solution for high throughput data analysis Peter Solagna EGI.eu Operations.
EGI-InSPIRE RI EGI Hands On Training for AEGIS Users EGI-InSPIRE N G I A E G I S EGI Hands On Training for AEGIS Users Institute of Physics.
SHIWA SIMULATION PLATFORM = SSP Gabor Terstyanszky, University of Westminster e-Science Workflows Workshop Budapest 09 nd February 2012 SHIWA is supported.
The EDGI (European Desktop Grid Initiative) infrastructure and its usage for the European Grid user communities József Kovács (MTA SZTAKI)
How to connect your DG to EDGeS? Zoltán Farkas, MTA SZTAKI
and Alexandre Duarte OurGrid/EELA Interoperability Meeting
Corso di Calcolo Parallelo Grid Computing
Alexandre Duarte CERN Fifth EELA Tutorial Santiago, 06/09-07/09,2006
Introduction to Grid Technology
WS-PGRADE for Molecular Sciences and XSEDE
5. Job Submission Grid Computing.
Pablo Saiz CAF and Grid User Forum
Initial job submission and monitoring efforts with JClarens
gLite Job Management Christos Theodosiou
DEGISCO project - Desktop Grids for application developers and users
GENIUS Grid portal Hands on
Presentation transcript:

EDGI European Desktop Grid Initiative Have you ever submitted jobs to gLite in one run? If not, I will show you how I did it…. Demonstration on the EGI User Forum Vilnius, Lithuania 12/04/2011 EDGI is supported by the FP7 Capacities Programme under contract nr RI

Here is the receipt for you…. Step 1: Choose an application you want to run Step 1: Choose an application you want to run Step 2: EDGI project performs the necessary preparations for you => application appears in the EDGI application repository Step 2: EDGI project performs the necessary preparations for you => application appears in the EDGI application repository Step 3: Collect info from the EDGI AR Step 3: Collect info from the EDGI AR Step 4: Prepare your inputs, create and submit jdl Step 4: Prepare your inputs, create and submit jdl Step 5: Query the status and download the outputs Step 5: Query the status and download the outputs Be happy with the results! Be happy with the results! Need more details? Let us see… Need more details? Let us see… 2

Step 3: Collect information from EDGI AR (authorized vos, ces) 3

Step 3: Collect information from the EDGI AR (files) 4

Step 4a: prepare your inputs Upload your individual input files to web server: Upload your individual input files to web server: … Create the description of your metajob: Create the description of your metajob: %Required 100% %SuccessAt 100% %Comment pools1.txt Arguments = "-i 0 -n pools.txt -f 22 -p 723“ Input = pools.txt= Queue[….] %Comment pools10000.txt Input = pools.txt= Queue 5

Step 4b: create and submit jdl Create your jdl: Create your jdl: Executable = "dsp"; Arguments = "-f 22 -i 22 -p 723 -n pools.txt"; InputSandbox = { "gsiftp://dev17-portal.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp", "pools.txt", "_3gb-metajob-dsp-10000" }; OutputSandbox = {"cost.txt"}; SubmitTo = "cr1.edgi-grid.eu:8443/cream-pbs-edgidemo"; Submit: Submit: glite-wms-job-submit -o id my-metajob jdl 6 Reference to the app in the EDGI AR Basic input file Extra input file

Step 5a: Query the status 7 Location of the status description

Step 5b: Query the status 8 Metajob handling rules Actual status of jobs Location of the mapping

Step 5b: Download the results 9 Download result as usual: glite-wms-job-output -i id --dir outputs =>./outputs/cost.txt Extract it: tar zxvf cost.txt =>./outputs/ /cost.txt See the mapping between your individual job definition and the jobids (which gives the name of directories storing the output files of your app) => one subjob id one job

And now, Step 1 and 2: Contact the EDGI project. The project will Contact the EDGI project. The project will Analyse and port your application to the EDGI infrastructure with your assistance Analyse and port your application to the EDGI infrastructure with your assistance Register the ported application on one or several EDGI DGs Register the ported application on one or several EDGI DGs Upload and configure your application in the EDGI AR Upload and configure your application in the EDGI AR Give access to you for using the EDGI services Give access to you for using the EDGI services 10

Volunteer/ Institutional Resources ARC grid gLite grid Eucalyptus/ Amazon What is behind: The EDGI infrastructure 11 ARC MCE atticmonitorAR CREAM MCE atticmonitorAR 3GBridge attic monitor AR User IF Bridge IF Attic FS DG client attic Monitor UI DG Pro- ject submit inspect upload down- load submit cloud DG client attic Demonstrated!

And finally here comes the live demonstration… 12