Presentation is loading. Please wait.

Presentation is loading. Please wait.

Published byByron Robertson Modified over 8 years ago

Similar presentations

Presentation on theme: ""— Presentation transcript:

1 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de http://www.interactive-grid.eu

2 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de Grid Zugriff für USCT Marcus Hardt IWR

3 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de Grid middleware gLite  Middleware := Layer between application and operating system  gLite Development driven by CERN ● Tools for data+computing of new accelerator ● 10TB/year * 20years, random access  Paradigm: Send job to where the data is  Job: Self contained application  Building blocks integrate functionality: ● Monitoring ● Resource Brokerage ● Computer cluster ● Data storage

4 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de Grid middleware gLite

5 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de g gLite installations in Europe

6 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de Many grid projects  72 grid-related EU-projects  Why so many small grids? ● Own the infrastructure ● Know their owners ● Influence on development  FZK involvement: ● EGEE ● DGRID (Rainer Stotzka et.al.) ‏ ● Interactive European Grid Project = int.eu.grid = i2g

7 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de Interactive grid  2 Year Project (May'06 - April'08) ‏  ~20 people  Mission ● 100% gLite compatible extensions ● Bring grid to new user communities ● Improve useability  Application areas ● Fusion ● Medicine (USCT) ‏ ● Environment ● Astrophysics

8 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de Background 12 clusters in 7 countries ~ 785 CPUs & 40TB disk

9 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de What is interactivity?  1. Avoid queuing of important jobs  2. Online job steering  3. Interactive stdin / stdout transport  4. Direct (network) access to compute resources => This is my task

10 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de The application  The application: Ultrasound CT (USCT) ‏ ● New method for medical imaging ● Application: Breast cancer diagnosis USCT setup

11 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de USCT reconstruction := “Black Box”  Algorithm: ● Based on ellipsoidal backprojection (SAFT) ‏ ● Converts ultrasound signals to 3D volume graphics Input: ~ 20 GB Output: ~ 8 GB ● Computing time: 4096 2 (2D)/35 MB...128 2 x100/20 GB...4096 2 x 3410/20 GB 1hour...1.5 Months...150 Years  Matlab ● Problem solving environment similar to Maple, Mathematica, Scilab... ● Strategic development platform ● But: not possible to “submit matlab to the grid”

12 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de The USCT application at FZK  Computation takes long (days, weeks, years) ‏

13 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de The USCT application at FZK  Computation takes long (days, weeks, years) ‏  Grid in order to speed up  Goal: ● Seamless, interactive, grid access ● from Matlab

14 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de  Initial approach to parallel execution: ● Partitioning of data ● Many parallel jobs Using gLite

15 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de  Lets take a closer look Using gLite

16 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de  Using gLite  Goal: ● Seamless ● Interactive ● Grid access ● From matlab

17 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de What's missing?  Seamless ● Don't compile standalone application  Interactive ● No overhead (< 10 s) ‏ ● No manual data movement  From Matlab ● Run Matlab-functions remotely   Goal: ● Seamless ● Interactive ● Grid access ● From matlab

18 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de What's missing?  Seamless ● Don't compile standalone application  Interactive ● No overhead (< 10 s) ‏ ● No manual data movement  From Matlab ● Run Matlab-functions remotely   Goal: ● Seamless ● Interactive ● Grid access ● From matlab  Beispiel: Grosse Excel-Tabelle ● Excel soll lokal laufen ● Berechnungen im Grid

19 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de  GridSolve ● Client interface for Java, C, Fortran, Matlab, Octave ● Easy to use: y=problem(x) y=gs_call('problem', x)‏ Transport input parameters to remote side Execute “problem” Transport result back ● Implements an client/agent/server solution ● Developed at ICL, University Tennessee, Knoxville Improving grid access with GridSolve => Reduce complexity of the grid to one function call

20 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de How to do it?  1. Make Matlab run on gLite  2. Integrate GridSolve with gLite => G rid i n M atlab using G ridsolv e & R PC GIMGER

21 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de GridSolve (GS)/gLite integration  Create GS-service hosts (GS-agent) ‏  Send 100s of GS-servers to gLite infrastructure ‏ ● Setup build infrastructure ● Package GridSolve ● Create gLite jobs ● Install GridSolve on WorkerNodes (WN) ‏  Ensure network connectivity ● GS-client, GS-agent, GS-proxy, GS-server

22 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de GridSolve startup on gLite

23 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de GridSolve ready for action

24 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de GridSolve ready in action

25 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de GridSolve ready in action

26 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de Putting things together  GridSolve + gLite = GIMGER

27 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de Demonstration  Simulation: Mandelbrot fractal  Using the same infrastructure

28 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de  Movie of the life demonstration: ● http://marcus.hardt-it.de/grid4matlab  Life demo on int.eu.grid Life-Demo

29 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de Result  Simulation works  Reasonable speedup (4x on 8 machines) ‏

30 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de Source code

31 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de Summary  We can ● Use the grid from matlab...... for hand-tuned functions ● Run simple simulations in our infrastructure  We want to... ● Use real code Cope with the data (20 GB in, 8 GB out) ‏ ● Use gLite data handling methods Identify Bottlenecks ● Automatically send Matlab functions to the grid   Goal: ● Seamless ● Interactive ● Grid access ● From matlab

32 http://interactive-grid.eu Marcus.Hardt@iwr.fzk.de

Download ppt ""

Similar presentations

Challenges for Interactive Grids a point of view from Int.Eu.Grid project Remote Instrumentation Services in Grid Environment RISGE BoF Manchester 8th.

Challenges for Interactive Grids a point of view from Int.Eu.Grid project Remote Instrumentation Services in Grid Environment RISGE BoF Manchester 8th.
Samford University Virtual Supercomputer (SUVS) Brian Toone 4/14/09.

Samford University Virtual Supercomputer (SUVS) Brian Toone 4/14/09.

S. Gadomski, ATLAS computing in Geneva, journee de reflexion, 14 Sept 20071 ATLAS computing in Geneva Szymon Gadomski description of the hardware the.

S. Gadomski, "ATLAS computing in Geneva", journee de reflexion, 14 Sept ATLAS computing in Geneva Szymon Gadomski description of the hardware the.
O. Stézowski IPN Lyon AGATA Week 15-19 September 2003 Legnaro Data Analysis – Team #3 ROOT as a framework for AGATA.

O. Stézowski IPN Lyon AGATA Week September 2003 Legnaro Data Analysis – Team #3 ROOT as a framework for AGATA.
Www.uni-c.dk1 WWW.UNI-C.DK Computational Steering on the GRID Using a 3D model to Interact with a Large Scale Distributed Simulation in Real-Time Michael.

Computational Steering on the GRID Using a 3D model to Interact with a Large Scale Distributed Simulation in Real-Time Michael.
Software Frameworks for Acquisition and Control European PhD – 2009 Horácio Fernandes.

Software Frameworks for Acquisition and Control European PhD – 2009 Horácio Fernandes.
NetSolve Henri Casanova and Jack Dongarra University of Tennessee and Oak Ridge National Laboratory

NetSolve Henri Casanova and Jack Dongarra University of Tennessee and Oak Ridge National Laboratory
DataGrid Kimmo Soikkeli Ilkka Sormunen. What is DataGrid? DataGrid is a project that aims to enable access to geographically distributed computing power.

DataGrid Kimmo Soikkeli Ilkka Sormunen. What is DataGrid? DataGrid is a project that aims to enable access to geographically distributed computing power.
Member of the ExperTeam Group Ralf Ratering Pallas GmbH Hermülheimer Straße 10 50321 Brühl, Germany

Member of the ExperTeam Group Ralf Ratering Pallas GmbH Hermülheimer Straße Brühl, Germany
High Performance Computing (HPC) at Center for Information Communication and Technology in UTM.

High Performance Computing (HPC) at Center for Information Communication and Technology in UTM.
Enabling Grids for E-sciencE www.eu-egee.org Medical image processing web portal : Requirements analysis. An almost end user point of view … H. Benoit-Cattin,

Enabling Grids for E-sciencE Medical image processing web portal : Requirements analysis. An almost end user point of view … H. Benoit-Cattin,
EUROPEAN UNION Polish Infrastructure for Supporting Computational Science in the European Research Space Cracow Grid Workshop’10 Kraków, October 11-13,

EUROPEAN UNION Polish Infrastructure for Supporting Computational Science in the European Research Space Cracow Grid Workshop’10 Kraków, October 11-13,
JCE A Java-based Commissioning Environment tool Hiroyuki Sako, JAEA Hiroshi Ikeda, Visible Information Center Inc. SAD Workshop.

JCE A Java-based Commissioning Environment tool Hiroyuki Sako, JAEA Hiroshi Ikeda, Visible Information Center Inc. SAD Workshop.
Fabien Viale 1 Matlab & Scilab Applications to Finance Fabien Viale, Denis Caromel, et al. OASIS Team INRIA -- CNRS - I3S.

Fabien Viale 1 Matlab & Scilab Applications to Finance Fabien Viale, Denis Caromel, et al. OASIS Team INRIA -- CNRS - I3S.
SICSA student induction day, 2009Slide 1 Social Simulation Tutorial Session 6: Introduction to grids and cloud computing International Symposium on Grid.

SICSA student induction day, 2009Slide 1 Social Simulation Tutorial Session 6: Introduction to grids and cloud computing International Symposium on Grid.
DEVSView: A DEVS Visualization Tool Wilson Venhola.

DEVSView: A DEVS Visualization Tool Wilson Venhola.
Test Of Distributed Data Quality Monitoring Of CMS Tracker Dataset H->ZZ->2e2mu with PileUp - 10,000 events ( ~ 50,000 hits for events) The monitoring.

Test Of Distributed Data Quality Monitoring Of CMS Tracker Dataset H->ZZ->2e2mu with PileUp - 10,000 events ( ~ 50,000 hits for events) The monitoring.
Cluster currently consists of: 1 Dell PowerEdge 2950 3.6Ghz Dual, quad core Xeons (8 cores) and 16G of RAM Original GRIDVM - SL4 VM-Ware host 1 Dell PowerEdge.

Cluster currently consists of: 1 Dell PowerEdge Ghz Dual, quad core Xeons (8 cores) and 16G of RAM Original GRIDVM - SL4 VM-Ware host 1 Dell PowerEdge.

Similar presentations

Ads by Google