Presentation is loading. Please wait.

Presentation is loading. Please wait.

What’s New in Work Queue

Published byYanti Gunawan Modified over 5 years ago

Similar presentations

Presentation on theme: "What’s New in Work Queue"— Presentation transcript:

1 What’s New in Work Queue
Michael Albrecht, University of Notre Dame CCL Workshop, June 2012

2 Overview New “Batch Job” Systems Work Queue Enhancements
Moab / Cluster MPI Queue Work Queue Enhancements Hierarchical Work Queue

3 Batch Job Systems Abstraction Layer
Library for generic task submission Used by Makeflow

4 Moab & other Clusters Similar to “SGE”
Explicitly support Moab scheduler Support other, similar schedulers Set name, submit and remove commands moab

5 MPI Queue Workflow W W W W W W W W W W W W W W

6 MPI Queue Workflow W W W W F W W W W W

7 MPI Queue Enables arbitrary computation on “MPI-only” clusters
Ranks 1-N talk to Rank 0, which acts as a “foreman” Assumes shared parallel filesystem for cluster API very similar to Work Queue Fully supported by Batch Job/Makeflow

8 Work Queue

9 Work Queue is Wonderful
Easily harness 100’s-1000’s of cores Combine multiple resources for one project Dynamically scale computational resources Private Cluster Campus Condor Pool Public Cloud Provider Shared SGE Makefile Makeflow Local Files and Programs sge_submit_workers W ssh Wv condor_submit_workers Hundreds of Workers in a Personal Cloud submit tasks

10 Work Queue has Limits Bandwidth, file size, and computation length constrain potential number of workers

11 Work Queue has Limits T0 ??? T1 T2 T3 T4 W1 W2 W3 W4 W5 T5 Natural parallelism of a workflow leaves extra workers idle

12 Work Queue has Limits Synchronous transfer leaves network resources idle, increases dispatch time

13 “…except for the problem of too many layers of indirection”
Add more indirection! “All problems in computer science can be solved by another layer of indirection” -David Wheeler “…except for the problem of too many layers of indirection” Kevlin Henney

14 Hierarchical Work Queue
Master (Makeflow) Worker Worker Worker Worker Worker Worker Worker Worker

15 Hierarchical Work Queue
Master (Makeflow) Foreman Foreman Worker Worker Worker Worker Worker Worker Worker Worker

16 Hierarchical Work Queue
Master (Makeflow) Shared FS Foreman Foreman Worker Worker Worker Worker Worker Worker Worker Worker

17 Hierarchical Work Queue
Master (Makeflow) Shared FS Shared FS Foreman Foreman Worker Worker Worker Worker Worker Worker Worker Worker

18 Hierarchical Work Queue
Master (Makeflow) Shared FS Foreman Foreman Shared FS Shared FS Foreman Foreman Worker Worker Worker Worker Worker Worker Worker Worker

19 Hierarchical Work Queue
Coming Soon! 19

Download ppt "What’s New in Work Queue"

Similar presentations

1 Real-World Barriers to Scaling Up Scientific Applications Douglas Thain University of Notre Dame Trends in HPDC Workshop Vrije University, March 2012.

1 Real-World Barriers to Scaling Up Scientific Applications Douglas Thain University of Notre Dame Trends in HPDC Workshop Vrije University, March 2012.
Dan Bradley Computer Sciences Department University of Wisconsin-Madison Schedd On The Side.

Dan Bradley Computer Sciences Department University of Wisconsin-Madison Schedd On The Side.
Experience with Adopting Clouds at Notre Dame Douglas Thain University of Notre Dame IEEE CloudCom, November 2010.

Experience with Adopting Clouds at Notre Dame Douglas Thain University of Notre Dame IEEE CloudCom, November 2010.
Setting up of condor scheduler on computing cluster Raman Sehgal NPD-BARC.

Setting up of condor scheduler on computing cluster Raman Sehgal NPD-BARC.
Introduction to Scalable Programming using Makeflow and Work Queue Dinesh Rajan and Mike Albrecht University of Notre Dame October 24 and November 7, 2012.

Introduction to Scalable Programming using Makeflow and Work Queue Dinesh Rajan and Mike Albrecht University of Notre Dame October 24 and November 7, 2012.
1 Condor Compatible Tools for Data Intensive Computing Douglas Thain University of Notre Dame Condor Week 2011.

1 Condor Compatible Tools for Data Intensive Computing Douglas Thain University of Notre Dame Condor Week 2011.
1 Opportunities and Dangers in Large Scale Data Intensive Computing Douglas Thain University of Notre Dame Large Scale Data Mining Workshop at SIGKDD August.

1 Opportunities and Dangers in Large Scale Data Intensive Computing Douglas Thain University of Notre Dame Large Scale Data Mining Workshop at SIGKDD August.
1 Models and Frameworks for Data Intensive Cloud Computing Douglas Thain University of Notre Dame IDGA Cloud Computing 8 February 2011.

1 Models and Frameworks for Data Intensive Cloud Computing Douglas Thain University of Notre Dame IDGA Cloud Computing 8 February 2011.
A Grid Parallel Application Framework Jeremy Villalobos PhD student Department of Computer Science University of North Carolina Charlotte.

A Grid Parallel Application Framework Jeremy Villalobos PhD student Department of Computer Science University of North Carolina Charlotte.
6/2/20071 Grid Computing Sun Grid Engine (SGE) Manoj Katwal.

6/2/20071 Grid Computing Sun Grid Engine (SGE) Manoj Katwal.
Introduction to Makeflow Li Yu University of Notre Dame 1.

Introduction to Makeflow Li Yu University of Notre Dame 1.
Condor Overview Bill Hoagland. Condor Workload management system for compute-intensive jobs Harnesses collection of dedicated or non-dedicated hardware.

Condor Overview Bill Hoagland. Condor Workload management system for compute-intensive jobs Harnesses collection of dedicated or non-dedicated hardware.
Building Scalable Elastic Applications using Makeflow Dinesh Rajan and Douglas Thain University of Notre Dame Tutorial at CCGrid, May 13 2013 Delft, Netherlands.

Building Scalable Elastic Applications using Makeflow Dinesh Rajan and Douglas Thain University of Notre Dame Tutorial at CCGrid, May Delft, Netherlands.
Building Scalable Scientific Applications using Makeflow Dinesh Rajan and Peter Sempolinski University of Notre Dame.

Building Scalable Scientific Applications using Makeflow Dinesh Rajan and Peter Sempolinski University of Notre Dame.
Building Scalable Applications on the Cloud with Makeflow and Work Queue Douglas Thain and Patrick Donnelly University of Notre Dame Science Cloud Summer.

Building Scalable Applications on the Cloud with Makeflow and Work Queue Douglas Thain and Patrick Donnelly University of Notre Dame Science Cloud Summer.
Introduction to Makeflow and Work Queue CSE 40822 – Cloud Computing – Fall 2014 Prof. Douglas Thain.

Introduction to Makeflow and Work Queue CSE – Cloud Computing – Fall 2014 Prof. Douglas Thain.
ADLB Update Recent and Current Adventures with the Asynchronous Dynamic Load Balancing Library Rusty Lusk Mathematics and Computer Science Division Argonne.

ADLB Update Recent and Current Adventures with the Asynchronous Dynamic Load Balancing Library Rusty Lusk Mathematics and Computer Science Division Argonne.
SCD FIFE Workshop - GlideinWMS Overview GlideinWMS Overview FIFE Workshop (June 04, 2013) - Parag Mhashilkar Why GlideinWMS? GlideinWMS Architecture Summary.

SCD FIFE Workshop - GlideinWMS Overview GlideinWMS Overview FIFE Workshop (June 04, 2013) - Parag Mhashilkar Why GlideinWMS? GlideinWMS Architecture Summary.
Design of an Active Storage Cluster File System for DAG Workflows Patrick Donnelly and Douglas Thain University of Notre Dame 2013 November 18 th DISCS-2013.

Design of an Active Storage Cluster File System for DAG Workflows Patrick Donnelly and Douglas Thain University of Notre Dame 2013 November 18 th DISCS-2013.
Parallelization with the Matlab® Distributed Computing Server (MDCS) @ CBI cluster December 3, 2013 - Matlab Parallelization with the Matlab Distributed.

Parallelization with the Matlab® Distributed Computing Server CBI cluster December 3, Matlab Parallelization with the Matlab Distributed.

Similar presentations

Ads by Google