Miron Livny Computer Sciences Department University of Wisconsin-Madison Harnessing the Capacity of Computational.

Slides:

Advertisements

Similar presentations

Condor Project Computer Sciences Department University of Wisconsin-Madison Introduction Condor.

Advertisements

Jaime Frey Computer Sciences Department University of Wisconsin-Madison OGF 19 Condor Software Forum Routing.

Dan Bradley Computer Sciences Department University of Wisconsin-Madison Schedd On The Side.

Condor-G: A Computation Management Agent for Multi-Institutional Grids James Frey, Todd Tannenbaum, Miron Livny, Ian Foster, Steven Tuecke Reporter: Fu-Jiun.

A Computation Management Agent for Multi-Institutional Grids

Workload Management Massimo Sgaravatto INFN Padova.

The Condor Data Access Framework GridFTP / NeST Day 31 July 2001 Douglas Thain.

The Difficulties of Distributed Data Douglas Thain Condor Project University of Wisconsin

Condor Overview Bill Hoagland. Condor Workload management system for compute-intensive jobs Harnesses collection of dedicated or non-dedicated hardware.

Evaluation of the Globus GRAM Service Massimo Sgaravatto INFN Padova.

Jim Basney Computer Sciences Department University of Wisconsin-Madison Managing Network Resources in.

Jaeyoung Yoon Computer Sciences Department University of Wisconsin-Madison Virtual Machines in Condor.

Miron Livny Computer Sciences Department University of Wisconsin-Madison From Compute Intensive to Data.

Zach Miller Computer Sciences Department University of Wisconsin-Madison What’s New in Condor.

Alain Roy Computer Sciences Department University of Wisconsin-Madison An Introduction To Condor International.

Distributed Systems Early Examples. Projects NOW – a Network Of Workstations University of California, Berkely Terminated about 1997 after demonstrating.

Track 1: Cluster and Grid Computing NBCR Summer Institute Session 2.2: Cluster and Grid Computing: Case studies Condor introduction August 9, 2006 Nadya.

History of the National INFN Pool P. Mazzanti, F. Semeria INFN – Bologna (Italy) European Condor Week 2006 Milan, 29-Jun-2006.

Condor Tugba Taskaya-Temizel 6 March What is Condor Technology? Condor is a high-throughput distributed batch computing system that provides facilities.

Central Reconstruction System on the RHIC Linux Farm in Brookhaven Laboratory HEPIX - BNL October 19, 2004 Tomasz Wlodek - BNL.

Grid Computing I CONDOR.

Grid Workload Management & Condor Massimo Sgaravatto INFN Padova.

1 The Roadmap to New Releases Todd Tannenbaum Department of Computer Sciences University of Wisconsin-Madison

Miron Livny Computer Sciences Department University of Wisconsin-Madison Welcome and Condor Project Overview.

Miron Livny Computer Sciences Department University of Wisconsin-Madison Condor : A Concept, A Tool and.

Condor: High-throughput Computing From Clusters to Grid Computing P. Kacsuk – M. Livny MTA SYTAKI – Univ. of Wisconsin-Madison

NGS Innovation Forum, Manchester4 th November 2008 Condor and the NGS John Kewley NGS Support Centre Manager.

Alain Roy Computer Sciences Department University of Wisconsin-Madison ClassAds: Present and Future.

STORK: Making Data Placement a First Class Citizen in the Grid Tevfik Kosar and Miron Livny University of Wisconsin-Madison March 25 th, 2004 Tokyo, Japan.

Douglas Thain, John Bent Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, Miron Livny Computer Sciences Department, UW-Madison Gathering at the Well: Creating.

The Roadmap to New Releases Derek Wright Computer Sciences Department University of Wisconsin-Madison

What is SAM-Grid? Job Handling Data Handling Monitoring and Information.

Tevfik Kosar Computer Sciences Department University of Wisconsin-Madison Managing and Scheduling Data.

Review of Condor,SGE,LSF,PBS

STORK: Making Data Placement a First Class Citizen in the Grid Tevfik Kosar University of Wisconsin-Madison May 25 th, 2004 CERN.

Campus grids: e-Infrastructure within a University Mike Mineter National e-Science Centre 14 February 2006.

Automatic Statistical Evaluation of Resources for Condor Daniel Nurmi, John Brevik, Rich Wolski University of California, Santa Barbara.

HIGUCHI Takeo Department of Physics, Faulty of Science, University of Tokyo Representing dBASF Development Team BELLE/CHEP20001 Distributed BELLE Analysis.

Peter Couvares Associate Researcher, Condor Team Computer Sciences Department University of Wisconsin-Madison

Nick LeRoy Computer Sciences Department University of Wisconsin-Madison Hawkeye.

Scheduling & Resource Management in Distributed Systems Rajesh Rajamani, May 2001.

Condor on WAN D. Bortolotti - INFN Bologna T. Ferrari - INFN Cnaf A.Ghiselli - INFN Cnaf P.Mazzanti - INFN Bologna F. Prelz - INFN Milano F.Semeria - INFN.

Nicholas Coleman Computer Sciences Department University of Wisconsin-Madison Distributed Policy Management.

Condor Services for the Global Grid: Interoperability between OGSA and Condor Clovis Chapman 1, Paul Wilson 2, Todd Tannenbaum 3, Matthew Farrellee 3,

Jaime Frey Computer Sciences Department University of Wisconsin-Madison What’s New in Condor-G.

Miron Livny Computer Sciences Department University of Wisconsin-Madison Condor and (the) Grid (one of.

Managing Network Resources in Condor Jim Basney Computer Sciences Department University of Wisconsin-Madison

Matthew Farrellee Computer Sciences Department University of Wisconsin-Madison Condor and Web Services.

Parag Mhashilkar Computing Division, Fermi National Accelerator Laboratory.

Douglas Thain, John Bent Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, Miron Livny Computer Sciences Department, UW-Madison Gathering at the Well: Creating.

4/9/ 2000 I Datagrid Workshop- Marseille C.Vistoli Wide Area Workload Management Work Package DATAGRID project Parallel session report Cristina Vistoli.

April 25, 2006Parag Mhashilkar, Fermilab1 Resource Selection in OSG & SAM-On-The-Fly Parag Mhashilkar Fermi National Accelerator Laboratory Condor Week.

NeST: Network Storage John Bent, Venkateshwaran V Miron Livny, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau.

Todd Tannenbaum Computer Sciences Department University of Wisconsin-Madison Condor NT Condor ported.

Jaime Frey Computer Sciences Department University of Wisconsin-Madison Condor and Virtual Machines.

George Kola Computer Sciences Department University of Wisconsin-Madison Data Pipelines: Real Life Fully.

Condor Project Computer Sciences Department University of Wisconsin-Madison Condor Introduction.

Condor on Dedicated Clusters Peter Couvares and Derek Wright Computer Sciences Department University of Wisconsin-Madison

Condor A New PACI Partner Opportunity Miron Livny

Condor – A Hunter of Idle Workstation

University of Technology

NTC 324 RANK Education for Service-- ntc324rank.com.

Pegasus and Condor Gaurang Mehta, Ewa Deelman, Carl Kesselman, Karan Vahi Center For Grid Technologies USC/ISI.

Welcome to HTCondor Week #18 (year 33 of our project)

Condor and Multi-core Scheduling

Basic Grid Projects – Condor (Part I)

Genre1: Condor Grid: CSECCR

Website Testing Checklist

Condor-G Making Condor Grid Enabled

GLOW A Campus Grid within OSG

Presentation transcript:

Miron Livny Computer Sciences Department University of Wisconsin-Madison Harnessing the Capacity of Computational Grids for High Energy Physics Jim Basney, Miron Livny, Paolo Mazzanti

Background This work is the result of an ongoing collaboration between the Condor Team at the University of Wisconsin Madison and the Bologna section of INFN  Collaboration started in 1996  An INFN Condor pool with more than 170 CPUs is serving the INFN community ( )  New features were developed and tested as a result of this collaboration

Data Transfer Challenge In order to harness for HEP the processing capacity of large collections of commodity (desktop and clusters) computing resources we need effective mechanisms and policies to manage the transfer and placement of checkpoint and data files and means to established affinity between execution sites and data storage sites.

Need to take into account › Network topology and capabilities › Distribution, capabilities and availability of storage resources › Distribution, capacity and availability of computing resources › Impact on interactive users

The Condor HTC System Condor is a distributed job and resource management system that employs a novel matchmaking approach to allocate resources to jobs.  Symmetric - Requests and Offers  Open - No centralized schema  Dynamic - Easy to change information and semantics  Expressive - Full power of Boolean expressions

ClassAd examples Resource Offer [ OpSys = "Solaris2.6"; Arch = "Sun4u"; Memory = 256; LoadAvg = 0.25; Cluster = "UWCS"; Requirements = My.LoadAvg < 0.3 Rank = (Target.Group == "AI”); ] Resource Request [ Group= ”AI"; Requirements = Target.Memory > 80 && Target.OpSys == "Solaris2.6" && Target.Arch == "Sun4u"; Rank = (Target.Cluster == "UWCS”); ]

Checkpoint Domains › Every Computational resource belongs to a checkpoint domain › Jobs can start on any resource › Checkpoint is saved to the local (domain) checkpoint server › Jobs are restarted only on local (domain) computational resources › Checkpoints can migrate

I/O Domains › Each resource belongs to an I/O domain. A domain may consist of a single machine. › User stages input data on storage devices and updates the ClassAds of the jobs and/or the resources to reflect the location and availability of the data. › User is responsible for moving output data to storage system › Condor monitors and reports I/O activity performed via remote I/O.

Ongoing I/O related work › Improve performance and mapping capabilities of Remote I/O capabilities of Condor. › Provide interfaces to SRB (SDSC), SAM (FERMI) and CORBA (LBL) data storage systems. › Support co-scheduling of processing and network resources › Develop staging services and interface them with the matchmaking frame work. › Extend reporting and monitoring capabilities

Visit us at