Is 99% Utilization of a Supercomputer a Good Thing? Scheduling in Context: User Utility Functions Cynthia Bailey Lee Department of Computer Science and.

Slides:

Advertisements

Similar presentations

Evaluating the Cost-Benefit of Using Cloud Computing to Extend the Capacity of Clusters Presenter: Xiaoyu Sun.

Advertisements

Studies of the User-Scheduler Relationship Cynthia Bailey Lee Advisor: Allan E. Snavely Department of Computer Science and Engineering San Diego Supercomputer.

Explicit Preemption Placement for Real- Time Conditional Code via Graph Grammars and Dynamic Programming Bo Peng, Nathan Fisher, and Marko Bertogna Department.

Towards Provision of Quality of Service Guarantees in Job Scheduling Mohammad IslamPavan Balaji P. SadayappanD. K. Panda Computer Science and Engineering.

1 Advancing Supercomputer Performance Through Interconnection Topology Synthesis Yi Zhu, Michael Taylor, Scott B. Baden and Chung-Kuan Cheng Department.

UCSD SAN DIEGO SUPERCOMPUTER CENTER 1 Symbiotic Space-Sharing: Mitigating Resource Contention on SMP Systems Professor Snavely, University of California.

Scheduling of parallel jobs in a heterogeneous grid environment Scheduling of parallel jobs in a heterogeneous grid environment Each site has a homogeneous.

The Forgotten Factor: FACTS on Performance Evaluation and its Dependence on Workloads Dror Feitelson Hebrew University.

Senior Design Project: Parallel Task Scheduling in Heterogeneous Computing Environments Senior Design Students: Christopher Blandin and Dylan Machovec.

CPU Scheduling Algorithms

Parallel Job Scheduling Algorithms and Interfaces Research Exam for Cynthia Bailey Lee Department of Computer Science and Engineering University of California,

Quality-Aware Segment Transmission Scheduling in Peer-to-Peer Streaming Systems Cheng-Hsin Hsu Senior Research Scientist Deutsche Telekom R&D Lab USA Los.

On Economics and the User-Scheduler Relationship in HPC and Grid Systems Cynthia Bailey Lee Advisor: Allan E. Snavely Department of Computer Science and.

Integrated Risk Analysis for a Commercial Computing Service Chee Shin Yeo and Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab. Dept.

Embedded System Design Framework for Minimizing Code Size and Guaranteeing Real-Time Requirements Insik Shin, Insup Lee, & Sang Lyul Min CIS, Penn, USACSE,

Computer Architecture and Operating Systems CS 3230: Operating System Section Lecture OS-3 CPU Scheduling Department of Computer Science and Software Engineering.

COT 4600 Operating Systems Spring 2011 Dan C. Marinescu Office: HEC 304 Office hours: Tu-Th 5:00-6:00 PM.

1 TUD-PDS A Periodic Portfolio Scheduler for Scientific Computing in the Data Center Kefeng Deng, Ruben Verboon, Kaijun Ren, and Alexandru Iosup Parallel.

Suggesting Friends using the Implicit Social Graph Maayan Roth et al. (Google, Inc., Israel R&D Center) KDD’10 Hyewon Lim 1 Oct 2014.

Alexandre A. P. Suaide VI DOSAR workshop, São Paulo, 2005 STAR grid activities and São Paulo experience.

Marcos Dias de Assunção 1,2, Alexandre di Costanzo 1 and Rajkumar Buyya 1 1 Department of Computer Science and Software Engineering 2 National ICT Australia.

Resource Provisioning based on Lease Preemption in InterGrid Mohsen Amini Salehi, Bahman Javadi, Rajkumar Buyya Cloud Computing and Distributed Systems.

Meta Scheduling Sathish Vadhiyar Sources/Credits/Taken from: Papers listed in “References” slide.

1 520 Student Presentation GridSim – Grid Modeling and Simulation Toolkit.

Scheduling. Alternating Sequence of CPU And I/O Bursts.

INTERNATIONAL CONFERENCE ON INFORMATION SCIENCE AND TECHNOLOGY, P.P , MARCH An ANFIS-based Dispatching Rule For Complex Fuzzy Job Shop Scheduling.

Alternating Sequence of CPU And I/O Bursts. Histogram of CPU-burst Times.

Predicting Queue Waiting Time in Batch Controlled Systems Rich Wolski, Dan Nurmi, John Brevik, Graziano Obertelli Computer Science Department University.

Scheduling in HPC Resource Management System: Queuing vs. Planning Matthias Hovestadt, Odej Kao, Alex Keller, and Achim Streit 2003 Job Scheduling Strategies.

Service-oriented Resource Broker for QoS-Guaranteed in Grid Computing System Yichao Yang, Jin Wu, Lei Lang, Yanbo Zhou and Zhili Sun Centre for communication.

CSCI1600: Embedded and Real Time Software Lecture 19: Queuing Theory Steven Reiss, Fall 2015.

Project Design Alain Esteva-Ramirez School of Computing and Information Sciences Florida International University Bárbara Morales-Quiñones Department of.

Performance Analysis of Preemption-aware Scheduling in Multi-Cluster Grid Environments Mohsen Amini Salehi, Bahman Javadi, Rajkumar Buyya Cloud Computing.

Job Scheduling P. (Saday) Sadayappan Ohio State University.

QoPS: A QoS based Scheme for Parallel Job Scheduling M. IslamP. Balaji P. Sadayappan and D. K. Panda Computer and Information Science The Ohio State University.

Scheduling MPI Workflow Applications on Computing Grids Juemin Zhang, Waleed Meleis, and David Kaeli Electrical and Computer Engineering Department, Northeastern.

Author Utility-Based Scheduling for Bulk Data Transfers between Distributed Computing Facilities Xin Wang, Wei Tang, Raj Kettimuthu,

Efficient Point Coverage in Wireless Sensor Networks Jie Wang and Ning Zhong Department of Computer Science University of Massachusetts Journal of Combinatorial.

Chapter 4 CPU Scheduling. 2 Basic Concepts Scheduling Criteria Scheduling Algorithms Multiple-Processor Scheduling Real-Time Scheduling Algorithm Evaluation.

Lecture 5 Scheduling. Today CPSC Tyson Kendon Updates Assignment 1 Assignment 2 Concept Review Scheduling Processes Concepts Algorithms.

CPU Scheduling Algorithms CSSE 332 Operating Systems Rose-Hulman Institute of Technology.

Event Based Simulation of The Backfilling Algorithm OOP tirgul No

Lizhe Wang, Gregor von Laszewski, Jai Dayal, Thomas R. Furlani

Parallel Job Scheduling Algorithms and Interfaces

CPU SCHEDULING.

Dan C. Marinescu Office: HEC 439 B. Office hours: M, Wd 3 – 4:30 PM.

Copyright ©: Nahrstedt, Angrave, Abdelzaher

CPU scheduling 6. Schedulers, CPU Scheduling 6.1. Schedulers

Cross layer design is wireless multi-hop network

Scheduling (Priority Based)

CPSC 457 Operating Systems

Chapter 6: CPU Scheduling

ICS 143 Principles of Operating Systems

CS 143A - Principles of Operating Systems

CPU Scheduling G.Anuradha

Module 5: CPU Scheduling

Operating Systems Lecture 15.

3: CPU Scheduling Basic Concepts Scheduling Criteria

Chapter5: CPU Scheduling

COT 4600 Operating Systems Spring 2011

Chapter 6: CPU Scheduling

A Characterization of Approaches to Parrallel Job Scheduling

Chapter 5: CPU Scheduling

Lecture 2 Part 3 CPU Scheduling

Hawk: Hybrid Datacenter Scheduling

Operating System , Fall 2000 EA101 W 9:00-10:00 F 9:00-11:00

Chapter 6: CPU Scheduling

Module 5: CPU Scheduling

Chapter 6: CPU Scheduling

Module 5: CPU Scheduling

Presentation transcript:

Is 99% Utilization of a Supercomputer a Good Thing? Scheduling in Context: User Utility Functions Cynthia Bailey Lee Department of Computer Science and Engineering and San Diego Supercomputer Center University of California, San Diego November 13, 2006

What is a Utility Function? Outline: What is a Utility Function? Metrics Metric Pitfalls 8 am 12–1pm 5 pm 8 am 9 am time u(t) Other factors: coordinate with other grid sites or sensors, paper deadlines, weather and hurricane prediction, …

Some Proposed Models What is a Utility Function? Proposed Models Real Functions Scheduling LINEAR DECAY Customizable start value and slope [Chun et al. 2002, Irwin et al. 2004] Common metrics such wait time and expansion factor implicitly use a (non- customizable) linear decay model.  EXPONENTIAL DECAY Customizable start value PIECEWISE LINEAR DECAY User customizes all points My favorite

Real Functions What is a Utility Function? Proposed Models Real Functions Scheduling Randomly selected SDSC users provided these functions for jobs they were submitting Utility is in terms of the SDSC charge unit (“SU”)

More Real Functions What is a Utility Function? Proposed Models Real Functions Scheduling

Parallel Job Scheduling Explicitly by Utility Function What is a Utility Function? Proposed Models Real Functions Scheduling Finding the best solution is NP-hard  “Tennis Court Scheduling” (Human-powered scheduling) Still practiced occasionally at most centers (officially and not) -- a phone call to sysadmins gets a job a reservation or to the front of the queue) Custom Heuristics Sort by current value, a combination of start value and slope [Chun and Culler 02, Irwin, Grit, Chase 04] Linear Solver / CPLEX

Comparing Schedulers What is a Utility Function? Proposed Models Real Functions Scheduling Expansion Factor 25/50/75 %ile Aggregate Utility rounded, x10^7 Time to Solution* EASY 0/0/ s Conservative Backfilling 0/82/ s Priority Queues + EASY 0/2883/ s Genetic Algorithm 0/289/ s * For 500 scheduling rounds (need to do a round when a job arrives or ends). All timings done with a (not particularly optimized) modular scheduling simulator framework in Python ( me for code). Expansion Factor 25/50/75 %ile Aggregate Utility rounded, x10^7 EASY 0/0/ Conservative Backfilling 0/82/ Priority Queues + EASY 0/2883/

Questions For more information, see paper: –Gathering real users’ utility functions: Lee, Cynthia Bailey and Allan Snavely. "On the User- Scheduler Dialogue: Studies of User-Provided Runtime Estimates and Utility Functions." International Journal of High Performance Computing Applications, vol. 20, pp , Contact: Cynthia Lee