Vincent Keller, Ralf Gruber, EPFL Intelligent GRID Scheduling Service (ISS) K. Cristiano, A. Drotz, R.Gruber, V. Keller, P. Kunszt, P. Kuonen, S. Maffioletti, P. Manneback, M.-C. Sawley, U. Schwiegelshohn, M. Thiémard, A. Tolou, T.-M. Tran, O. Wäldrich, P. Wieder, C. Witzig, R. Yahyapour, W. Ziegler, “Application-oriented scheduling for HPC Grids”, CoreGRID TR-0070 (2007) available on

Outline ISS Goals Applications & Resources characterization ISS architecture Decision model : CFM ISS Modules/Services Implementation Status Testbeds (HW & SW)

Goals of ISS 1. Find most suited computational resources in a HPC Grid for a given component 2. Use best an existing HPC Grid 3. Predict best evolution of an HPC Grid

Γ model : Characteristic parameters of an application task * O: Number of operations per node [Flops] W: Number of main memory accesses per node [Words] Z: Number of messages to be sent per node S: Number of words sent by one node [Words] V a =O/W:Number of operations per memory access [Flops/Word]  a = O/S: Number of operations per word sent [Flops/Word] * suppose the parallel subtasks are well equilibrated

Γ model : Characteristic parameters of a parallel machine P: Number of nodes in a machine R  : Peak performance of a node [Flops/s] M  : Peak main memory bandwidth of a node [Words/s] V M =R  / M  :Number of operations per memory access [Flops/Word] r a = min (R , M  * V a ): Peak task performance on a node [Flops/s] t c = O/r a :Minimum computation time [s] Note: r a = R  min (1, V a /V M )

Γ model : Characteristic parameters of the internode network C  :Total network bandwidth of a machine [Words/s] L: Latency of the network [s] :Average distance (= number of links passed) V c =P R  / C  : Number of operations per sent word [Flops/Word] b=C  /(P* ): Inter-node communication bandwidth per node [Words/s] t b =S/b:Time needed to send S words through the network [s] t L =LZ:Latency time [s] T=t c + t b + t L :Minimum turn around time of a task *  M =(r a /b)(1+t L /t b ): Number of operations per word sent [Flops/Word] B=b L:Message size taking L to be transfered * I/O is not considered and communication cannot be hidden behind computation

 model (One value per application and machine)  > 1 Speedup  =  a /  M Task/application:  a = O / S [flops/64bit word] Machine (if LZ/S<<1):  M = r a / b [flops/64bit word] Efficiency

Parameters of some Swiss HPC machines *  32 for half of C  **  10 *** decommissioned

Example: Speculoos Pleiades 2 GbE  =3.8 Pleiades 1 FE  =1.4 Pleiades 2+ GbE  =1.6

ISS/VIOLA environment

ISS : Job Execution Process Goal: Find most suited machines in a Grid to run application components

Cost Function Model

CPU Costs K e licence fees K l Results waiting time K w Energy Costs K eco Data Transfer Costs K d All the costs are expressed in Electronic Cost Unit (ECU)

Cost Function Model : CPU costs with investment cost, maintenance fees, bank interest, etc..

Cost Function Model : Broker The broker computes a list of machines with their relative costs for a given application component This ordered list is sent to the MSS for final decision and submission

Other important goal of ISS Simulation to evolve cluster resources in a Grid (uses the same simulator as to determine , ,  using statistical application execution data over a long period in time (same data as to determine , ,  Support tool to decide on how to choose new Grid resource

Side products VAMOS monitoring service (measurement of R a,  ) Application optimization (increase V a, R a ) Processor frequency adaptation (reduce energy consumption)

What exists? Simulator to determine , ,  VAMOS monitoring service to determine  Cost Function Model

What is in implementation phase? Interface between ISS and MSS (first version ready by end of June 07) R a monitoring (ready by end of Mai 07) Cost Function Model (beta version ready by end of 07) Simulator to predict new cluster acquisition (by the end of 07)

Application testbed CFD, MPI: SpecuLOOS (3D spectral element method) CFD, OpenMP: Helmholtz (3D solver with spectral elements) Plasma physics, single proc: VMEC (3D MHD equilibrium solver) Plasma physics, single proc: TERPSICHORE (3D ideal linear MHD stability analysis) Climate, POP-C++: Alpine3D (multiphysics, components) Chemistry : GAMESS (ab-initio molecular quantum chemistry)

First hardware testbed UNICORE/MSS/ISS GRID Pleiades 1 (132 single proc nodes, FE switch, OpenPBS/Maui) Pleiades 2 (120 single proc nodes, GbE switch, Torque/Maui) Pleiades 2+ (99 dual proc/dual core nodes, GbE switch, Torque/Maui) CONDOR pool EPFL (300 single & multi proc nodes, no interconnect network)

CSCS: SMP/vector Low  m cluster EPFL: SMP/NUMA High  m cluster ETHZ: SMP/NUMA High  m cluster EIF: NoW CERN: egee Grid SWING Switch ISSISS ISS as a SwissGrid metascheduler

Conclusions Automatic: Find best suited machines for a given application Monitor application behaviours on single node and network Guide towards: Better usage of overall GRID Extend existing GRID by best suited machines for an application set Single node optimization and better parallelization