1. Building the communication performance model of heterogeneous clusters based on a switched network Alexey Lastovetsky Alexey.Lastovetsky@ucd.ie Alexey.Lastovetsky@ucd.ie Vladimir Rychkov Vladimir.Rychkov@ucd.ie Heterogeneous Computing Laboratory School of Computer Science and Informatics University College Dublin, Ireland http://hcl.ucd.ie

2. September 20Heteropar'072 Objectives Clusters based on switched networks homogeneous clusters: communication operation = a combination of point-to-point parameters a series of the measurements between any two processors + statistics heterogeneous clusters: communication operation = a combination of averaged point-to-point parameters series of the measurements for every pair of processors + statistics + averaging inaccurate prediction for significantly different processors Heterogeneous communication performance model point-to-point communication: processor parameters:- fixed delays- variable delays link parameters:- transmission rate Problem: how to estimate the parameters? more than 2 linear parameters – design of communication experiments processor-specific parameters – minimization of the number of measurements

3. September 20Heteropar'073 Related work ModelBuilding Hockney linear regression 1 roundtrip for different message sizes LogP saturation to find the gap a series of measurements for a message size direct measurements + statistics to find the overhead a series of roundtrips for a message size formula to find latency from the roundtrip execution time and the overhead from PLogP parameters LogGP PLogP saturation to find the gap a series of measurements for different message sizes direct measurements + statistics to find the overheads a series of roundtrips for different message sizes formula to find latency from the roundtrip execution time and the gap for empty messages

4. September 20Heteropar'074 Heterogeneous comm. performance model Parameters cannot be found from other models the Hockney/LogGP parameters are insufficient to find variable processing delays and transmission rates the PLogP parameters are functions of message size Roundtrips are not enough to find all parameters empty roundtrips between each 3 processors → → fixed delays (processor parameters) roundtrips for 2 different message sizes → → fixed and variable parts how to separate variable delays (processor parameters) and transmission rate (link parameter) from variable part?

5. September 20Heteropar'075 Design of communication experiments Fixed processing delays ( unknowns) experiments Variable processing delays ( unknowns) Transmission rates ( unknowns) experiments

6. September 20Heteropar'076 Design of communication experiments Additional experiments Solution

7. September 20Heteropar'077 Selection of message size Observations one-to-many piecewise linear leap for large messages

8. September 20Heteropar'078 Building Experiments a small number of experiments particular message sizes (0 and ) fast roundtrips (one-to-one, one-to-two) Calculations comparisons simple expressions to get values Averaging within solution ( fixed, variable processing delays, and transmission rates) within measurements (for accurate measurement of the communication execution time)

9. September 20Heteropar'079 Comparison Communication Model Number of measurements LAM-Fast time, sec OpenMPI-Giga time, sec Hockney0.280.17 LogGP-- PLogP334.0463.11 Hetero-p2p2.140.33

10. September 20Heteropar'0710 Experiments Cluster 11 x Intel Xeon 2.8/3.4/3.6 2 x P4 3.2/3.4 1 x Celeron 2.9 2 x AMD Opteron 1.8 Gigabit Ethernet Open MPI 1.1.4

11. September 20Heteropar'0711 Experiments ScatterGather Cluster: 8 x Sun Ultra 5/10, Fast Ethernet, LAM 7.1.3

12. September 20Heteropar'0712 Conclusion Estimation of the parameters of the heterogeneous communication performance model Design of communication experiments 1-to-1 and 1-to-2 roundtrips particular message sizes Costs a relatively small number of fast measurements simple systems of linear equations Accuracy averaging within measurements and solution