1. Lincoln University Canterbury New Zealand Evaluating the Parallel Performance of a Heterogeneous System Elizabeth Post Hendrik Goosen formerly of Department of Computer Science University of Cape Town, South Africa

2. Lincoln University Canterbury New Zealand Introduction Why measure and evaluate parallel performance? What to measure How to evaluate Why speedup and efficiency are not appropriate Alternate methods of performance evaluation –Power weight –Linear speed –Linear efficiency Conclusions

3. Lincoln University Canterbury New Zealand Why measure parallel performance? Increasing use of parallel processing –Clusters, such as Beowulfs –Networks of Workstations (NOWs) Scalability –Does performance improve as more processors are added? –Will performance continue to improve as more processors are added? Efficiency –Is the best possible performance being achieved? –Where and how can performance be improved? –Is the best algorithm being used?

4. Lincoln University Canterbury New Zealand What makes systems heterogeneous? Different architectures Different memory and cache capabilities Different operating systems Communications Non-dedicated systems

5. Lincoln University Canterbury New Zealand Why use a real application to measure performance? Real users want to know how long real applications take Measures such as: –MIPS –MFLOPS –Kernels –Vendor-tuned benchmarks do not take into account factors such as: –input/output, –communication time, –memory needs and usage, –idle time etc.

6. Lincoln University Canterbury New Zealand Background Climatology application –Cloud radiation simulation to model and measure reflectivity, transmissivity, and absorptivity of a heterogeneous strato-cumulus cloud deck. Environment –Network of Unix workstations, five different models of Silicon Graphics and Sun workstations with varying CPU performance and memory capacity, –connected by a 10Mbit Ethernet network.

7. Lincoln University Canterbury New Zealand Serial times for processors CPU, System and Elapsed times

8. Lincoln University Canterbury New Zealand Parallel performance - What to measure? Elapsed time CPU time System time Communication time Idle time

9. Lincoln University Canterbury New Zealand Parallel performance - How to evaluate? Elapsed time graphs Speedup Efficiency Power weight Linear speed Linear efficiency

10. Lincoln University Canterbury New Zealand Heterogeneous Parallel group CPU, System and Elapsed times

11. Lincoln University Canterbury New Zealand Speedup Speedup is: the ratio of the serial time taken on one processor compared to the parallel time on all processors Elapsed time on 1 processor Elapsed time on n processors

12. Lincoln University Canterbury New Zealand Speedup – how to calculate it? Which single processor elapsed time should be used? –Elapsed time of fastest processor? –Elapsed time of slowest processor? –Mean elapsed time of all processors used?

13. Lincoln University Canterbury New Zealand Speedup vs Perfect Speedup (calculated with means)

14. Lincoln University Canterbury New Zealand Efficiency Efficiency is: the ratio of the speedup divided by the number of processors used Speedup for n processors n

15. Lincoln University Canterbury New Zealand Efficiency (calculated with means)

16. Lincoln University Canterbury New Zealand Speedup and efficiency not appropriate Conclusion that speedup and efficiency are not appropriate for evaluating parallel performance on a heterogeneous system, and even have their limitations on a homogeneous system.

17. Lincoln University Canterbury New Zealand Power weight (Zhang et al) Power weight is: ratio of performance of each processor as compared to performance of the fastest processor Elapsed time for fastest processor Elapsed time for n th processor

18. Lincoln University Canterbury New Zealand Linear Speed (Crowl) Linear Speed is: the amount of work done in unit time 1 Total elapsed time

19. Lincoln University Canterbury New Zealand Linear Speed

20. Lincoln University Canterbury New Zealand Linear Efficiency (Post and Goosen) Linear Efficiency is: ratio of work done in parallel application compared to potential amount of work that could be done by all processors Linear speed for parallel application on n processors Sum of single processor linear speeds for n processors

21. Lincoln University Canterbury New Zealand Linear Efficiency

22. Lincoln University Canterbury New Zealand Advantages of linear speed and linear efficiency Linear speed for each machine is independent of all other machines Linear speeds can be recalculated dynamically as workloads vary and used to determine each machine’s current performance compared to its serial capacity. Dynamic calculation of linear speeds and linear efficiency can be used for dynamic load- balancing algorithms.

23. Lincoln University Canterbury New Zealand Conclusions Important to measure overall elapsed time, as well as components of CPU, communication, idle time etc. Speedup and efficiency are not appropriate for evaluating parallel performance, especially for heterogeneous systems Linear speed and linear efficiency provide useful ways of evaluating parallel performance for both heterogeneous and homogeneous systems. Linear speed and linear efficiency can be calculated dynamically and be used in dynamic load balancing algorithms.

24. Lincoln University Canterbury New Zealand