1. PERFORMANCE MEASURES

2. COMPUTATIONAL MODELS Equal Duration Model:  It is assumed that a given task can be divided into n equal subtasks, each of which can be executed by one processor.  If ts is the execution time of the whole task using a single processor, then the time taken by each processor to execute its subtask is tm = ts / n.  Since, according to this model, all processors are executing their subtasks simultaneously, then the time taken to execute the whole task is tm = ts/n.

3.  The speedup factor of a parallel system can be defined as the ratio between the time taken by a single processor to solve a given problem instance to the time taken by a parallel system consisting of n processors to solve the same problem instance.  The speedup factor resulting from using n processors is equal to the number of processors used, n.

4.  One important factor has been overlooked in the above derivation. This factor is the communication overhead, which results from the time needed for processors to communicate and possibly exchange data while executing their subtasks.  Assume that the time incurred due to the communication overhead is called tc then the actual time taken by each processor to execute its subtask is given by

5. Case 1: If then the potential speedup factor is approximately n Case 2: If then the potential speedup factor is Case 3: If then the potential speedup factor is, for

6. Efficiency  In order to scale the speedup factor to a value between 0 and 1, we divide it by the number of processors, n.  The resulting measure is called the efficiency, j.  The efficiency is a measure of the speedup achieved per processor.  According to the simple equal duration model, the efficiency j is equal to 1 if the communication overhead is ignored.  However if the communication overhead is taken into consideration, the efficiency can be expressed as

7.  Although simple, the equal duration model is however unrealistic.  This is because it is based on the assumption that a given task can be divided into a number of equal subtasks that can be executed by a number of processors in parallel.  However, it is sufficient here to indicate that real algorithms contain some (serial) parts that cannot be divided among processors.  These (serial) parts must be executed on a single processor.

8. Parallel Computation with Serial Sections Model In this computational model, it is assumed that a fraction f of the given task (computation) is not dividable into concurrent subtasks. The remaining part (1 - f ) is assumed to be dividable into concurrent subtasks.

9. Performing similar derivations to those done in the case of the equal duration model will result in the following. The time required to execute the task on n processors is The speedup factor is therefore given by

10.  According to this equation, the potential speedup due to the use of n processors is determined primarily by the fraction of code that cannot be divided.  If the task (program) is completely serial, that is, f = 1, then no speedup can be achieved regardless of the number of processors used. This principle is known as Amdahl’s law.  It is interesting to note that according to this law, the maximum speedup factor is given by

11.  According to Amdahl’s law the improvement in performance (speed) of a parallel algorithm over a sequential one is limited not by the number of processors employed but rather by the fraction of the algorithm that cannot be parallelized.  A first glance at Amdahl’s law indicates that regardless of the number of processors used, there exists an intrinsic limit on the potential usefulness of using parallel architectures.  For some time and according to Amdahl’s law, researchers were led to believe that a substantial increase in speedup factor would not be possible by using parallel architectures

12. The efficiency is defined as the ratio between the speedup factor and the number of processors, n. The efficiency can be computed as:

13. Amdahl’s Law Speedup factor of a parallel system as the ratio between the time taken by a single processor to solve a given problem instance to the time taken by a parallel system consisting of n processors to solve the same problem instance.

14. An interesting observation to make here is that according to Amdahl’s law, f is fixed and does not scale with the problem size, n. However, it has been practically observed that some real parallel algorithms have a fraction that is a function of n. Let us assume that f is a function of n such that Hence, This is clearly in contradiction to Amdahl’s law. It is therefore possible to achieve a linear speedup factor for large-sized problems, given that a condition that has been practically observed

15. Gustafson–Barsis’s Law  In 1988, Gustafson and Barsis at Sandia Laboratories studied the paradox created by Amdahl’s law and the fact that parallel architectures comprised of hundreds of processors were built with substantial improvement in performance.  In introducing their law, Gustafson recognized that the fraction of indivisible tasks in a given algorithm might not be known a priori.  They argued that in practice, the problem size scales with the number of processors, n.  This contradicts the basis of Amdahl’s law. Recall that Amdahl’s law assumes that the amount of time spent on the parts of the program that can be done in parallel, (1 - f ), is independent of the number of processors, n.

16. Gustafson and Brasis postulated that when using a more powerful processor, the problem tends to make use of the increased resources. They found that to a first approximation the parallel part of the program, not the serial part, scales up with the problem size. They postulated that if s and p represent respectively the serial and the parallel time spent on a parallel system, then s + p x n represents the time needed by a serial processor to perform the computation. They therefore, introduced a new factor, called the scaled speedup factor, SS(n), which can be computed as:

17.  This equation shows that the resulting function is a straight line with a slope = (1 - n).  This shows clearly that it is possible, even easier, to achieve efficient parallel performance than is implied by Amdahl’s speedup formula.  Speedup should be measured by scaling the problem to the number of processors, not by fixing the problem size.

18. How faster can we run? Suppose we have this serial problem with 12 tasks. How fast can we run given 3 processors?

19. Running in parallel Execution time reduces from 12 secs to 4 secs!

20. Load imbalance What if all processors can’t execute tasks with the same speed? ◦Load imbalance (ending parts for W2 and W3)

21. Dependence amongst tasks What if tasks 3, 7 and 11 are dependent? ◦Execution time increases from 4 to 6!

22. Scalability How much faster can a given problem be solved with N workers instead of one? How much more work can be done with N workers instead of one? What impact for the communication requirements of the parallel application have on performance? What fraction of the resources is actually used productively for solving the problem?

23. Scalability Simple model s: is the serial part p: is the parallel part Why serial part? ◦Algorithm limitations: dependencies ◦Bottlenecks: shared resources ◦Startup overhead: starting a parallel program ◦Communication: interacting with others workers

24. Scaling Strong Scaling: Keeping the problem size fixed and pushing in more workers or processors ◦Goal: Minimize time to solution for a given problem Weak Scaling: Keeping the work per worker fixed and adding more workers/processors (the overall problem size increases) ◦Goal: solve the larger problems

25. Strong Scaling Work done Single worker runtime: Same problem in parallel on N workers takes:

26. Weak Scaling Work done Single worker runtime: Same problem in parallel on N workers takes:

27. Performance: Strong Scaling Performance = work / time Strong scaling: ◦Serial performance: ◦Parallel performance: ◦Speedup: The same!

28. Example: Strong Scaling s: 0.1, p: 0.9, N: 1000 | s: 0.9, p: 0.1, N: 1000 Strong scaling: ◦Serial performance: ◦ 1 | 1 ◦Parallel performance and speedup: 9.91 | 1.11

29. Strong Scaling changing “work” Work: p ( only the parallelizable part) Strong scaling: ◦Serial performance: since ◦Parallel performance: ◦Speedup: Differ by factor of ‘p’ But speedup is the same

30. Example: Strong Scaling s: 0.1, p: 0.9, N: 1000 | s: 0.9, p: 0.1, N: 1000 Strong scaling: ◦Serial performance: 0.9 | 0.1 ◦Parallel performance: 8.9 | 0.11 ◦Speedup: 9.9 | 1.11

31. Performance: Weak Scaling Weak scaling: ◦Serial performance: ◦Parallel performance:

32. Gustafson’s law In case we recover Amdahl’s law. For and when N is large, ◦ is linear in ◦hence, we can cross Amdahl’s law and get unlimited performance!

33. Gustafson’s law In the case ◦Speedup is linear in N

34. Weak Scaling changing “work” Work: p ( only the parallelizable part) Weak scaling: ◦Serial performance: ◦Parallel performance: ◦Speedup: Differ by factor of ‘p’

35. Scalability: Weak Scaling For, ◦Linear in with a slope of 1 ◦Makes us believer that all is well and application scales perfectly ◦Remember previously, in Gustafson’s equation, it had a slope of (1-s)<1

36. Parallel Efficiency How effectively CPU computational power can be used in a parallel program.

37. For, (is there any significance of it?) ◦As N increases, decreases to 0 For, ◦Efficiency reaches a limit of 1-s=p. Weak scaling enables us to use at least a certain fraction of CPU power even when N is large ◦The more CPUs are used, the more CPU cycles are wasted (What do you understand by it?) Effect of

38. Changing “work” If work is defined as only the parallel part ‘p’ For, we get ◦Implied perfect scalability! ◦And no CPU cycles are wasted! ◦Is that true?

39. Example Some application has flops only in the parallel region s: 0.9 and p: 0.1 4 CPUs spend 90% idle! MFLOPs rate a factor of N higher than serial case

40. Serial performance and strong scalability Ran the same code on 2 different architectures Single thread performance is the same Serial fraction `s’ much smaller on arch2 due to greater memory bandwidth

41. Serial performance and strong scalability Scalar optimizations (studied earlier) should be done in serial or parallel parts? Assuming serial part can be accelerated by parallel performance becomes: In case parallel part is improved parallel performance becomes:

42. Serial performance and strong scalability When does optimizing the serial part pays off more than optimizing the parallel part: Does not depend on If, it pays off to optimize parallel region as serial optimization would be beneficial after a very large value of N.

43. Are these simple models enough? Amdahl’s law for strong scaling Gustafson’s law for weak scaling Problems with these models? ◦Communication ◦More memory (affects both strong and weak scaling?) ◦More cache (affects both strong and weak scaling?)