1. Dr. Tansel Dökeroğlu tanseldoker@yahoo.com.tr
University of Turkish Aeronautical Association Computer Engineering Department Ceng 442 Introduction to Parallel Computing
Dr. Tansel Dökeroğlu

2. A presentation of a paper from journals:
Journal of Parallel and Distributed Computing (CUDA)
IEEE Transactions on Parallel and Distributed Systems
A project with MPI or OpenMP at the end of the semestr

3. The University of Adelaide, School of Computer Science
An Introduction to Parallel Programming
The University of Adelaide, School of Computer Science
17 January 2019
Why Parallel Computing?
Chapter 2 — Instructions: Language of the Computer

4. The University of Adelaide, School of Computer Science
17 January 2019
Roadmap
Why we need ever-increasing performance.
Why we’re building parallel systems.
Why we need to write parallel programs.
How do we write parallel programs?
What we’ll be doing.
Concurrent, parallel, distributed!
Chapter 2 — Instructions: Language of the Computer

5. Changing times
From 1986 – 2002, microprocessors were speeding like a rocket, increasing in performance an average of 50% per year. Generally, improving performance by increasing clock speed.
Since then, it’s dropped to about a 20% increase per year.

6. # of transistors in an integrated circuit has doubled every two years
Why do we need parallel programs? (from the perspective of Moore’s law)
The number of transistors in a dense integrated circuit is doubling approximately every two years.
# of transistors in an integrated circuit has doubled every two years

7. An intelligent solution
Instead of designing and building faster microprocessors, put multiple processors on a single integrated circuit.

8. Now it’s up to the programmers
Adding more processors doesn’t help much if programmers aren’t aware of them…
… or don’t know how to use them.
Serial programs don’t benefit from this approach (in most cases).

9. Why we need ever-increasing performance
Computational power is increasing, but so are our computation problems and needs.
Problems we never dreamed of have been solved because of past increases, such as decoding the human genome.
More complex problems are still waiting to be solved.

10. Climate modeling

11. Protein folding
Protein folding is the process by which a protein structure assumes its functional shape or conformation.

12. Drug discovery

13. Energy research

14. Data analysis

15. Why we’re building parallel systems
Up to now, performance increases have been attributable to increasing density of transistors.
But there are inherent problems.

16. A little physics lesson
Smaller transistors = faster processors (shorter distance for electricity to travel; can’t have cycle speed faster than it takes for answer to flow out of circuits).
Faster processors = increased power consumption.
Increased power consumption = increased heat.
Increased heat = unreliable processors.

17. Solution Move away from single-core systems to multicore processors.
“core” = central processing unit (CPU)
Introducing parallelism!!!

18. Why we need to write parallel programs
Think of running multiple instances of your favorite game. This is not parallel programming.
What you really want is for it to run faster.

19. Approaches to the serial problem
Rewrite serial programs so that they’re parallel.
Write translation programs that automatically convert serial programs into parallel programs.
This is very difficult to do.
Success has been limited.

20. More problems
Some coding constructs can be recognized by an automatic program generator, and converted to a parallel construct.
However, it’s likely that the result will be a very inefficient program.
Sometimes the best parallel solution is to step back and devise an entirely new algorithm.

21. Example
Compute n values and add them together.
Serial solution:

22. Example (cont.) We have p cores, p much smaller than n.
Each core performs a partial sum of approximately n/p values.
Each core uses it’s own private variables
and executes this block of code independently of the other cores.

23. Example (cont.)
After each core completes execution of the code, there is a private variable my_sum containing the sum of the values computed by its calls to Compute_next_value.
Once all the cores are done computing their private my_sum, they form a global sum by sending results to a designated “master” core which adds the final result

24. Example (cont.)

25. But wait!
There’s a much better way to compute the global sum.

27. Better parallel algorithm
Don’t make the master core do all the work.
Share it among the other cores.
Pair the cores so that core 0 adds its result with core 1’s result.
Core 2 adds its result with core 3’s result, etc.
Work with odd and even numbered pairs of cores.

28. Better parallel algorithm (cont.)
Repeat the process now with only the evenly ranked cores.
Core 0 adds result from core 2.
Core 4 adds the result from core 6, etc.
Now cores divisible by 4 repeat the process, and so forth, until core 0 has the final result.

29. Multiple cores forming a global sum

30. Analysis of the performance
In the first example, the master core performs 7 receives and 7 additions.
In the second example, the master core performs 3 receives and 3 additions.
The improvement is more than a factor of 2

31. Analysis (cont.)
The difference is more dramatic with a larger number of cores.
If we have 1000 cores:
The first example would require the master to perform 999 receives and 999 additions.
The second example would only require 10 receives and 10 additions.
That’s an improvement of almost a factor of 100

32. How do we write parallel programs?
Task parallelism
Partition various tasks of the problem among the cores.
Data parallelism
Partition the data used in solving the problem among the cores.
Each core carries out similar operations on it’s part of the data.

33. Professor P
15 questions
300 exams

34. Professor P’s grading assistants
Grader #1
Grader #3
Grader #2

35. Division of work – data parallelism
Grader #1
Grader #3
100 exams
100 exams
Grader #2
100 exams

36. Division of work – task parallelism
Grader #1
Grader #3
Questions
Questions 1 - 5
Grader #2
Questions

37. Division of work – data parallelism

38. Division of work – task parallelism
Tasks
Receiving
Addition

39. Coordination Cores usually need to coordinate their work.
Communication – one or more cores send their current partial sums to another core.
Load balancing – share the work evenly among the cores so that one is not heavily loaded.
Synchronization – because each core works at its own pace, make sure cores do not get too far ahead of the rest.

40. What we’ll be doing
Learning to write programs that are explicitly parallel.
Using the C/C++ language.
Using three different extensions to C.
Posix Threads (Pthreads)
Message-Passing Interface (MPI)
OpenMP (time permitting)

41. Type of parallel systems
Shared-memory
The cores can share access to the computer’s memory.
Coordinate the cores by having them examine and update shared memory locations.
Distributed-memory
Each core has its own, private memory.
The cores must communicate explicitly by sending messages across a network.

42. Type of parallel systems
Shared-memory
Distributed-memory

43. Terminology
Concurrent computing – a program is one in which multiple tasks can be in progress at any instant.
Parallel computing – a program is one in which multiple tasks cooperate closely to solve a problem
Distributed computing – a program may need to cooperate with other programs to solve a problem.

44. Concluding Remarks
The laws of physics have brought us to the doorstep of multicore technology.
Serial programs typically don’t benefit from multiple cores.
Automatic parallel program generation from serial program code isn’t the most efficient approach to get high performance from multicore computers.

45. Concluding Remarks
Learning to write parallel programs involves learning how to coordinate the cores.
Parallel programs are usually very complex and therefore, require sound program techniques and development.

46. Hello world with MPI Programming

50. How OpenMP works

51. How OpenMP works