2. Group Members Hamza Zahid (131391) Fahad Nadeem khan Abdual Hannan AIR UNIVERSITY MULTAN CAMPUS

3. Shared memory VS Message passing

4. Topics Message passing Shared memory Difference b/w message passing and shared memory

5. Message Passing

6. INTRODUCTION The architecture is used to communicate data among a set of processors without the need for a global memory Each PE has its own local memory and communicates with other PE using message

8. MP network Two important factors must be considered;  Link bandwidth –the number of bits that can be transmitted per unit of times(bits/s)  Message transfer through the network

9. Process communication  Processes running on a given processor use what is called internal channels to exchange messages among themselves  Processes running on different processors use the external channesls to exchange messages

10. Data exchanged  Data exchanged among processors cannot be shared; it is rather copied (using send/ receive messages)  An important advantage of this form of data exchange is the elimination of the need for synchronization constructs, such as semaphores, which results in performance improvement

11. Message-Passing Interface – MPI Standardization  MPI is the only message passing library which can be considered a standard. It is supported on virtually all HPC platforms. Practically, it has replaced all previous message passing libraries. Portability  There is no need to modify your source code when you port your application to a different platform that supports the MPI standard

12. Message-Passing Interface – MPI Performance Opportunities  Vendor implementations should be able to exploit native hardware features to optimize performance. Functionality  Over 115 routines are defined. Availability  A variety of implementations are available, both vendor and public domain.

13. MPI basics  Start Processes  Send Messages  Receive Messages  Synchronize  With these four capabilities, you can construct any program.  MPI offers over 125 functions.

14. Shared memory

15. Introduction  Processors communicate with shared address space  Easy on small-scale machines  Shared memory allows multiple processes to share virtual memory space.  This is the fastest but not necessarily the easiest (synchronization- wise) way for processes to communicate with one another.  In general, one process creates or allocates the shared memory segment.  The size and access permissions for the segment are set when it is created.

17. Uniform Memory Access (UMA)  Most commonly represented today by Symmetric Multiprocessor (SMP) machines  Identical processors  Equal access and access times to memory  Sometimes called CC-UMA - Cache Coherent UMA. Cache coherent means if one processor updates a location in shared memory, all the other processors know about the update. Cache coherency is accomplished at the hardware level.

18. Shared Memory (UMA)

19. Non-Uniform Memory Access (NUMA)  Often made by physically linking two or more SMPs  One SMP can directly access memory of another SMP  Not all processors have equal access time to all memories  Memory access across link is slower  If cache coherency is maintained, then may also be called CC- NUMA - Cache Coherent NUMA

20. Shared Memory (NUMA)

21. Advantages  Global address space provides a user-friendly programming perspective to memory  Model of choice for uniprocessors, small-scale MPs  Ease of programming  Lower latency  Easier to use hardware controlled caching  Data sharing between tasks is both fast and uniform due to the proximity of memory to CPUs

22. Disadvantages  Primary disadvantage is the lack of scalability between memory and CPUs. Adding more CPUs can geometrically increases traffic on the shared memory-CPU path, and for cache coherent systems, geometrically increase traffic associated with cache/memory management.  Programmer responsibility for synchronization constructs that ensure "correct" access of global memory.  Expense: it becomes increasingly difficult and expensive to design and produce shared memory machines with ever increasing numbers of processors.

23. Difference

24. Message Passing vs. Shared Memory Difference: how communication is achieved between tasks  Message passing programming model –Explicit communication via messages –Loose coupling of program components –Analogy: telephone call or letter, no shared location accessible to all  Shared memory programming model –Implicit communication via memory operations (load/store) –Tight coupling of program components –Analogy: bulletin board, post information at a shared space Suitability of the programming model depends on the problem to be solved. Issues affected by the model include:  Overhead, scalability, ease of programming

25. Message Passing vs. Shared Memory Hardware Difference: how task communication is supported in hardware  Shared memory hardware (or machine model) –All processors see a global shared address space Ability to access all memory from each processor –A write to a location is visible to the reads of other processors  Message passing hardware (machine model) –No global shared address space –Send and receive variants are the only method of communication between processors (much like networks of workstations today, i.e. clusters) Suitability of the hardware depends on the problem to be solved as well as the programming model.

26. Programming Model vs. Architecture Machine  Programming Model –Join at network, so program with message passing model –Join at memory, so program with shared memory model –Join at processor, so program with SIMD or data parallel Programming Model  Machine –Message-passing programs on message-passing machine –Shared-memory programs on shared-memory machine –SIMD/data-parallel programs on SIMD/data-parallel machine

27. Separation of Model and Architecture  Shared Memory –Single shared address space –Communicate, synchronize using load / store –Can support message passing  Message Passing –Send / Receive –Communication + synchronization –Can support shared memory