1. Lecture 5 Barriers and MPI Introduction Topics Barriers Uses implementations MPI Introduction Readings – Semaphore handout dropboxed January 24, 2012 CSCE 713 Advanced Computer Architecture

2. – 2 – CSCE 713 Spring 2012 Overview Last Time Posix Pthreads: create, join, exit, mutexes /class/csce713-006 Code and Data Readings for today http://csapp.cs.cmu.edu/public/1e/public/ch9-preview.pdfNew Website alive and kicking; dropbox too! From Last Lecture’s slides: Gauss-Seidel Method, Barriers, Threads Assignment Next time performance evaluation, barriers and MPI intro

3. – 3 – CSCE 713 Spring 2012 Threads programming Assignment 1.Matrix addition (embarassingly parallel) 2.Versions  Sequential  Sequential with blocking factor  Sequential Read without conversions  Multi threaded passing number of threads as command line argument (args.c code should be distributed as an example) 3.Plot of several runs 4.Next time 5.Test Data in /class/csce713-006/Data A100, A400, B100, B400

4. – 4 – CSCE 713 Spring 2012 Next few slides are mostly from Parallel Programming by Pacheco /class/csce713-006/Code/Pacheco contains the code (again only for use with this course do not distribute)

5. – 5 – CSCE 713 Spring 2012 Recall use of mutex to control access to a variable Copyright © 2010, Elsevier Inc. All rights Reserved

6. – 6 – CSCE 713 Spring 2012 Barriers - synchronizing threads Synchronizing threads after a period of computation No thread proceeds until all others have reached the barrierNo thread proceeds until all others have reached the barrier E.g., last time iteration of Gauss-Siedel, sync check for convergence Examples of barrier uses and implementations 1.Using barriers for testing convergence, i.e. satisfying a completion criteria 2.Using barriers to time the slowest thread 3.Using barriers for debugging

7. – 7 – CSCE 713 Spring 2012 Using barriers for testing convergence, i.e. satisfying a completion criteria Slide 43 of Lecture 3 (slightly modified) General computation Stop when all x i from this iteration calculated

8. – 8 – CSCE 713 Spring 2012 Using barriers to time the slowest thread Copyright © 2010, Elsevier Inc. All rights Reserved

9. – 9 – CSCE 713 Spring 2012 Using barriers for debugging Copyright © 2010, Elsevier Inc. All rights Reserved

10. – 10 – CSCE 713 Spring 2012 Barrier Implementations 1.Mutex, threadCounter, busy-waits 2.Mutex plus barrier semaphore 3.Mutex and condition variables

11. – 11 – CSCE 713 Spring 2012 Busy-waiting and a Mutex Implementing a barrier using busy-waiting and a mutex is straightforward. We use a shared counter protected by the mutex. When the counter indicates that every thread has entered the critical section, threads can leave the critical section. Copyright © 2010, Elsevier Inc. All rights Reserved

12. – 12 – CSCE 713 Spring 2012 Busy-waiting and a Mutex Copyright © 2010, Elsevier Inc. All rights Reserved We need one counter variable for each instance of the barrier, otherwise problems are likely to occur.

13. – 13 – CSCE 713 Spring 2012 Implementing a barrier with semaphores Copyright © 2010, Elsevier Inc. All rights Reserved

14. – 14 – CSCE 713 Spring 2012 Condition Variables A condition variable is a data object that allows a thread to suspend execution until a certain event or condition occurs. When the event or condition occurs another thread can signal the thread to “wake up.” A condition variable is always associated with a mutex. Copyright © 2010, Elsevier Inc. All rights Reserved

15. – 15 – CSCE 713 Spring 2012 Condition Variables Copyright © 2010, Elsevier Inc. All rights Reserved

16. – 16 – CSCE 713 Spring 2012 Implementing a barrier with condition variables Copyright © 2010, Elsevier Inc. All rights Reserved

17. – 17 – CSCE 713 Spring 2012 POSIX Semaphores $ man -k semaphore sem_close (3) - close a named semaphore sem_destroy (3) - destroy an unnamed semaphore sem_getvalue (3) - get the value of a semaphore sem_init (3) - initialize an unnamed semaphore sem_open (3) - initialize and open a named semaphore sem_overview (7) - Overview of POSIX semaphores sem_post (3) - unlock a semaphore sem_timedwait (3) - lock a semaphore sem_trywait (3) - lock a semaphore sem_unlink (3) - remove a named semaphore sem_wait (3) - lock a semaphore $ man –s 7 semaphores No manual entry for semaphore in section 7

18. – 18 – CSCE 713 Spring 2012 Implementing a barrier with condition variables Copyright © 2010, Elsevier Inc. All rights Reserved

19. – 19 – CSCE 713 Spring 2012 Pthread Barriers // Barrier variable pthread_barrier_t barr; // Barrier initialization if(pthread_barrier_init(&barr, NULL, THREADS)) { printf("Could not create a barrier\n"); return -1; } // Create Threads for(int i = 0; i < THREADS; ++i) { if(pthread_create(&thr[i], NULL, &entry_point, (void*)i)) …

20. – 20 – CSCE 713 Spring 2012 Wait on Barrier in thread function // Synchronization point int rc = pthread_barrier_wait(&barr); if(rc != 0 && rc !=PTHREAD_BARRIER_SERIAL_THREAD){ printf("Could not wait on barrier\n"); exit(-1);}

21. – 21 – CSCE 713 Spring 2012 System V Semaphores (Sets) In System V IPC there are semaphore sets Commands ipcs - provide information on ipc facilitiesipcs - provide information on ipc facilities ipcrm - remove a message queue, semaphore set or shared memory segment etc.ipcrm - remove a message queue, semaphore set or shared memory segment etc. System Calls: semctl (2) - semaphore control operations semget (2) - get a semaphore set identifier semop (2) - semaphore operations semtimedop (2) - semaphore operations

22. – 22 – CSCE 713 Spring 2012 MPI Introduction

23. – 23 – CSCE 713 Spring 2012 A distributed memory system Copyright © 2010, Elsevier Inc. All rights Reserved

24. – 24 – CSCE 713 Spring 2012 A shared memory system Copyright © 2010, Elsevier Inc. All rights Reserved

25. – 25 – CSCE 713 Spring 2012 A tree-structured global sum Copyright © 2010, Elsevier Inc. All rights Reserved

26. – 26 – CSCE 713 Spring 2012 An alternative tree-structured global sum Copyright © 2010, Elsevier Inc. All rights Reserved

27. – 27 – CSCE 713 Spring 2012 Identifying MPI processes Common practice to identify processes by nonnegative integer ranks. p processes are numbered 0, 1, 2,.. p-1 Copyright © 2010, Elsevier Inc. All rights Reserved

28. – 28 – CSCE 713 Spring 2012 Our first MPI program Copyright © 2010, Elsevier Inc. All rights Reserved

29. – 29 – CSCE 713 Spring 2012 Compilation mpicc -g -Wall -o mpi_hello mpi_hello.c

30. – 30 – CSCE 713 Spring 2012 Execution Copyright © 2010, Elsevier Inc. All rights Reserved mpiexec -n mpiexec -n 1./mpi_hello mpiexec -n 4./mpi_hello run with 1 process run with 4 processes

31. – 31 – CSCE 713 Spring 2012 Execution Copyright © 2010, Elsevier Inc. All rights Reserved mpiexec -n 1./mpi_hello mpiexec -n 4./mpi_hello Greetings from process 0 of 1 ! Greetings from process 0 of 4 ! Greetings from process 1 of 4 ! Greetings from process 2 of 4 ! Greetings from process 3 of 4 !

32. – 32 – CSCE 713 Spring 2012 MPI Programs Written in C. Has main. Uses stdio.h, string.h, etc. Need to add mpi.h header file. Identifiers defined by MPI start with “MPI_”. First letter following underscore is uppercase. For function names and MPI-defined types. Helps to avoid confusion. Copyright © 2010, Elsevier Inc. All rights Reserved

33. – 33 – CSCE 713 Spring 2012 MPI Components MPI_Init Tells MPI to do all the necessary setup.MPI_Finalize Tells MPI we’re done, so clean up anything allocated for this program. Copyright © 2010, Elsevier Inc. All rights Reserved

34. – 34 – CSCE 713 Spring 2012 Basic Outline Copyright © 2010, Elsevier Inc. All rights Reserved

35. – 35 – CSCE 713 Spring 2012 Communicators A collection of processes that can send messages to each other. MPI_Init defines a communicator that consists of all the processes created when the program is started. Called MPI_COMM_WORLD. Copyright © 2010, Elsevier Inc. All rights Reserved

36. – 36 – CSCE 713 Spring 2012 Communicators Copyright © 2010, Elsevier Inc. All rights Reserved number of processes in the communicator my rank (the process making this call)