1. ITCS 4145 Parallel Computing Spring 2016
Instructor
Dr. Barry Wilkinson
University of North Carolina Charlotte
© B. Wilkinson Modification date: Jan 13, 2016
outline.1 1 1

2. Outline
These slides constitute the course outline and include the course syllabus and course policies.

3. Parallel Computing
The use of multiple processors or computers to solve problems at a greater computational speed than using a single computer/processor.
Basic idea is simple – using N computers/ processors collectively on a problem should lead to a faster solution.
Can tackle problems that could not be solved in a reasonable time otherwise.
Also solve problems with higher precision and/or with more memory requirements.

4. Computer platforms for parallel computing
Multiple interconnected computers
- Cluster Computing, group of interconnected computers typically using Ethernet switch and physically all in one room.
2. A single computer system with multiple internal processors or cores
- Usually shares a common main memory
Computer system with attached graphic processing unit (GPU)
- GPUs have large number of execution cores, now used for high performance computing as well as original graphics application
Combination of above most likely.

5. UNCC cluster used in course
ssh to user gateway
cci-gridgw.uncc.edu
Login
coit-grid06
cci-grid07
cci-grid08
coit-grid01
coit-grid02
coit-grid03
coit-grid04
cci-grid09
C2050 GPU
(448 core)
C2050 GPU
(448 core)
K20 GPU
(2496 core)
cci-grid09:
Dual AMD Opteron GHz 16-core each, with 16 GB main memory
RAID1 disks,
redundant power supplies, VM, web server, user gateway login, SSH, NAT, DHCP, DNS, and LDAP, etc.
coit-grid01-4: Each two Xeon 3.4 Ghz processors, 8GB main memory
coit-grid06:
GPU server, with NVIDIA 2050 GPU
cci-grid08:
GPU server, E GHz 6-core Xeon processor with NVIDIA K20 GPU, 32 GB main memory
cci-grid07:
GPU server, X GHz quad-core Xeon processor with NVIDIA 2050 GPU, 12GB main memory
switch
cci-grid05: Four quad-core 2.93Ghz Xeon processors 64GB main memory
1.2 TB disk (RAID 5)
cci-grid05
All user’s home directories on cci-grid05 (NFS)
Has all three types of platform.
Note: Some names have been changed from coit- to cci-

6. Your own computer
Generally you will need your own computer (desktop or laptop, Windows, Mac, or Linux), as programming will be done on that computer in addition to using a remote cluster.
(You can use a lab computer although that might not be as convenient for you.)
All the software will be provided for download as a VirtualBox Ubuntu virtual machine. You will need to install VirtualBox. Detailed instructions are provided (“Pre-assigment”).
Detailed instructions also provided for a native installation on a Linux platform if you wish to do that instead. Generally also applies to a Mac.

7. Course Prerequisites UNC-C Catalog prerequisites:
ITCS 2214 (Data Structures), and
ITCS 3181/3182 (Computer Architecture).
Computer Architecture not strictly necessary so long as you know what a computer consists of (processors, memory and I/O) and how it operates.
What is needed -- basic skills in C and Java.

8. Course Text
Parallel Programming: Techniques and Application Using Networked Workstations and Parallel Computers, 2nd edition, by B. Wilkinson and M. Allen, Prentice Hall Inc., 2005, ISBN
Recommended, not strictly necessary as very detailed on-line notes provided, some derived from the course text – with updated materials since 2005

9. Home Page https://moodle2.uncc.edu “ITCS-4145-091-Spring 2016-22703”
For course notes, assignments, announcements, etc.:
UNCC Moodle-2:
“ITCS Spring ”

10. Course Contents
Demand for computational speed: grand challenge problems.
Potential for speed-up using multiple process(or)s: speed-up factor, max speed up, Amdahl's law, Gustafson's law.
Parallel computers: architectural types, shared memory, distributed memory, GPU systems.

11. Course Contents (Continued)
Programming with shared memory programming I: processes, fork, fork-join pattern, threads, Pthreads, thread pool pattern.
OpenMP: thread team pattern, directives/constructs, parallel, shared and local variables, work-sharing, sections, for, loop scheduling, for reduction, single master.

12. Course Contents (Continued)
Programming with shared memory programming II: Accessing shared data, critical sections, locks, condition variables, serializing code, deadlock, semaphores, monitors, Pthreads program example.
OpenMP continued: Sharing data and synchronization, critical, barrier, atomic, flush.
Shared memory performance issues: specifying parallelism, dependency analysis (Bernstein's conditions), data shared in caches, false sharing, sequential consistency, code re-ordering

13. Course Contents (Continued)
Message passing: MPI, point-to-point message passing, message tags, MPI communicator, blocking send/recv, command line compiling and executing MPI programs, instrumenting code for execution time, Eclipse IDE PTP
Message passing patterns: MPI collective routines, broadcast, scatter, gather, reduce, barrier, alltoall broadcast.
Implementation issues: barrier, local synchronization, safety and deadlock, safe MPI routines, synchronous message passing, asynchronous (non-blocking) message passing, changing to synchronous message passing.

14. Course Contents (Continued)
Pattern programming concepts: problem addressed, low message-passing patterns, point to point data transfer, broadcast, scatter, gather, reduce, all-to-all broadcast, higher level message-passing patterns, workpool, pipeline, divide and conquer, all-to-all, iterative synchronous patterns, iterative synchronous all-to-all, stencil, advantages and disadvantages of patterns.

15. Course Contents (Continued)
Our pattern programming tools:
Suzaku: macros, routines, implementation
Seeds pattern programming framework: module method, bootstrapping class, network and multicore versions, workpool programming examples - Monte Carlo pi, matrix addition, matrix multiplication.

16. Course Contents (Continued)
Hybrid programming: Combining MPI and OpenMP to take advantage of clusters that have both distributed-memory and shared-memory. Discussion of whether hybrid is any better than using only MPI or only OpenMP.

17. Course Contents (Continued)
CPU-GPU systems: data parallel pattern, ... GPU architecture, programming in CUDA, issues for achieving high performance.

18. Course Grade Undergraduate Grade Scale A 90 - 100 B 80 - 89.99F
Typically some curving will take place, based upon class averages. If you get around the class average, expect an average grade.

19. Computation of Grades Class quizzes (2) 25% Assignments (8*) 50%
Final exam 25%
* including the pre-assignment setting up the software
Read small print
The assessment and percentages may be modified
The course total displayed in Moodle may not compute according to the above and should not be used.

20. Code of Student Academic Integrity
All submitted assignments must be your own work. Copied work or work done by more than one person (unless specifically instructed) will not be accepted - at the very minimum on the first occasion, zero credit for the whole assignment - and you may be subject to disciplinary action.
The Code of Student Academic Integrity applies to both assignments and quizzes/tests.
Required statement

21. Due dates for assignments
Each assignment will have posted due date (for full credit) posted on Moodle, which is firm unless you have documented system/software problems.
Late penalty
Late Penalty
< 24 hours 10%
>= 24 hours and < 48 hours 20%
>= 48 hours and < 72 hours 30%
>= 72 hours 100%

22. Attendance Attendance in class is expected
If you the miss a class, it can have a deleterious effect on your grade:
2 marks off overall accumulated marks (out of 100) for each class missed without good reason,
after 3 missed classes.
If you must miss class, see the instructors beforehand to avoid loosing marks.

23. Mid-Term Unsatisfactory Grades
UNC-Charlotte requires midterm unsatisfactory grades to be submitted by the end of the 8th week of class.
Mid term grades
Undergraduates: Satisfactory or D or F
will be based upon all work graded by week 7, which
probably will be two assignments and one class quiz.

24. Disability Services UNC Charlotte is committed to access to education.
If you have a disability and need academic accommodations, please provide a letter of accommodation from Disability Services early in the semester.
For more information on accommodations, contact the Office of Disability Services at or visit their office at Fretwell 230.

25. Instructor details
Barry Wilkinson Department of Computer Science University of North Carolina, Charlotte Home page: Office Hours Wednesday: 2 pm to 4 pm 
Skype will be on during office hours, Skype ID abwilkinson
or in person in office, Woodward 435G
outline.25 25

26. Course Teaching Assistants
Raja Challa
Office hours/location: TBD, CCI Student Center 2nd floor Woodward.
Primary contact for Pre-assignment, Assignment 2, 4, and 6
Mohan Nallapareddy
Office hours/location:TBD, CCI Student Center 2nd floor Woodward.
Primary contact for Assignment 1, 3, 5, and 7
Contact Dr. Wilkinson about course materials, assignment write-up, or issues with the virtual machine.

27. Questions