1. Fall 2008 CS 668 Parallel Computing Prof. Fred Annexstein fred.annexstein@uc.edu Office Hours: 11-1 MW or by appointment Tel: 513-556-1807

2. Lecture 1: Welcome Goals of this course Syllabus, policies, grading Blackboard Resources LINC Linux cluster Introduction/Motivation for HPPC Scope of the Problems in Parallel Computing

3. Goals Primary: –Provide an introduction to the computing systems, programming approaches, common numerical and algorithmic methods used for high performance parallel computing Secondary: –Have an course meeting competency requirements of RRSCS –Provide hands-on parallel programming experience

4. Official Syllabus Available on Blackboard Textbook Parallel Programming in C with MPI and OpenMP, Michael J. Quinn Other Recommended Texts -Parallel Programming With Mpi, Peter Pacheco -Introduction to Parallel Computing: Design and Analysis of Algorithms: Ananth Grama, Anshul Gupta, George Karpis, Vipin Kumar - Using MPI - 2nd Edition: Portable Parallel Programming with the Message Passing Interface by William Gropp

5. Workload/Grading Exams (1 or 2) –Graded 30% of Grade Written exercises (3-4) –May/may not be graded Programming Assignments (3-4) –May be done in groups of at most 2 –MPI programming, performance measurement Research papers (1) –Discussion research questions, strengths, weaknesses, interesting points, contemporary bibliography Final project (1) –Individual or Group programming project and report

6. Policies Missed Exams: –Missed exams can not be made up unless pre- approved. Please see the instructor as soon as possible in the event of a conflict. Academic Honesty: –Plagiarism on assignments, quizzes or exams will not be tolerated. See your student code of conduct (http://www.uc.edu/conduct/Code_of_Conduct.html) for more on the consequences of academic misconduct. There are no “small” offenses.http://www.uc.edu/conduct/Code_of_Conduct.html

7. Blackboard Syllabus and my contact info Announcements Lecture slides Assignment handouts Web resources relevant to the course Discussion board Grades

8. What is the Ralph Regula School? The Ralph Regula School of Computational Science is a statewide, virtual school focused on computational science. It is a collaborative effort of the Ohio Board of Regents, Ohio Supercomputer Center, Ohio Learning Network and Ohio's colleges and universities. With funding from NSF, the school acts as a coordinating entity for a variety of computational science education activities aimed at making education in computational science available to students across Ohio, as well as to workers seeking continuing education about this technology. Website: http://www.rrscs.orghttp://www.rrscs.org

9. CS LINC Cluster Michal Kouril’s links –http://www.ececs.uc.edu/~kourilm/clusters/http://www.ececs.uc.edu/~kourilm/clusters/ –See README file for instructions on running MPI code on beowulf.linc.uc.edu Accounts –ECE/CS students should already have an account –I can request accounts for the non-ECE/CS students Access –Remote access only, the cluster is in the ECE/CS server/machine room on the 8 th floor of Rhodes, visible through windows in the 890’s hallway

10. Why HPPC? Who needs a roomful of computers anyway? My PC and XBOX run at GFLOP rates ( Billion Floating Point Operations per second) NCSA TeraGrid IA-64 Linux Cluster (http://www.ncsa.uiuc.edu/UserInfo/Resources/ Hardware/TGIA64LinuxCluster/)

11. Needed by People who solve Science and Engineering problems Materials / Superconductivity Fluid Flow Weather/Climate Structural Deformation Genetics / Protein interactions Seismic Many Research Projects in Natural Sciences and Engineering cannot exist without HPPC

12. Applications Videos – Applications in Physics and Geology Simulation of Large-Scale Structure of Universe http://www.youtube.com/watch?v=8C_dnP2fvxkSimulation of Large-Scale Structure of Universe http://www.youtube.com/watch?v=8C_dnP2fvxk Stability Simulation – http://www.youtube.com/watch?v=ZCMiLJOXrpcStability Simulation – http://www.youtube.com/watch?v=ZCMiLJOXrpc Super Volcano Movie - Show first 1:00 minute http://www.youtube.com/watch?v=unGODG7N1BsSuper Volcano Movie - Show first 1:00 minute http://www.youtube.com/watch?v=unGODG7N1Bs

13. Why are the problems so large? 3-Dimensional –If you want to increase the level of resolution by factor of 10, problem size increases by 10 3 Many Length Scales (both time and space) –If you want to observe the interactions between very small local phenomenon and larger more global phenomenon The number of relationships between data items grows quadraticly. –Example: human genome 3.2 G base pairs means about 5,000,000,000,000,000,000=5E relations

14. How can you solve these problems? Take advantage of parallelism –Large problems generally have many operations which can be performed concurrently Parallelism can be exploited at many levels by the computer hardware –Within the CPU core, multiple functional units, pipelining –Within the Chip, many cores –On a node, multiple chips –In a system, many nodes

15. However…. Parallelism has overheads –At the core and chip level the cost is complexity and money –Most applications get only a fraction of peak performance (10%-20%) –At the chip and node level, memory bus can get saturated if too many cores –Between nodes, the communication infrastructure is typically much slower than the CPU

16. Necessity Yields Modest Success Power of CPUs keeps growing exponentiallyPower of CPUs keeps growing exponentially Parallel programming environments changing very slowly – much harder than sequentialParallel programming environments changing very slowly – much harder than sequential Two standards have emerged MPI library, for processes that do not share memoryMPI library, for processes that do not share memory OpenMP directives, for processes that do share memoryOpenMP directives, for processes that do share memory

17. Why MPI? MPI = “Message Passing Interface”MPI = “Message Passing Interface” Standard specification for message- passing librariesStandard specification for message- passing libraries Very PortableVery Portable Libraries available on virtually all parallel computersLibraries available on virtually all parallel computers Free libraries also available for networks of workstations or commodity clustersFree libraries also available for networks of workstations or commodity clusters

18. Why OpenMP? OpenMP an application programming interface (API) for shared-memory systemsOpenMP an application programming interface (API) for shared-memory systems Based on model of creating and scheduling multi-threaded computations.Based on model of creating and scheduling multi-threaded computations. Supports higher performance parallel programming of symmetrical multiprocessorsSupports higher performance parallel programming of symmetrical multiprocessors

19. What are the Costs? Commercial Parallel Systems Relatively costly per processorRelatively costly per processor Primitive programming environmentsPrimitive programming environments Scientists looked for alternativeScientists looked for alternative Beowulf Concept circa 1994 NASA project (written by Sterling and Becker)NASA project (written by Sterling and Becker) Commodity processorsCommodity processors Commodity interconnectCommodity interconnect Linux operating systemLinux operating system Message Passing Interface (MPI) libraryMessage Passing Interface (MPI) library High performance/$ for certain applicationsHigh performance/$ for certain applications

20. How are they Programmed? Task Dependence Graph Begin with Directed graphBegin with Directed graph Vertices = tasks Edges = dependencesVertices = tasks Edges = dependences Edges are removed as tasks completeEdges are removed as tasks complete Data Parallelism Independent tasks apply same operation to different elements of a data setIndependent tasks apply same operation to different elements of a data set Functional Parallelism Independent tasks apply different operations to different data elementsIndependent tasks apply different operations to different data elements Pipelining Divide a process into stagesDivide a process into stages Produce and consume several items simultaneouslyProduce and consume several items simultaneously

21. Why not just use a Compiler? Parallelizing compiler - Detect parallelism in sequential programParallelizing compiler - Detect parallelism in sequential program Produce parallel executable programProduce parallel executable programAdvantages Can leverage millions of lines of existing serial programs Saves time and labor- Requires no retraining of programmersSaves time and labor- Requires no retraining of programmers Sequential programming easier than parallel programmingSequential programming easier than parallel programmingDisadvantages Parallelism may be irretrievably lost when programs written in sequential languagesParallelism may be irretrievably lost when programs written in sequential languages Simple example: Compute all partial sums in an arraySimple example: Compute all partial sums in an array Performance of parallelizing compilers on broad range of applications still up in airPerformance of parallelizing compilers on broad range of applications still up in air

22. Can we Extend Existing Languages? Programmer can give directives or clues to the complier about how to parallelize Advantages Easiest, quickest, and least expensiveEasiest, quickest, and least expensive Allows existing compiler technology to be leveragedAllows existing compiler technology to be leveraged New libraries can be ready soon after new parallel computers are availableNew libraries can be ready soon after new parallel computers are availableDisadvantages Lack of compiler support to catch errorsLack of compiler support to catch errors Easy to write programs that are difficult to debugEasy to write programs that are difficult to debug

23. Or Create New Parallel Languages? Advantages Allows programmer to communicate parallelism to compiler directlyAllows programmer to communicate parallelism to compiler directly Improves probability that executable will achieve high performanceImproves probability that executable will achieve high performanceDisadvantages Requires development of new compilersRequires development of new compilers New languages may not become standardsNew languages may not become standards Programmer resistanceProgrammer resistance

24. Where are we in 2008? Performance makes Low-level approaches popularPerformance makes Low-level approaches popular Augment existing language with low-level parallel constructs and directivesAugment existing language with low-level parallel constructs and directives MPI and OpenMP are prime examplesMPI and OpenMP are prime examplesAdvantages EfficiencyEfficiency PortabilityPortabilityDisadvantages More difficult to program and debugMore difficult to program and debug

25. Programming Assignment #1 Log into beowulf.linc.uc.edu and run some simple sample programs.

26. Reading Assignment #1 on Blackboard