1. Feeding Parallel Machines – Any Silver Bullets? Novica Nosović ETF Sarajevo 8th Workshop “Software Engineering Education and Reverse Engineering” Durres, Albania, 8 – 13 September 2008

2. Outline Parallelism through decades (5 min) Cirrent trends in programming models (10 min) Most likely way to the (near) future (5 min) Discussion!!! (my favorite)

3. 29.9.2016 3 What is Parallel Machine? A parallel machine is a collection of processing elements that cooperate to solve large problems fast Some broad issues: – Resource Allocation: » how large a collection? » how powerful are the elements? » how much memory? – Data access, Communication and Synchronization » how do the elements cooperate and communicate? » how are data transmitted between processors? » what are the abstractions and primitives for cooperation? – Performance and Scalability » how does it all translate into performance? » how does it scale?

4. Parallelism levels Bit Level Parallelism Instruction Level Parallelism Thread Level Parallelism Data Level Parallelism Program...

5. 29.9.2016 5 Performance Slowdown!

6. 29.9.2016 6 Sea Change in Chip Design Intel 4004 (1971): 4-bit processor, 2312 transistors, 0.4 MHz, 10 micron PMOS, 11 mm 2 chip Processor is the new transistor? RISC II (1983): 32-bit, 5 stage pipeline, 40,760 transistors, 3 MHz, 3 micron NMOS, 60 mm 2 chip 125 mm 2 chip, 0.065 micron CMOS = 2312 RISC II+FPU+Icache+Dcache –RISC II shrinks to ~ 0.02 mm 2 at 65 nm

7. 29.9.2016 7 Problems with Sea Change Algorithms, Programming Languages, Compilers, Operating Systems, Architectures, Libraries, … not ready to supply Thread Level Parallelism or Data Level Parallelism for 1000 CPUs / chip, Architectures not ready for 1000 CPUs / chip Unlike Instruction Level Parallelism, cannot be solved by just by computer architects and compiler writers alone, but also cannot be solved without participation of computer architects

8. 29.9.2016 8 Applications of Parallel Processing and their Interactions

9. 29.9.2016 9 Principles vs. Devils in Details

10. 29.9.2016 10 Paradigms of Parallel Processing

11. 29.9.2016 11 Clusters and Networks of Workstations Connect multiple complete machines together using standard fast interconnects – Little or no hardware development cost – Each node can boot separately and operate independently – Interconnect can be attached at I/O bus (most common) or on memory bus (higher speed but more difficult) Clustering initially used to provide fault tolerance Clusters of SMPs (CluMPs) – Connect multiple n-way SMPs using a cache-coherent memory bus, fast message passing network or non cache-coherent interconnect Build message passing MPP by connecting multiple workstations using fast interconnect connected to I/O Bus. Main advantage?

12. 20 years of "free lunch" is over! “Just wait a year and the processor gets faster" – does not help!!! Paradigm shift in software (re)engineering is needed!

13. Manycore to come Not only cores that double like chromosomes communication network on chip very tightly coupled memory architecture is changing - bandwidth has increased dramatically GPUs, Cell... different memory model and cache coherency

14. Software is not ready!!! traditional model - threads! works well with shared memory distributed memory... end of threads??? but VM! like JVM!? VM manages processors, distributed memory... for photo editing, multimedia on desktop, speech recognition (lacks floating point footage!!)

15. The bigest wall! How can SE keep pace with these evolving machines that are rendering the existing application base obsolete? Entirely different way to program is needed... something developers are not used to There is a real void in the tools !!!... we're starting to move to processors that have distributed memory where the thread shared memory model doesn't work!!!

16. 29.9.2016 16 Summary: Why Feeding Parallel Machines? Increasingly attractive – Economics, technology, architecture, application demand Increasingly central and mainstream Parallelism exploited at many levels – Instruction-level parallelism – Multiprocessor servers – Large-scale multiprocessors (“MPPs”) Spectrum of feeding methods make sense – Different cost, performance and scalability

17. 29.9.2016 17 Where are Parallel Machines Going? Application Software System Software SIM D Message Passing Shared Memory Dataflo w Systolic Arrays Architectur e Uncertainty of direction paralyzed parallel software development! Old view: Divergent architectures, no predictable pattern of growth.

18. 29.9.2016 18 Today Extension of “computer architecture” to support communication and cooperation – Instruction Set Architecture plus Communication Architecture Defines – Critical abstractions, boundaries, and primitives (interfaces) – Organizational structures that implement interfaces (hw or sw) Compilers, libraries and OS are important bridges today

19. 29.9.2016 19 Modern Layered Framework CA D MultiprogrammingShared address Message passing Data parallel DatabaseScientific modeling Parallel applications Programming models Communication abstraction User/system boundary Compilation or library Operating systems support Communication hardware Physical communication medium Hardware/software boundary

20. 29.9.2016 20 Remember! parallelism is the future of computing

21. 29.9.2016 21 Thanks! Feedback, anonymous or not, welcome