DEV490 Easy Multi-threading for Native.NET Apps with OpenMP ™ and Intel ® Threading Toolkit Software Application Engineer, Intel EMEA

Agenda Hyper-Threading technology and What to do about it The answer is: multithreading! Introduction to OpenMP ™ Maximizing performance with OpenMP and Intel ® Threading Tools

How HT Technology Works Physical processors Logical processors visible to OS Physical processor resource allocation Throughput Time Resource 1 Resource 2 Resource 3 Thread 2 Thread 1 Without Hyper- Threading With Hyper- Threading Thread 2 Thread 1 Resource 1 Resource 2 Resource 3 + Higher resource utilization, higher output with two simultaneous threads

HT Technology, Not Magic Arch State MultiprocessorHyper-Threading Data Caches CPU Front- End Out-of- Order Execution Engine Data Caches CPU Front- End Out-of- Order Execution Engine Data Caches CPU Front- End Out-of- Order Execution Engine HT Technology increases processor performance by improving resource utilization

HT Technology: What’s Inside Instruction TLB Next Instruction PointerInstruction Streaming Buffers Trace Cache Fill Buffers Register Alias Tables Trace Cache Next IP Return Stack Predictor HT Technology increases processor performance at extremely low cost

Taking Advantage of HT HT is transparent to OS and apps Software simply “sees” multiple CPUs Software usage scenarios to benefit from HT: Multithreading – inside one application Multitasking – among several applications OS support enhances HT benefits: smart scheduling to account for logical CPUs halting logical CPUs in the idle loop implemented in: Windows* XP, Linux* 2.4.x To take advantage of HT Technology, multithread your application!

What? Multithreading Strategy How? Multithreading Implementation Multithreading Your Application

Multithreading Strategy Exploit Task Parallelism Exploit Data Parallelism

Exploiting Task Parallelism Partition work into disjoint tasks Execute the tasks concurrently (on separate threads) Compress Thread Encrypt Thread Data

Exploiting Data Parallelism Thread A Thread N … Partition data into disjoint sets Assigns the sets to concurrent threads

Multithreading Implementation API / Library Win32* threading API P-threads MPI Programming language mechanisms Java* C# Programming language extension OpenMP ™

What is OpenMP ™ ? Compiler extension for easy multithreading Ideally WITHOUT CHANGING C/C++ CODE Three components: #pragma s (compiler directives) – most important API and Runtime Library Environment Variables Benefits: Easy way to exploit Hyper-Threading Portable and standardized Allows incremental parallelization Dedicated profiling tools

Programming Model Fork-Join Parallelism: main thread creates a team of additional threads team threads join at the end of parallel region All memory (variables) is shared by default Parallelism can be added incrementally sequential program evolves into parallel parallel regions team of threads main thread

Our First OpenMP ™ Program All OpenMP ™ pragmas start with omp Pragma parallel starts team of n threads Everything inside is executed n times! All memory is shared! void func() { int a, b, c, d; a = a + b; c = c + d; } void func() { int a, b, c, d; #pragma omp parallel { a = a + b; c = c + d; }

Task Assignment to Threads Work-sharing pragmas assign tasks to threads Pragma sections assigns non-iterative tasks void func() { int a, b, c, d; #pragma omp parallel { #pragma omp sections { #pragma omp section a = a + b; //one thread #pragma omp section c = c + d; //another thread }

Work-sharing Pragmas Work-sharing pragmas can be merged with pragma parallel void func() { int a, b, c, d; #pragma omp parallel sections { #pragma omp section a = a + b; //one thread #pragma omp section c = c + d; //another thread } #pragma omp parallel sections { #pragma omp section do_compress(); #pragma omp section do_encrypt(); } Two functions execute in parallel: #pragma omp parallel sections implements Task-Level Parallelism!

Work-sharing for Loops Pragma for assigns loop iterations to threads There are many ways to do this… By the way, what does the example do?? void func() { int a[N], sum = 0; #pragma omp parallel for for (int i = 0; i < N; i++) { sum += a[i]; } #pragma omp parallel for implements Data-Level Parallelism!

Variable Scope Rules Implicit Rule 1: All variables defined outside omp parallel are shared by all threads Implicit Rule 2: All variables defined inside omp parallel are local to every thread Implicit Exception: In omp for, the loop counter is always local to every thread Explicit Rule 1: Variables listed in shared() clause are shared by all threads Explicit Rule 2: Variables listed in private() clause are local to every threads

What’s private, what’s shared? void func() { int a, i; #pragma omp parallel for \ shared(c) private(d, e) for (i = 0; i < N; i++) { int b, c, d, e; a = a + b; c = c + d * e; }

Synchronization Problem sum++; mov eax, [sum] add eax, 1 mov [sum], eax mov eax, [sum] add eax, 1 mov [sum], eax s s+1 s s s thread 1 executes thread 2 executes thrd 1’s eax thrd 2’s eax s sum t+0 t+1 t+2 clks s+1 C code CPU instructions mov [sum], eax s+1 t+3 Not what we expected!

Synchronization Pragmas #pragma omp single – execute next operator by one (random) thread only #pragma omp barrier – hold threads here until all arrive at this point #pragma omp atomic – executes next memory access op atomically (i.e. w/o interruption from other threads) #pragma omp critical [ name ] – let only one thread at a time execute next op int a[N], sum = 0; #pragma omp parallel for for (int i = 0; i < N; i++) { #pragma omp critical sum += a[i]; // one thread at a time }

omp for schedule Schemes schedule clause defines how loop iterations assigned to threads Compromise between two opposite goals: Best thread load balancing With minimal controlling overhead N N/2 C f single thread schedule(static) schedule(dynamic, c) schedule(guided, f)

Reduction Can we do smarter than synchronizing all accesses to one common accumulator? Alternative: private accumulator in every thread to produce the final value, all private accumulators summed up (reduced) in the end Reduction ops: +, *, -, &, ^, |, &&, || int a[N], sum = 0; #pragma omp parallel for reduction(+: sum) for (int i = 0; i < N; i++) { sum += a[i]; // no synchronization needed }

OpenMP ™ Quiz What do these things do?? omp parallel omp sections omp for omp parallel for private shared reduction schedule omp critical omp barrier

OpenMP ™ Pragma Cheatsheet Fundmental pragma, starts team of threads omp parallel Work-sharing pragmas (can merge w parallel) omp sections omp for Clauses for work-sharing pragmas private, shared, reduction – variable’s scope schedule – scheduling control Synchronization pragmas omp critical [name] omp barrier

What Else Is in OpenMP ™ ? Advanced variable scope and initialization Advanced synchronization pragmas OpenMP ™ API and environment variables control number of threads manual low-level synchronization… Task queue model of work-sharing Intel-specific extension until standardized Compatibility with Win32* and P-threads 

Task Queue Work-sharing For more complex iteration models beyond counted for-loops struct Node { int data; Node* next; }; Node* head; for (Node* p = head; p != NULL; p = p->next) { process(p->data); } Node* p; #pragma intel omp taskq shared(p) for (p = head; p != NULL; p = p->next) //only 1 thrd { #pragma intel omp task process(p->data); // queue task to be executed } // on any available thread

Related TechEd Sessions DEV490: Easy Multi-threading for Native Microsoft®.NET Apps With OpenMP ™ and Intel ® Threading Toolkit HOLINT02: Multithreading for Microsoft ®.NET Made Easy With OpenMP ™ and Intel ® Threading Toolkit HOLINT01: Using Hyper-Threading Technology to enhance your native and managed Microsoft ®.NET applications

Summary & Call to Action Hyper-Threading Technology increases processor performance at very low cost HT Technology is available on mass consumer desktop today To take advantage of HT Technology, multithread your application! Use OpenMP ™ to multithread your application incrementally and easily! Use Intel Threading Tools to achieve maximum performance

Resources  developer.intel.com/software/products/ compilers/  Intel ® Compiler User Guide

THANK YOU! developer.intel.com Please remember to complete your online Evaluation Form!!

Reference

Quest to Exploit Parallelism Level of Parallelism Exploited byTechnologyProblem InstructionMultiple, pipelined, out- of-order exec. units, branch prediction NetBurst™ microarch. (pipelined, superscalar, out- of-order, speculative) Utilization DataSIMDMMX, SSE, SSE2 Scope Task, ThreadMultiple processors SMPPrice

Watch HT in Action! In-Order PipelineOut-of-Order PipelineIn- Order

Intel ® Compiler Switches for Multithreading & OpenMP ™ Automatic parallelization /Qparallel OpenMP ™ support /Qopenmp /Qopenmp_report{0|1|2}

Key Terms Multiprocessing (MP) hardware technology to increase processor performance by increasing number of CPUs Hyper-Threading Technology (HT) hardware technology to increase processor performance by improving CPU utilization Multithreading (MT) software technology to improve software functionality & increase software performance by utilizing multiple (logical) CPUs

Community Resources Most Valuable Professional (MVP) Newsgroups Converse online with Microsoft Newsgroups, including Worldwide User Groups Meet and learn with your peers

evaluations evaluations

© 2003 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.