1. Computer Engg, IIT(BHU)
CUDA-1
3/12/2013
Computer Engg, IIT(BHU)

2. CUDA is a set of developing tools to create applications that will perform execution on GPU (Graphics Processing Unit).
CUDA compiler uses variation of C with future support of C++
 CUDA was developed by NVidia and as such can only run on NVidia GPUs of G8x series and up.

3. GPU
It is a processor optimized for 2D/3D graphics, video, visual computing, and display.
It is highly parallel, highly multithreaded multiprocessor optimized for visual computing.
It provide real-time visual interaction with computed objects via graphics images, and video.

4. GPU
It serves as both a programmable graphics processor and a scalable parallel computing platform.
Heterogeneous Systems: combine a GPU with a CPU

5. GPU-Evolution 1980’s – No GPU. PC used VGA controller
1990’s – Add more function into VGA controller
1997 – 3D acceleration functions:
Hardware for triangle setup and rasterization Texture mapping Shading
2000 – A single chip graphics processor ( beginning of GPU
term)
2005 – Massively parallel programmable processors
2007 – CUDA (Compute Unified Device Architecture)

6. GPU Graphic Trend OpenGL – an open standard for 3D programming
DirectX – a series of Microsoft multimedia programming interfaces
New GPU are being developed every 12 to 18 months
New idea of visual computing:
combines graphics processing and parallel computing
Heterogeneous System – CPU + GPU

7. GPU Graphic Trend GPU evolves into scalable parallel processor
GPU Computing: GPGPU and CUDA
GPU unifies graphics and computing
GPU visual computing application: OpenGL, and DirectX

8. Why CUDA
CUDA provides ability to use high-level languages such as C to develop application that can take advantage of high level of performance and scalability that GPUs architecture offer. 
 GPUs allow creation of very large number of concurrently executed threads at very low system resource cost

9. Why CUDA
CUDA also exposes fast shared memory (16KB) that can be shared between threads. 
 Full support for integer and bitwise operations.
Compiled code will run directly on GPU

10. CUDA programming Model
The GPU is seen as a compute device to execute a portion of an application that
Has to be executed many times
Can be isolated as a function
Works independently on different data
Such a function can be compiled to run on the device. The resulting program is called a Kernel

11. CUDA Programming Model
The batch of threads that executes a kernel is organized as a grid of thread blocks

12. CUDA Programming Model
Thread Block
Batch of threads that can cooperate together
Fast shared memory
Synchronizable
Thread ID
Block can be one-, two- or three-dimensional arrays

13. CUDA Programming Model
Grid of Thread Block
Limited number of threads in a block
Allows larger numbers of thread to execute the same kernel with one invocation
Blocks identifiable via block ID
Leads to a reduction in thread cooperation
Blocks can be one- or two-dimensional arrays

14. CUDA Programming Model

15. CUDA Memory Model

16. CUDA Memory Model Shared Memory Is on-chip:
much faster than the local and global memory,
as fast as a register when no bank conflicts,
divided into equally-sized memory banks.
Successive 32-bit words are assigned to successive banks,
Each bank has a bandwidth of 32 bits per clock cycle.

17. CUDA Memory Model Shared Memory
memory request requires two cycles for a warp
One for the first half, one for the second half of the warp
No conflicts between threads from first and second half

18. CUDA API An Extension to the C Programming Language
Function type qualifiers to specify execution on host or device
Variable type qualifiers to specify the memory location on the device
A new directive to specify how to execute a kernel on the device
Four built-in variables that specify the grid and block dimensions and the block and thread indices

19. CUDA API Function type qualifiers __device__ __global__ __host__
Executed on the device
Callable from the device only.
__global__
Executed on the device,
Callable from the host only.
__host__
Executed on the host,

20. CUDA API Variable Type Qualifiers __device__
Resides in global memory space,
Has the lifetime of an application,
Is accessible from all the threads within the grid and from the host through the runtime library.
__constant__ (optionally used together with __device__)
Resides in constant memory space,
__shared__ (optionally used together with __device__)
Resides in the shared memory space of a thread block,
Has the lifetime of the block,
Is only accessible from all the threads within the block.

21. CUDA API Execution Configuration (EC)
Must be specified for any call to a __global__ function.
Defines the dimension of the grid and blocks
specified by inserting an expression between function name and argument list: function:
__global__ void Func(float* parameter);
must be called like this:
Func<<< Dg, Db, Ns >>>(parameter);

22. CUDA API Execution Configuration (EC) Where Dg, Db, Ns are :
Dg is of type dim3  dimension and size of the grid
Dg.x * Dg.y = number of blocks being launched;
Db is of type dim3  dimension and size of each block
Db.x * Db.y * Db.z = number of threads per block;
Ns is of type size_t  number of bytes in shared memory that is dynamically allocated in addition to the statically allocated memory
Ns is an optional argument which defaults to 0.

23. CUDA API Built-in Variables
gridDim is of type dim3 dimensions of the grid.
blockIdx is of type uint3  block index within the grid.
blockDim is of type dim3  dimensions of the block.
threadIdx is of type uint3  thread index within the block.