This chapter describes the modern the graphics processing unit (GPU) architecture. It focuses on nVidia's GPU architecture and CUDA programming language. Computer graphics has become very popular, especially in the video games and simulation community. Graphics hardware first came to use in 1980s, though they were called GPUs much later. GPU is a term introduced by nVidia in 1999. The CUDA programming model is a parallel programming model that provides an abstract view of how the processes can be run on underlying GPU architectures. There is a memory hierarchy used by CUDA and supported by the streaming multiprocessor and GPU architectures. The GPU evolution got a boost from the release of the graphics industry's most widely used application programming interface of Silicon Graphics Inc (SGI)-GL by SGI. A faster communication channel between CPU and GPU allowed even higher performance. The modern GPU offers tremendous potential to solve largely parallelizable general purpose problems using the GPU.