Fireship · In 100 Seconds

Nvidia CUDA
in 100 Seconds

CUDA is NVIDIA's parallel computing platform — the technology that lets developers put the GPU to work on far more than video games, and the engine behind modern AI training.

Source: youtube.com/watch?v=pPStdjuYzSI · Channel: Fireship

Agenda

What We'll Cover

The Platform

  • What is CUDA? — from graphics chip to AI workhorse
  • Why GPUs are built for parallel work — CPU vs GPU

The Programming Model

  • How CUDA works: host, device & execution flow
  • Building your first CUDA application in C++
  • Where to go next

Section 1 · What Is CUDA?

From Graphics Chip to AI Workhorse

Definition

CUDA — Compute Unified Device Architecture — is NVIDIA's parallel computing platform.

Introduced by NVIDIA in 2007 and built on the prior work of Ian Buck and John Nickolls, it lets developers use the GPU for general-purpose computation — computing over large blocks of data in parallel.

What it unlocked

  • A CUDA kernel lets a developer tap directly into the GPU's raw power
  • Unlocked the true potential of the deep neural networks behind AI
  • Data scientists around the world are using it right now to train the most powerful machine learning models
Before CUDA: the GPU is a graphics engine — frames for games 2007: NVIDIA introduces CUDA Kernel taps raw GPU power Compute in parallel over big data AI training unlocked

Section 2 · Why GPUs Are Built for Parallel Work

The Same Math, Millions of Times

The GPU was historically used for exactly what its name implies: computing graphics. That job shaped the hardware — and the hardware is what CUDA harnesses.

1080p at 60 FPS Over two million pixels on screen Each pixel may be recalculated every frame Lots of matrix multiplication Vector transformations in parallel
That demands hardware capable of doing a lot of matrix multiplication and vector transformations in parallel — and "a lot" is an understatement.

Section 2 · Two Kinds of Cores

24 Powerful Cores vs 16,000+ Simple Ones

CPU — the generalistGPU — the specialist
Cores24 cores, each powerful — e.g., Intel Core i916,000+ cores, each simple — e.g., RTX 4090
Work styleDifferent tasks, one at a timeOne operation on every pixel at once
Design goalVersatile — runs anything you throw at itOne thing done really well: go fast in parallel
Throughput—Measured in teraflops — trillions of floating-point operations per second
CUDA exists to give developers a way to harness all of those cores.

Section 3 · How CUDA Works

Host and Device: The Five-Step Flow

CUDA splits the work between two processors: the host (your CPU) and the device (the GPU).

Write a kernel — a function that runs on the GPU Copy input data into GPU memory Launch the kernel Execute in parallel — blocks of threads on a grid Copy the result back to main memory

Host — the CPU

Runs the main program, launches the kernel, and coordinates with the device.

Device — the GPU

Executes the kernel in units called blocks, which organize threads into a multi-dimensional grid.

Section 3 · The Execution Model

Threads Arranged Like the Data

Blocks of threads on a grid

The GPU runs a kernel as a multi-dimensional grid of blocks; each block is a group of threads.

In the demo launch add<<<1, 256>>>(A, B, C), the grid is one block of 256 threads.

Why multi-dimensional?

The arrangement of threads can mirror the shape of the data being processed — which matters when that data is itself multi-dimensional, such as the tensors used in deep learning.

Getting the launch configuration right is crucial for optimizing multi-dimensional data structures.

Section 4 · Building Your First CUDA Application

The Kernel: A Function That Runs on the GPU

An NVIDIA GPU CUDA toolkit — drivers, runtime, compilers, developer tools C++ — the demo is built in Visual Studio

The code

__global__ void add(int *A, int *B, int *C) {
  int index = blockIdx.x * blockDim.x + threadIdx.x;
  C[index] = A[index] + B[index];
}

What it means

  • __global__ defines a kernel — a function that runs on the GPU
  • Pointers A and B are the two vectors to add; C points to the result
  • The kernel may execute billions of times in parallel across threads

Section 4 · The Kernel Launch

Every Thread Finds Its Own Element

Because the kernel runs in parallel across many threads, each invocation first calculates the global index of the thread it is running on — so every thread writes to its own distinct element.

blockIdx.x — which block the thread belongs to blockDim.x — how large the blocks are threadIdx.x — its position within its block

Index formula

index = blockIdx.x * blockDim.x + threadIdx.x;
C[index] = A[index] + B[index];

Triple brackets configure the launch

add<<<1, 256>>>(A, B, C); controls how many blocks and how many threads per block run the code in parallel — here, one block of 256 threads.

Section 4 · Running the Program

Managed Memory, Sync, and 256 Threads

Managed memory — no manual copies

cudaMallocManaged tells CUDA the allocation can be accessed from both host CPU and device GPU — a convenience that skips the explicit copy step of the general flow. The CPU-side main uses a for loop to initialize the arrays.

Synchronize, then use the result

cudaDeviceSynchronize pauses CPU-side execution and waits until the GPU work is complete. When the kernel finishes, the data is available back on the host — ready to print to standard output, for instance.

cudaMallocManaged for-loop initialization on the CPU add<<<1, 256>>>(A, B, C) cudaDeviceSynchronize Compile and run with nvcc — the CUDA compiler

…and congratulations: you just ran 256 threads in parallel on your GPU.

Section 5 · Where to Go Next

Beyond the Intro: NVIDIA GTC

To go beyond this introduction, NVIDIA's GTC conference is the natural next step.

It comes up

Every year

Attendance

Free to attend virtually

Talks

Building massive parallel systems with CUDA

Data scientists around the world are using CUDA at this very moment to train the most powerful machine learning models.

Key Takeaways

The CUDA Mental Model

CUDA (Compute Unified Device Architecture) — NVIDIA's 2007 platform for general-purpose GPU computation, powering deep neural networks behind AI
GPUs are built for parallelism: 24 CPU cores vs 16,000+ GPU cores; throughput measured in teraflops
Work splits between host (CPU) and device (GPU): kernel, data over, launch, results back
Execution is organized into blocks of threads on a multi-dimensional grid — mapping onto data shapes like tensors
Kernels are marked __global__; each thread computes its own global index
cudaMallocManaged shares data CPU/GPU without manual copying; cudaDeviceSynchronize makes the host wait
A minimal program is one kernel + one triple-bracket launch + one sync — the demo runs 256 threads in parallel

The End

Go Parallel

CUDA turned the GPU from a graphics-only engine into the parallel computer behind AI training — and your first kernel is one function, one launch, and one sync call away.

Source: "Nvidia CUDA in 100 Seconds" — Fireship (youtube.com/watch?v=pPStdjuYzSI)

← → to navigate · swipe on mobile