Fireship · In 100 Seconds

Nvidia CUDA
Put the GPU to Work

CUDA — Nvidia's parallel computing platform — lets developers use the GPU for far more than video games: from the hardware built for massive parallelism to a first CUDA program in C++, this is the platform behind modern AI training.

Source: "Nvidia CUDA in 100 Seconds" — Fireship (youtube.com/watch?v=pPStdjuYzSI)

Agenda

What We'll Cover

The Five Sections

  • What Is CUDA? — platform & origins
  • Why GPUs are built for parallel work
  • How CUDA works: host, device & execution
  • Building your first CUDA application
  • Where to go next — NVIDIA GTC

What You'll Take Away

  • A mental model of the CUDA platform
  • Why 16,000+ GPU cores matter for parallel work
  • The five-step host/device flow
  • Kernels, grid coordinates & the launch syntax
  • A minimal demo: 256 threads in parallel

Section 1 · What Is CUDA?

Compute Unified Device Architecture

CUDA is a parallel computing platform that lets developers put the GPU to work on far more than video games — computing over large blocks of data in parallel.

Born at Nvidia, 2007

Developed by Nvidia in 2007, building on prior work of Ian Buck and John Nickolls. CUDA unlocked the true potential of the deep neural networks behind artificial intelligence.

Raw GPU Power, On Tap

A CUDA kernel lets a developer tap directly into the GPU's raw power — and data scientists around the world are using it right now to train the most powerful machine learning models.

Section 2 · Why GPUs Win at Parallel Work

The Generalist vs the Specialist

Play a game at 1080p and 60 FPS and over two million pixels may each need recalculating every frame — matrix multiplication and vector transformations, demanded in parallel.

CPU — the generalist

24 cores on an Intel Core i9 — each one powerful.

  • Versatile: handles whatever task you throw at it
  • A handful of strong cores, not thousands

GPU — the specialist

Over 16,000 cores on an RTX 4090 — each one simple.

  • One thing, done really well: go fast in parallel
  • Throughput measured in teraflops — trillions of floating-point operations per second

CUDA exists to give developers a way to harness all of those cores.

Section 3 · How CUDA Works

Host & Device: The Five-Step Handshake

CUDA splits the work between two processors: the host (your CPU) and the device (the GPU).

Write a CUDA kernel — a function that runs on the GPU Copy input data into GPU memory Launch the kernel — the CPU tells the GPU to run it in parallel Execute: blocks of threads on a multi-dimensional grid Copy the result back to main memory
Kernel first, data over, launch, execute, results back — every CUDA program is this handshake.

Section 3 · How CUDA Works (cont.)

Grids, Blocks & Threads

The GPU executes a kernel in units called blocks, and blocks organize threads into a multi-dimensional grid.

Grid — multi-dimensional array of blocks Block — unit of execution on the GPU Thread — runs one invocation of the kernel

The key idea: the grid mirrors the data

A multi-dimensional grid lets the arrangement of threads mirror the shape of the data being processed — and that matters when the data is itself multi-dimensional, like the tensors used in deep learning.

Section 4 · Building a CUDA App

The Kernel: A Function on the GPU

NVIDIA GPU CUDA toolkit — drivers, runtime, compilers, dev tools C++ (demo built in Visual Studio) 
__global__ void add(int *A, int *B, int *C) {
  int index = blockIdx.x * blockDim.x + threadIdx.x;
  C[index] = A[index] + B[index];
}

__global__ marks a kernel

It defines a function that runs on the GPU — this one adds two vectors, taking pointers A and B and writing the result to C.

Thread coordinates

blockIdx — which block · blockDim — how big the blocks are · threadIdx — position inside its block.

The kernel may execute billions of times in parallel, so each invocation computes its own index and writes its own distinct element.

Section 4 · Building a CUDA App (cont.)

Managed Memory & the Triple-Bracket Launch

Managed memory — cudaMallocManaged

Allocated data can be accessed from both the host CPU and the device GPU — no manual copying between them. CPU-side main() fills A and B in a for loop, ready for the GPU.

One block of 256 threads

add<<<1, 256>>>(A, B, C);
cudaDeviceSynchronize();

The triple brackets configure the launch: how many blocks and how many threads per block. Getting that config right is crucial for optimizing multi-dimensional data — like the tensors used in deep learning.

cudaDeviceSynchronize() pauses the CPU-side code until the GPU work is complete — then the result is available back on the host. Compile with nvcc, and congratulations: you just ran 256 threads in parallel.

Key Takeaways

The Complete Checklist

CUDA — Compute Unified Device Architecture — Nvidia's parallel platform, introduced in 2007
GPUs are hardware built for parallelism: 24 CPU cores vs 16,000+ GPU cores
Host (CPU) + device (GPU): kernel → data in → launch → execute → results back
Blocks of threads arranged on a multi-dimensional grid that mirrors the data
__global__ marks a kernel; each thread computes its own global index
cudaMallocManaged: data shared between CPU and GPU, no manual copies
cudaDeviceSynchronize: host waits until GPU work finishes
A minimal program: one kernel + launch config + sync — 256 threads in parallel

Where to go next — NVIDIA GTC

NVIDIA's GTC conference comes up every year, is free to attend virtually, and features talks on building massive parallel systems with CUDA.

The End

Go Compute
in Parallel

One kernel, a triple-bracket launch, one sync call — and 256 threads are running at once. It's the same pattern data scientists use right now to train the most powerful machine learning models.

Source: "Nvidia CUDA in 100 Seconds" — Fireship (youtube.com/watch?v=pPStdjuYzSI) · Full article in this repo.

← → to navigate · swipe on mobile