LLM Fine-Tuning Explained · Aishwarya Srinivasan

LLM Fine-Tuning
The Complete Guide

Where fine-tuning sits relative to pre-training, how open-weight and closed-weight models differ, the major methodologies — parameter-efficient, full, and reinforcement-based fine-tuning — and when to reach for each one.

Source: youtube.com/watch?v=Wx1oiBCmxjY · Channel: Aishwarya Srinivasan (@aishwaryasrinivasan)

Agenda

What We'll Cover

Foundations

Pre-training vs post-training — where fine-tuning fits
Scale: the parameter universe & weight access
PEFT — LoRA & QLoRA
Full fine-tuning

Methods & Practice

Reinforcement fine-tuning — verifiable rewards, RLHF, DPO
Choosing the right method — prompts, context, then the model
Data quality & evaluation — the real challenges
Key takeaways checklist

Section 1 · Where Fine-Tuning Fits

You Adapt — You Don't Train From Zero

Pre-training — the foundation labs' job

Massive runs on huge GPU clusters over trillions of tokens.

Produces the base model — Llama, GPT, Claude, Gemini
The thing you get out of the box: download or API
You did not train it — the foundation labs did

Post-training — everything after

Fine-tuning is the most common form of post-training.

Not a new concept — close to what traditional ML teams have done for years
Adapts a model already trained on a great deal of knowledge

Before LLMs: you owned the pipeline

Train a model from scratch on your own data
Fine-tune it on task-specific data
Ship your own model

Today: model-as-a-service

Base model served to you by download or API
Fine-tune on your specific use case
Profound implications for the compute, data & expertise required

Section 2 · Fine-Tuning at Scale

From Millions to Over a Trillion Parameters

Traditional ML — hundreds of thousands to a few million Small open models — a few billion Production models — 10 to 70 billion Frontier models — hundreds of billions Models like GLM — over a trillion

Why scale is the core challenge

Training memory grows with the number of trainable parameters
The run must hold weights, gradients and optimizer state
Even a fraction of a trillion-parameter model needs serious infrastructure

Every method discussed in this guide is, in one way or another, an answer to this problem of scale.

Section 2 (cont.) · Weight Access

Open-Weight vs Closed-Weight Models

Open-weight — full control

Llama · Mistral · Qwen · DeepSeek · GLM and others

Weights are open — you can change the parameters through fine-tuning
Download the weights, fine-tune on your own GPUs
Deploy the fine-tuned model yourself

API providers hosting these models are really just distributing the weights across GPUs.

Closed-weight — provider fine-tuning only

Claude · GPT · Gemini

No access to the weights — cannot train from your own infrastructure
Use the provider's native service: OpenAI's API, Vertex AI, Anthropic
Submit data → provider runs the job → access the fine-tuned version via API

Trade-off: no visibility into how the fine-tuning happens

Section 3 · Parameter-Efficient Fine-Tuning (PEFT)

Do You Need to Update All 70B Parameters?

PEFT is the first big bucket of fine-tuning methods — and the place most people get confused. It is where LoRA and QLoRA live.

Do I really need to update every single parameter in a 70-billion-parameter model just to make it better at my task?

The answer all of PEFT is built around

Usually no. Instead of updating billions of parameters, PEFT keeps the frozen base model and learns a compact adjustment on top of it.

LoRA — low-rank adaptation QLoRA — quantized low-rank adaptation

Section 3 · PEFT — LoRA

LoRA: Train 100 Million, Not 70 Billion

Freeze the original model weights Inject small trainable matrices into specific layers Train ~100 million parameters instead of 70 billion Merge the learned update into the frozen weights Inference with no extra latency or memory overhead

The "low rank" in the name

For each layer it touches, LoRA learns a small pair of matrices whose product forms the update — living in a much smaller dimensional space than the full weight matrix it adjusts.

What that buys you

A massive reduction in compute, memory and cost
The base model's knowledge stays intact and untouched
Adapters are small — store, swap, compare or delete them
Several fine-tunes of one base, no extra full copies

Section 3 · PEFT — QLoRA

QLoRA: A 70B Model on a Single H100

Quantize the frozen base model to 4-bit precision Run LoRA on top of the quantized base Keep the small trainable adapters at higher precision Fine-tune a 70B model on a single H100

Quantization is compression

Base weights are stored at much lower numerical precision, shrinking their memory footprint dramatically — at some cost in precision.

Compressed base, but adapters still learned precisely

The default starting point

QLoRA is the default starting point for most teams fine-tuning their own open-weight models today.

Start fast: Hugging Face's PEFT library

Section 4 · Full Fine-Tuning

Full Fine-Tuning: Most Flexibility, Highest Cost

The opposite approach — you update every single parameter in the model.

Most flexibility Usually the best quality — nothing is held fixed Expensive Slow Serious GPU infrastructure — billions or trillions of parameters updated

Reach for full fine-tuning only when PEFT is not giving you the quality bar you need — which is very rare in practice. For most teams, LoRA or QLoRA is more than enough.

Section 5 · Reinforcement Fine-Tuning

Three Flavors of Reinforcement Fine-Tuning

The category that has exploded in the last year — spanning classic human-preference alignment and today's reasoning models.

Verifiable rewards

Reward answers that can be checked automatically.

Powers OpenAI's o-series, DeepSeek R1 and most modern reasoning models.

RLHF

Human preferences → reward model → PPO.

The classic technique that powered ChatGPT — heavy data and a complex loop.

DPO

Skip the reward model entirely.

Optimizes preference pairs directly — easier, less compute, the open-weight default.

Section 5.1 · Verifiable Rewards

Reward the Reasoning That Works

Model samples several candidate solutions — often full reasoning chains Each candidate is checked against the ground truth or a test suite Correct reasoning paths are rewarded; wrong ones are pushed down Over time, the model learns to reason better

No human labelers at any point — that is what makes the loop scalable Powerful when the answer can be checked automatically: math, code with tests Behind OpenAI o-series · DeepSeek R1 · modern reasoning models

Section 5.2 · RLHF

RLHF: Steering the Model With Preferences

Reinforcement learning from human feedback — the classic technique that powered ChatGPT.

Collect pairs or rankings of model responses Human annotators say which responses they prefer Train a reward model to predict those preferences PPO uses the reward-model signal to update the language model

Expensive: a lot of high-quality human preference data Complex training loop Strong steer toward preferred style, safety and tone

PPO = proximal policy optimization, the core RL algorithm used inside RLHF.

Section 5.3 · DPO

DPO: Skip the Reward Model

Direct preference optimization — instead of training a reward model and optimizing against it, DPO optimizes the language model on the preference pairs in a single step.

	RLHF	DPO
Reward model	Trained separately on human preferences	Skipped entirely
Optimization	PPO steers the language model with reward-model signals	Direct optimization on the pairs in a single step
Cost & complexity	Heavy human data, complex loop	Much easier to implement, less compute
Typical role	Classic alignment, ChatGPT era	Default for preference tuning on open-weight models

Section 6 · Choosing the Right Method

Fine-Tuning Isn't the First Lever

Optimize your prompts — instructions, examples, structure Optimize your context — documents, retrieval, tool outputs Still a gap? Now fine-tune

Prompts & context are your benchmark

Both steps give a decent performance bump on their own. The gap that remains is exactly what fine-tuning has to close — so you can measure the improvement it actually delivers.

Match the method to your goal

Domain task or tone → start with QLoRA
QLoRA short of the quality bar (rare) → full fine-tuning
Style, safety, tone → RLHF or DPO
Reasoning on auto-verifiable tasks → verifiable rewards

Section 7 · The Real Challenges

Bad Data Makes a Good Model Worse

Fine-tuning with a bad dataset on a good model makes the model bad — not better.

A "bad" dataset does not only mean wrong answers:

Wrong answers Examples that misrepresent the real task Inconsistent formats → contradictory habits Examples that contradict what the model already knows Too few high-quality samples to steer behavior reliably

A model faithfully absorbs whatever patterns its training data contains — bad patterns become bad behavior. No clever parameter-efficient machinery or RL sophistication can compensate.

Section 7 (cont.) · Evaluation

Define the Yardstick Before You Start

The second biggest challenge: without a proper eval set and a clear metric in place first, you will have no idea whether fine-tuning improved anything.

A proper eval set

A representative, held-out collection of examples that stand for the real task — separate from anything used in training.

A clear metric

A measure you commit to in advance, one that matches the outcome you actually care about.

Treat fine-tuning like an experiment with a defined baseline and a defined yardstick — not a blind API call.

Key Takeaways

What to Remember

Fine-tuning is the most common form of post-training — adapt, don't train from scratch

Scale is the core challenge: billions to over a trillion parameters

Open weights → fine-tune on your own GPUs; closed weights → provider-native services only

Start with PEFT: LoRA trains ~100M injected parameters; QLoRA adds 4-bit — 70B on one H100

Full fine-tuning offers maximum flexibility and quality — and is rarely worth its cost

Three RL flavors: verifiable rewards, RLHF (reward model + PPO), DPO (direct pairs)

Optimize prompts and context first — they benchmark the gap fine-tuning must close

Data quality and evaluation are the two biggest failure points

The End

Fine-Tune With Understanding

Fine-tuning adapts an already pre-trained model to your use case — optimize prompts and context first, start with PEFT, escalate deliberately, and treat every run as an experiment with a baseline and a yardstick.

Source: "LLM Fine-Tuning Explained: The Complete Guide" — Aishwarya Srinivasan (youtube.com/watch?v=Wx1oiBCmxjY · @aishwaryasrinivasan) · Full article in this repo.

LLM Fine-TuningThe Complete Guide