Quantization: Squeezing Model Weights Down to Fit

A model's weights don't have to stay in full precision. Storing them in fewer bits per number shrinks a model enough to run on a laptop - at some cost to quality.

What's actually being reduced

A model's weights - the huge set of numbers that define what it knows - are normally stored in a fairly high-precision format, like 16-bit floating point (FP16). Quantization converts those numbers to a lower-precision format, like 8-bit or 4-bit integers (INT8, INT4), so each weight takes a fraction of the space it used to.

Why that matters

Lower precision means a smaller file and less memory to load and run the model. A model needing 32GB of VRAM at full precision might fit in 8-9GB after aggressive quantization - the difference between needing a data-center GPU and running on a gaming laptop.

The tradeoff

Fewer bits per number means less precision, and past a point that shows up as real quality loss - subtly worse reasoning, more mistakes on hard tasks, sometimes broken output at extreme settings. How much you lose depends on how aggressive it is: 8-bit is close to indistinguishable from full precision, 4-bit noticeably more of a compromise, lower gets risky fast.

GGUF: the common local format

GGUF is a file format built for storing quantized models efficiently, widely used by local-inference tools. A base model is often distributed as a family of GGUF files at different quantization levels, so you pick the size/speed vs. quality tradeoff that fits your hardware.

EXAMPLE

ollama pull command using a specific quantization tag: $ ollama pull llama3.1:8b-instruct-q4_K_M Same base model, different files: - q8_0 ~8.5GB, closest to full quality - q4_K_M ~4.9GB, common sweet spot - q2_K ~3.2GB, smallest, most quality loss

🛠️ EXERCISE — TRY IT YOURSELF

Download and compare two quantization levels of the same local model.

Pick a small open model available in GGUF format (e.g. via Ollama's library).
Pull two versions with different quantization tags, e.g. a q8_0 and a q4_K_M variant.
Check the file sizes of both and note the difference.
Run the same prompt - ideally something with a precise answer, like a small math or logic question - on both.
Compare the answers and, if possible, the response speed on your hardware.

✅ SELF-CHECK

☐ Did the smaller quantization noticeably reduce file size and/or memory use compared to the larger one?
☐ Did you notice any quality difference in the answers, especially on the more precise prompt?

QUICK QUIZ

What does quantizing a model's weights from FP16 to INT4 primarily trade away?