Mixture of Experts: Huge Models That Only Use a Slice of Themselves

A trillion-parameter model doesn't have to run a trillion parameters on every word. Mixture-of-experts architectures activate only a fraction of themselves per token - and that's the whole trick.

The basic idea

A mixture-of-experts (MoE) model is built from many separate "expert" subnetworks instead of one giant uniform network. For every token it processes, a small router decides which handful of experts get used - the rest sit idle. Total parameters describes the whole model as stored on disk; active parameters describes what runs for any given token.

Why this matters practically

Only active parameters need computing per token, so an MoE model can have a total parameter count many times larger than a dense model while costing roughly what a much smaller dense model costs to run, since most of it isn't touched on any given step. That's how models get well over a trillion total parameters while staying fast to serve.

Real examples

DeepSeek-V3 has 671 billion total parameters but activates only 37 billion per token, via 8 of 256 experts. Mixtral, an earlier open MoE model, uses 8 experts with 2 active per token. Kimi K3, released by Moonshot AI in mid-2026, goes further: 2.8 trillion total parameters across 896 experts, only 16 active per token - under 2% running at once.

The catch

All those parameters still need loading into memory, even idle ones, so MoE saves compute per token, not memory footprint. You still need serious hardware to host the model, just not to run every part of it on every request.

EXAMPLE

Total vs. active parameters, three models: - Mixtral: 8 experts, 2 active per token - DeepSeek-V3: 671B total, 37B active (8 of 256 experts) - Kimi K3: 2.8T total, ~16 of 896 experts active per token A request to any of these only pays compute for the active slice, not the full total.

🎬 AS A SHORT VIDEO

Mixture of Experts: Huge Models That Only Use a Slice of Themselves

🛠️ EXERCISE — TRY IT YOURSELF

Compare the total vs. active parameter counts of a few current MoE models and see how that relates to their pricing.

Look up the total and active parameter counts for two or three MoE models (e.g. DeepSeek-V3, Mixtral, Kimi K2 or K3).
Look up the per-token API pricing for the same models where available.
Also note the parameter counts of a dense model you're familiar with for comparison.
Check whether models with a lower active-parameter count tend to be priced lower, even if their total parameter count is huge.
Write one sentence explaining, in your own words, why total parameters alone doesn't predict how expensive a model is to run.

✅ SELF-CHECK

☐ Can you name the total and active parameter counts for at least two MoE models from memory now?
☐ Can you explain why a 2.8T-total model can be cheaper to run than a 70B dense model?

QUICK QUIZ

In a mixture-of-experts model with 2.8T total parameters and only 16 active experts per token, what does 'total parameters' mainly describe?