Running Open Models Locally (Ollama & Co.)

What Ollama does

Ollama is an open source program that downloads and runs open language models like Gemma, Qwen, DeepSeek, or gpt-oss locally on your machine. A command like ollama run gemma4 downloads the model the first time and then starts a chat right in your terminal – no cloud, no API key, no per-request running costs.

What's realistic

Small models (a few billion parameters) run fine on ordinary laptop hardware using just the CPU, though slower than in the cloud. For bigger models, Ollama automatically uses a GPU if one is available – supporting NVIDIA cards, AMD Radeon via ROCm, Apple Silicon via Metal, and additionally Intel and other AMD cards via Vulkan. If a model doesn't fit entirely in GPU memory, Ollama automatically splits it between GPU and CPU.

The limits versus cloud models

Local open models are smaller than the largest cloud models and usually fall short on complex reasoning, long context, or specialist knowledge. Memory needs grow with model size and context length; a model that's too big simply won't load, or crawls painfully slowly on CPU.

When it's worth it

Local models fit privacy-sensitive tasks, offline work, cost-free experimentation, and simple, repetitive jobs. For demanding coding or hard reasoning, cloud models like Claude usually stay ahead.