Optimizing Latency: Making Agents Feel Fast

Latency is response time, not cost - and the levers that shrink one don't always shrink the other. Here's what actually moves the needle on how fast an agent feels.

Latency vs. cost - not the same lever

It's easy to conflate "cheaper" with "faster," but they're separate axes. A pricier model with a short prompt can respond faster than a cheap model under a huge context. This entry is about response time - see cost control for the money side.

Model choice

Smaller, lighter models respond faster per token, since there's less computation per pass. Switching from claude-sonnet-5 to claude-fable-5 for tasks that don't need frontier reasoning cuts latency directly.

Capping the thinking budget

Reasoning models spend a variable amount of time "thinking" before answering. An explicit budget_tokens cap trades reasoning depth for a hard latency ceiling - useful when a task doesn't need exhaustive deliberation.

Prompt caching

Reusing an unchanged prefix from the cache skips redoing that computation. This mainly cuts the delay before first tokens appear, not the total generation time once underway.

Streaming for perceived speed

Streaming doesn't make the model faster - it makes the wait feel shorter, since tokens appear as generated instead of all at once. For a chat interface, that's often what matters to a user's sense of speed.

Parallelization and short context

Running independent sub-tasks concurrently cuts wall-clock time even though each call takes just as long. A shorter context also processes faster - trimming unneeded history or files pays off directly.

EXAMPLE

Same task, three latency configurations: Baseline: large model, no caching, full 8,000-token system prompt resent every call, no thinking cap, non-streamed. -> slow first token, slow total, feels sluggish even mid-generation. Optimized for perceived speed: same model, streaming turned on. -> same total time, but first tokens appear almost immediately - feels much faster to a user even though nothing computationally changed. Optimized for real speed: claude-fable-5 where the task allows it + prompt caching on the unchanging system prompt + `budget_tokens` capped at a modest ceiling + streaming on top. -> genuinely faster time-to-first-token AND faster total time, plus the perceived-speed benefit of streaming stacked on top. Rule of thumb table: | Lever | Helps TTFT | Helps total time | Helps perceived speed | |---|---|---|---| | Smaller/faster model | yes | yes | yes | | Prompt caching (cached part) | yes | partial | yes | | Thinking budget cap | yes | yes | yes | | Streaming | no | no | yes | | Parallelizing independent calls | no | yes (wall-clock) | yes | | Shorter context | yes | partial | yes |

🎬 AS A SHORT VIDEO

Optimizing Latency: Making Agents Feel Fast

🛠️ EXERCISE — TRY IT YOURSELF

Measure the actual difference between two latency levers on the same task.

Pick a repeatable task you can run twice with an unchanged, longish system prompt or instruction block.
Run it once with no caching and time from request to first token, and to completion.
Run it again with prompt caching enabled on the unchanged prefix, same task.
Compare time-to-first-token and total time between the two runs.
Separately, try the same task with streaming on vs. off and note how different the experience feels even if the total time is the same.

✅ SELF-CHECK

☐ Did caching noticeably reduce time-to-first-token on the second run?
☐ Could you tell the difference between 'actually faster' (caching, smaller model) and 'feels faster' (streaming) once you measured both?

QUICK QUIZ

What does streaming actually change about a model's response?