Computer Use: When AI Controls Mouse and Keyboard
Computer use lets a model see screenshots and control the mouse and keyboard — like a human at the screen, with real risks attached.
What computer use is
Normal tool use calls clearly defined functions. Computer use goes further: the model is shown screenshots of the screen and then controls the mouse and keyboard — clicking, typing, scrolling — like a human sitting at the computer. Anthropic introduced this capability in beta in October 2024 with Claude 3.5 Sonnet, the first major model to offer it.
How the cycle works
The model receives a screenshot image as context, decides where to click or what to type based on it, returns that action in structured form, the action gets executed, a new screenshot is taken — and the cycle starts again. This is the same tool_use/tool_result cycle as normal tool use, just with the screen instead of business logic as the "tool".
What it's good for
Computer use makes it possible to operate applications that have no API: legacy software, unusual interfaces, browser workflows. A browser agent can click through a website as if it were a human.
The limits
Anthropic itself warns: click coordinates and tool selection can be inaccurate, the model can hallucinate, and reliability drops on unfamiliar interfaces. The bigger danger: instructions hidden on a webpage or inside an image can manipulate the model (prompt injection) — with full screen access, the risk is higher than with plain text.
EXAMPLE
A browser agent gets the task 'find the cheapest flight from Berlin to Rome'. It takes a screenshot, sees a search field, clicks into it, types 'Berlin Rome', takes a new screenshot, clicks the result with the lowest price, and so on — every step runs through screenshot -> action -> new screenshot.
QUICK QUIZ
According to Anthropic's own documentation, what is the biggest security risk with computer use?
SOURCES
- Anthropic Docs: Computer Use Tool ↗ platform.claude.com
- Anthropic: Introducing computer use (announcement) ↗ www.anthropic.com