Computer Use
Letting a model drive a real computer the way a person does — looking at the screen, moving the mouse, typing.
Think of it like
A remote assistant on a screen-share who can see your desktop and click around to get things done.
Example
Asked to file an expense, the agent takes a screenshot, finds the form fields, clicks them, types the amounts, and hits submit.
How it actually works
The model is fed screenshots and returns actions — click at these coordinates, type this, scroll — in a loop, letting it use any app without a dedicated API. It is powerful and brittle: pixel-level targeting misfires, pages shift, and a wrong click can do real damage. Because it can touch anything on screen, it is one of the higher-risk agent modes and usually runs behind confirmations.
For product teams
It promises automation of any app, but reliability and safety are the whole ballgame.
For engineers
A vision-driven agent loop that maps screenshots to GUI actions (click, type, scroll) via coordinates.
Related
- Browser Agent — A narrower version scoped to the web browser.
- Approval Gate — The high-stakes actions that argue for a human checkpoint.
- Observation — Its reliance on reading images.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome