Computer Use
Get work done in software built for people
Most of the tools people use at work were built to be used with a mouse and keyboard. An agent must read the screen, click and type, and keep its place over a long task.
RL environments for computer use agents
RL environments for computer use agents are environments built on the kinds of software people use at work, with real data behind them. An agent reads the screen, clicks and types, and is rewarded for finishing the job. The same environment can serve agents that work from screenshots and browser agents that read the page structure.
Tasks don’t tell the agent which buttons to press. It has to find its way, keep its place over dozens of steps and make the choices an expert would. Models still fall short on open-ended design work. In FigmaTrace, a 27B open model trained on expert Figma recordings improved on five of six GUI agent benchmarks outside design.
We also capture how experts work. We record experts as they work and turn the recordings into step-by-step trajectories for training. FigmaTrace is our open dataset of 3,469 Figma design trajectories from more than 200 hours of expert work.
Much computer work has no single right answer. Tasks are written and checked by people who do that work.
Published research
FigmaTrace
Human design workflows in Figma, captured as training data for creative design tasks.
BLUR
573 tip-of-the-tongue questions that need search, reasoning and tool use.
Questions about computer use agent training
What is a computer use agent?
It is an AI agent that operates software through its interface, the way a person does, by reading the screen, clicking and typing. It can work in tools that have no API.
What training data do computer use agents need?
Two kinds help. Environments let an agent practice and earn a reward. Expert trajectories show how a skilled person moves through a task, including the drafts and the backtracking. FigmaTrace is a public example of the second.
Can you turn screen recordings into training data?
Yes. We record experts as they work and convert the recordings into trajectories. In FigmaTrace, splitting long recordings by design phase worked better than cutting them to fit a context length.
How realistic are the environments?
Realistic enough that an expert can do the task the way they would at work. Experts complete tasks by hand to confirm each one can be solved before it is used.
How do you handle personal data?
Private details in our environments are made up for the task. We also check what we deliver for personal data and credentials.