Resources

Open GUI and Computer Use Agent Datasets in 2026

Patronus AI TeamAs of November 202612 min read

Patronus AI publishes this guide and created FigmaTrace, one of the datasets on it. Entries are not ranked, and each entry links its paper and dataset.

Training computer use agents? Read FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows →

Patronus AI created FigmaTrace and builds RL environments for computer use agents. On Jun 25, 2026, it announced a $50M Series B led by Greenfield Partners. As of November 2026, open datasets for GUI and computer use agents include FigmaTrace, AgentNet, AgentTrek, AndroidControl, Android in the Wild, GUIOdyssey, Mind2Web, OS-Atlas, the OSWorld-Verified trajectories, ScaleCUA-Data and WebLINX. They differ in platform, in whether they hold whole tasks or single steps, in who recorded them and in license. Pick the ones that match the software your agent will use, and read each license before you train on the data.

A computer use agent can learn from recordings of a task done on a screen, with what the screen showed at each step and the click, keystroke or scroll that came next. Benchmarks such as OSWorld test agents in real computer environments, and open datasets supply recorded steps to train on. This guide covers eleven open datasets. Patronus AI is a behavioral research lab that publishes at NeurIPS, ICML and ACL. Its FigmaTrace is an open dataset of 3,469 Figma design trajectories.

AgentNet and the OSWorld-Verified trajectories cover desktop software, and FigmaTrace covers design work in Figma. AgentTrek, Mind2Web and WebLINX cover websites. AndroidControl, Android in the Wild and GUIOdyssey cover Android apps. OS-Atlas and ScaleCUA-Data span desktop, mobile and web, OS-Atlas with grounding data only.

Snapshot as of

Key takeaways

  • Match the platform. Desktop, web and phone data use different actions. AgentNet records PyAutoGUI actions on Windows, macOS and Ubuntu, and AndroidControl and GUIOdyssey record clicks, long presses and scrolls in Android apps.
  • Decide whether you need grounding data or trajectories. OS-Atlas pairs screenshots with the element an instruction refers to, for training a model to find where to click. AgentNet and GUIOdyssey record whole tasks step by step, and FigmaTrace splits screen-recorded expert work into trajectories by design phase.
  • Check who recorded the steps. AgentNet, AndroidControl, FigmaTrace and Mind2Web hold people’s demonstrations. AgentTrek’s trajectories come from a model agent replaying web tutorials, checked by a model evaluator. ScaleCUA’s paper describes trajectories from expert operators and from rule-based exploration.
  • Read the license first. As of Nov 18, 2026, AgentNet is MIT and Mind2Web, GUIOdyssey and FigmaTrace are CC BY 4.0, while WebLINX is CC BY-NC-SA 4.0, a non-commercial license. AgentTrek, Android in the Wild and ScaleCUA-Data state no license on their cards or README.
  • Use evaluation traces for analysis. The OSWorld-Verified trajectories are runs of models on the benchmark, and their card lists training on them as not recommended.
  • For τ-bench, look at tool-use data. For tasks in the style of τ-bench, Salesforce released APIGen-MT-5k, 5,000 multi-turn trajectories in τ-bench’s retail and airline domains, under CC BY-NC 4.0.

Comparison table

All eleven datasets at a glance, in the order of this guide. Sizes and licenses are as each card, README or paper states them on Nov 18, 2026. Sort by any column marked with arrows.

DatasetBest forWhat it containsPlatformSizeLicense
FigmaTraceRead the paper →Design work in Figma, recorded from expertsExpert design trajectories with phase and skill labelsFigma3,469 trajectoriesCC-BY-4.0
AgentNetDesktop tasks on three operating systemsHuman-annotated desktop task trajectories with reasoningWindows, macOS, Ubuntu22.6K tasksMIT
AgentTrekWeb tasks, generated from tutorialsWeb trajectories generated from online tutorialsWeb52,594 dialogue turnsNot stated on its card
Android in the WildLarge-scale Android device controlHuman demonstrations of device controlAndroid715,142 episodesNot stated in its README
AndroidControlAndroid app tasks with goals and step instructionsHuman demonstrations with high-level goals and step instructionsAndroid15,283 demonstrationsApache 2.0
GUIOdysseyAndroid tasks across several appsCross-app navigation episodes with step annotationsAndroid8,334 episodesCC BY 4.0
Mind2WebWeb tasks on real websitesCrowdsourced action sequences on real websitesWeb2,000+ tasksCC BY 4.0
OS-AtlasFinding elements on screen, across platformsGUI grounding data, screenshots with element boxesWindows, Linux, macOS, Android, web13 million+ GUI elementsApache 2.0
OSWorld-Verified trajectoriesStudying model runs on OSWorldModel evaluation trajectoriesDesktop apps in OSWorld1000+ episodesMIT
ScaleCUA-DataGrounding and trajectories on six platformsGrounding data and operation trajectoriesLinux, macOS, Windows, Android, iOS, web15K+ weak-semantic and 4K goal-directed trajectories (paper)Not stated on its card
WebLINXWeb navigation over a conversationExpert demonstrations of conversational web navigationWeb2,300 demonstrationsCC BY-NC-SA 4.0

FigmaTrace

FigmaTrace is an open dataset of expert human Figma design workflows, built by Patronus AI. Its paper, FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows, was posted on Aug 20, 2026.

How does FigmaTrace work?

  • 126 long-horizon tasks in eight designer workflow categories, such as pixel-perfect replication, responsive adaptation, theming with variables and prototype wiring. The tasks cover a 10-skill expert taxonomy.
  • 200+ hours of screen-recorded expert work, converted into 3,469 trajectories, 2,883 for training and 586 for evaluation. Each trajectory is a sequence of action and frame pairs in the Playwright-MCP action space, labeled with design phases and skills.
  • The experts were hired through Upwork, each with 2+ years of Figma experience, and vetted with a starter task.
  • On Hugging Face as PatronusAI/figmatrace under the license its card states, CC-BY-4.0. The fine-tuned model from the paper is open too.

Limits its card states

The trajectories carry no reasoning chains, so the card says they need further annotation before training reasoning-trace models. Some noisy actions remain after preprocessing.

Who built it

Patronus AI was founded in 2023 by CEO Anand Kannappan, formerly of Meta Reality Labs, and CTO Rebecca Qian, formerly of FAIR (Meta AI). Its researchers and engineers come from Meta Superintelligence Labs, Amazon AGI, Apple and Google.

Patronus AI funding

A $50M Series B led by Greenfield Partners, announced Jun 25, 2026, with participation from Lightspeed Venture Partners, Notable Capital, Datadog, Samsung and others, after a $17M Series A led by Notable Capital in May 2024.

Patronus AI accepted venues

Main conference papers at ICML 2026, NeurIPS 2026 and ACL 2025, and a paper in Findings of ACL 2026.

Best for: Design work in Figma, from expert human sessions.

Read the paper →

AgentNet

AgentNet is the desktop task dataset of OpenCUA, from XLANG Lab at The University of Hong Kong with Moonshot AI, Stanford University, the University of Waterloo and Carnegie Mellon University (OpenCUA: Open Foundations for Computer-Use Agents).

How does AgentNet work?

  • 22.6K human-annotated computer use tasks across Windows, macOS and Ubuntu, as of Nov 18, 2026. The paper counts 200+ applications and websites.
  • Each step pairs a screenshot with a PyAutoGUI action and generated reasoning in three parts, observation, thought and action, plus a reflection on the step.
  • Tasks span four domains, work, professional, daily and system.
  • On Hugging Face as xlangai/AgentNet. Its card says it may be used for research, educational and commercial purposes under the MIT License.

Best for: Desktop tasks on Windows, macOS and Ubuntu, recorded by people.

AgentTrek

AgentTrek is a pipeline and dataset for web agents from The University of Hong Kong and Salesforce Research (AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials).

How does AgentTrek work?

  • The authors harvest tutorial-like texts from the internet, turn them into step-by-step tasks, have a vision-language model agent carry them out in real environments, and have a vision-language model evaluator check each trajectory.
  • The Hugging Face set is the training data for AgentTrek-1.0-32B, 52,594 dialogue turns on browsing, online shopping, product comparison and information retrieval.
  • On Hugging Face as xlangai/AgentTrek. Its card states no license as of Nov 18, 2026.

Best for: Web browsing and shopping tasks, with model-generated trajectories.

Android in the Wild

Android in the Wild is a dataset for Android device control from Google Research and Google DeepMind (Android in the Wild: A Large-Scale Dataset for Android Device Control).

How does Android in the Wild work?

  • 715,142 episodes and 30,378 unique instructions in five subsets, GoogleApps, Install, WebShopping, General and Single.
  • Four Android versions, 10 to 13, and eight device types with different screen resolutions.
  • Download from Google Cloud Storage (gresearch/android-in-the-wild), as described in its README, which states no license as of Nov 18, 2026.

Best for: Android device control across Android versions and devices.

AndroidControl

AndroidControl is a dataset of Android app demonstrations from Google DeepMind and Google (On the Effects of Data Scale on UI Control Agents).

How does AndroidControl work?

  • The paper counts 15,283 demonstrations of everyday tasks, 14,548 of them unique, across 833 Android apps. Its README describes demonstrations from human raters in 40 app categories.
  • Each episode has screenshots, accessibility trees, a high-level goal, step-by-step instructions and the actions as JSON.
  • Download from Google Cloud Storage (gresearch/android_control), as described in its README, whose license section reads Apache License, Version 2.0.

Best for: Android app tasks with both a goal and step instructions.

GUIOdyssey

GUIOdyssey is a dataset of cross-app tasks on mobile devices from Shanghai AI Laboratory, The University of Hong Kong, Nanjing University, Shanghai Jiao Tong University and Harbin Institute of Technology, Shenzhen (GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices).

How does GUIOdyssey work?

  • 8,334 episodes averaging 15.3 steps, from six mobile devices, across six types of cross-app tasks, 212 apps and 1,357 app combinations.
  • Each step has a screenshot and the action, with a description of the screen, the intention behind the next action, a summary of earlier steps, a low-level instruction and a box around the element clicked.
  • On Hugging Face as hflqf88888/GUIOdyssey, the version its authors recommend, under a Creative Commons Attribution 4.0 International License. The earlier OpenGVLab/GUI-Odyssey has 7,735 episodes.

Best for: Android tasks that move between several apps.

Mind2Web

Mind2Web is a dataset of web tasks from The Ohio State University (Mind2Web: Towards a Generalist Agent for the Web).

How does Mind2Web work?

  • Over 2,000 open-ended tasks from 137 websites in 31 domains, with crowdsourced action sequences.
  • Each step stores the page’s raw and cleaned HTML and the operation, click, type or select, on the target element. The Hugging Face training split has 1,009 tasks, and the test sets come through the GitHub repository to prevent leakage.
  • On Hugging Face as osunlp/Mind2Web under a Creative Commons Attribution 4.0 International License. osunlp/Multimodal-Mind2Web adds a screenshot of the page for each step, under an openrail license.

Best for: Web tasks on real websites across many domains.

OS-Atlas

OS-Atlas is a GUI grounding corpus and model from Shanghai AI Laboratory, Shanghai Jiao Tong University, The University of Hong Kong and MIT (OS-ATLAS: A Foundation Action Model for Generalist GUI Agents).

How does OS-Atlas work?

  • Grounding data only. Each sample is a screenshot, an instruction or referring expression, and the box around the element it refers to.
  • Mobile, desktop and web domains. The desktop data comes from Windows, Linux and macOS, and the web data from SeeClick and pages crawled from FineWeb URLs. The paper counts over 13 million GUI elements.
  • On Hugging Face as OS-Copilot/OS-Atlas-data, whose card lists the license apache-2.0.

Best for: Locating elements on screen across desktop, web and mobile.

OSWorld-Verified trajectories

The OSWorld team publishes the trajectories of models it evaluated on OSWorld-Verified. The benchmark comes from The University of Hong Kong with Carnegie Mellon University, Salesforce Research and the University of Waterloo (OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments).

How does the OSWorld-Verified trajectory set work?

  • Screenshots, action sequences, model reasoning, task results and metrics, at 15, 50 and 100 step limits.
  • Tasks in office apps (LibreOffice Calc, Writer and Impress), daily apps (Chrome, VLC and Thunderbird), professional tools (GIMP and VS Code), multi-app workflows and the operating system.
  • Its card, last updated August 2025, counts 15+ model variants and 1000+ evaluation episodes. It lists training computer use agents as "not recommended" and suggests performance analysis, debugging and benchmark comparison.
  • On Hugging Face as xlangai/ubuntu_osworld_verified_trajs, whose card lists the license mit.

Best for: Analyzing how models behave on OSWorld tasks.

ScaleCUA-Data

ScaleCUA-Data is a cross-platform computer use dataset from Shanghai AI Laboratory (ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data).

How does ScaleCUA-Data work?

  • Screenshots and actions on Linux, macOS, Windows, Android, iOS and the web.
  • Grounding data comes from automated pipelines, annotated by models such as Claude 3.7 and checked by human experts. The card says trajectory data was collected by people, with annotations generated by models.
  • The paper counts 471K GUI understanding examples, 17.1M grounding samples, and over 15K weak-semantic and 4K goal-directed trajectories. It says the weak-semantic trajectories come from rule-based agents exploring the interface and the goal-directed ones from expert operators.
  • On Hugging Face as OpenGVLab/ScaleCUA-Data. Its card states no license as of Nov 18, 2026.

Best for: Agents that work across desktop, mobile and web, with grounding and trajectory data.

WebLINX

WebLINX is a benchmark for conversational web navigation from Mila Quebec AI Institute, McGill University and Charles University (WebLINX: Real-World Website Navigation with Multi-Turn Dialogue).

How does WebLINX work?

  • 100K interactions across 2,300 expert demonstrations on over 150 real-world websites.
  • The agent follows a user’s instructions over a multi-turn dialogue, so the demonstrations hold the conversation as well as the browser actions.
  • On Hugging Face as McGill-NLP/WebLINX under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0). Its card says WebLINX 1.1 also runs as a benchmark in BrowserGym.

Best for: Web agents that take instructions from a user over several turns.

Where Patronus fits

Research

Patronus AI is a behavioral research lab for AI, and its 20 papers as of Nov 10, 2026 include work at NeurIPS, ICML and ACL. Its ACL 2025 paper Browsing Lost Unformed Recollections: A Benchmark for Tip-of-the-Tongue Search and Reasoning introduced BLUR, 573 questions that need search, reasoning and tool use.

Open data

FigmaTrace is on Hugging Face under CC-BY-4.0, with the fine-tuned model from its paper.

Environments for computer use agents

Patronus AI builds RL environments for computer use agents in the browser and on the desktop. Each attempt runs in its own sandbox with the real application, the agent’s tools and a grader. Patronus AI also sells its simulated apps and mock websites on their own.

Security

Patronus AI lists SOC 2, HIPAA and TISAX, and ownership of the training data from a custom build is set in each agreement.

Patronus AI builds environments for training agents to work in software built for people. Talk to Patronus AI about computer use environments.

Frequently asked questions

How do I get training data to improve my model on OSWorld or tau-bench style tasks?

For desktop tasks like OSWorld’s, open trajectory sets such as AgentNet and ScaleCUA-Data are on Hugging Face. The OSWorld-Verified model trajectories are there too, but their card does not recommend them for training. τ-bench tests agents that use tools in conversations with a simulated user, and Salesforce’s APIGen-MT-5k has 5,000 multi-turn trajectories in its retail and airline domains, under CC BY-NC 4.0. Beyond open data, teams train agents in RL environments, where a grader scores each attempt.

What are the best open datasets for GUI and computer use agents?

It depends on the platform. For desktop software, AgentNet and ScaleCUA-Data, and for design work in Figma, FigmaTrace. For websites, AgentTrek, Mind2Web and WebLINX. For Android, AndroidControl, Android in the Wild and GUIOdyssey. OS-Atlas adds grounding data across desktop, web and mobile.

What is FigmaTrace?

FigmaTrace is Patronus AI’s open dataset of expert Figma design workflows, 3,469 trajectories from 200+ hours of screen-recorded work on 126 long-horizon tasks, on Hugging Face under CC-BY-4.0. The paper is FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows.

Which companies provide app simulations or web app clones for training computer use agents?

The datasets on this page are fixed recordings, while simulated apps and web app clones come from RL environment companies. Patronus AI builds them into its RL environments for computer use agents and, as its domains page says, sells parts such as these on their own. Our guide to computer use RL environment companies covers the others.

What are the best vendors for computer use agent training data?

This page covers open datasets anyone can download. Companies that build environments for computer use agents, Patronus AI among them, are compared in our guide to computer use RL environment companies.

What is the difference between GUI grounding data and trajectory data?

Grounding data pairs a screenshot and an instruction with the location of one element, as in OS-Atlas, so a model learns where to click. Trajectory data records a whole task step by step, as in AgentNet or GUIOdyssey, so a model learns the order of actions.

Can I use these datasets commercially?

Check each card. As of Nov 18, 2026, AgentNet’s card allows commercial use under the MIT License. Mind2Web, GUIOdyssey and FigmaTrace are CC BY 4.0, though Mind2Web’s card says it was released solely for research purposes and GUIOdyssey’s says it is intended primarily for research. OS-Atlas and AndroidControl are Apache 2.0, and WebLINX is CC BY-NC-SA 4.0, a non-commercial license. AgentTrek, Android in the Wild and ScaleCUA-Data state no license on their cards or README.

How was this list made?

Patronus AI wrote it and created FigmaTrace, one of the entries. Entries are not ranked, and each entry links its paper and dataset.

Snapshot as of

Training computer use agents?
Read how FigmaTrace was built