Rebecca Qian
Cofounder and CTO
Rebecca is cofounder and CTO of Patronus AI and a coauthor on its agent benchmarks and environments, including TRAIL, MEMTRACK and Generative Simulators. Before Patronus, she was at FAIR (Meta AI), where she published on fairer NLP and dialogue evaluation.
At Patronus· Newest first
Newest first
Papers
SpeedrunBench
Video game speedrunning as a test of whether agents can beat strategies humans already know.
DETOUR
A two-agent benchmark for tip-of-the-tongue search across several turns.
TRACE
54 kinds of reward exploit in code RL, and how well models catch them.
Generative Simulators
Environments that generate their own tasks, world dynamics and rewards.
MEMTRACK
Testing whether agents keep track of facts spread across workplace tools.
TRAIL
A benchmark for finding the step where an agent went wrong.
BLUR
573 tip-of-the-tongue questions that need search, reasoning and tool use.
GLIDER
A small, explainable judge model for grading LLM interactions.
Lynx
An open-source model for detecting hallucinations in retrieval-augmented generation.
FinanceBench
Open-book financial question answering about publicly traded companies.
SimpleSafetyTests
100 prompts for finding critical safety risks in language models.
Posts
Introducing SpeedrunBench: A Challenging Benchmark for Frontier Agents
Introducing Generative Simulators: Autonomously Scaling Environments for Agents
No papers or posts here yet.