No items found.

MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environments

Darshan Deshpande, Varun Gangal, Hersh Mehta, Anand Kannappan, Rebecca Qian, Peng Wang

Darshan Deshpande*, Varun Gangal*, Hersh Mehta, Anand Kannappan, Rebecca Qian, Peng Wang

*Equal contribution

“Desert Ribbons”, Morocco
03

Abstract

Recent works on context and memory benchmarking have primarily focused on conversational instances but the need for evaluating memory in dynamic enterprise environments is crucial for its effective application. We introduce MEMTRACK, a benchmark designed to evaluate long-term memory and state tracking in multi-platform agent environments. MEMTRACK models realistic organizational workflows by integrating asynchronous events across multiple communication and productivity platforms such as Slack, Linear and Git. Each benchmark instance provides a chronologically platform-interleaved timeline, with noisy, conflicting, cross-referring information as well as potential codebase/file-system comprehension and exploration. Consequently, our benchmark tests memory capabilities such as acquistion, selection and conflict resolution. We curate the MEMTRACK dataset through both manual expert driven design and scalable agent based synthesis, generating ecologically valid scenarios grounded in real world software development processes. We introduce pertinent metrics for Correctness, Efficiency, and Redundancy that capture the effectiveness of memory mechanisms beyond simple QA performance. Experiments across SoTA LLMs and memory backends reveal challenges in utilizing memory across long horizons, handling cross-platform dependencies, and resolving contradictions. Notably, the best performing GPT-5 model only achieves a 60% Correctness score on MEMTRACK. This work provides an extensible framework for advancing evaluation research for memory-augmented agents, beyond existing focus on conversational setups, and sets the stage for multi-agent, multi-platform memory benchmarking in complex organizational settings

03

Runs

Cite this
TextBibTeXCopy
Darshan Deshpande, Varun Gangal, Hersh Mehta, Anand Kannappan, Rebecca Qian, Peng Wang. MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environments. arXiv:2510.01353, 2025.
@misc{deshpande2025memtrackevaluatinglongtermmemory, title={MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environments}, author={Darshan Deshpande and Varun Gangal and Hersh Mehta and Anand Kannappan and Rebecca Qian and Peng Wang}, year={2025}, eprint={2510.01353}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2510.01353}, }
NeurIPS 2025 workshop
October 1, 2025
NeurIPS 2025 Workshop
Workshop
Workshop on Scaling Environments for Agents (SEA)
Poster
memtrack
More research
All research →
“Konari, Iran”, Zagros Mountains“Konari, Iran”, Zagros Mountains
GEM Workshop at ACL 2026
meta

DETOUR: An Interactive Benchmark for Dual-Agent Search and Reasoning

January 30, 2026
ACL 2026 workshop
Workshop
detour
GEM Workshop at ACL 2026
“Alluvial Fan”, Taklimakan Desert, China“Alluvial Fan”, Taklimakan Desert, China
ICML 2026
meta

Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis

January 27, 2026
ICML 2026
Main
trace
ICML 2026
“Capillaries”, Meta River, Colombia–Venezuela border“Capillaries”, Meta River, Colombia–Venezuela border
meta

TRAIL: Trace Reasoning and Agentic Issue Localization

May 13, 2025
arXiv 2505.08638
trail
From the blog
All posts →