Abstract
As machine learning systems become increasingly agentic, static dataset mixes, hand-written environments and arenas, and fixed benchmarks are proving insufficient for post-training and evaluation beyond a certain extent, as continuous training or hill-climbing through repeated evaluation is bound to saturate them. This limitation persists even for interactive benchmarks with fixed configurations. Recent work, therefore, argues for dynamic benchmarks that co-evolve with model capabilities, but typically scales only one environmental component at a time (the tasks, tools, or reward design). In this position paper, we showcase the need for jointly scaling all three components as a more robust foundation for dynamic evaluation and training. Our position builds on curriculum learning and meta-evaluation theory through which we sketch a class of joint-scaling algorithms and present a GridWorld thought experiment showing that scaling any single component can hit fundamental limits. We recommend that future environment-centric benchmarking and post-training methods be designed with this joint-scaling paradigm in consideration.







