HorizonCAT

Model

Each task t has a human time; x_t = log₂ minutes. An agent with horizon h (log₂ minutes) and slope β succeeds on one run with probability

u_t is the task's difficulty beyond its length. Its posterior comes from every other agent's runs (marginal maximum likelihood, leave-one-agent-out) and is stored as 10 equal-mass points, so each prediction averages over what is still unknown about the task. v_at is this agent's own quirk on this task. The first back-test showed that leaving it out makes intervals too narrow, so σ_v is estimated by held-out predictive likelihood across agents (see Back-test).

Choosing the next task

The tool runs the task with the highest expected reduction in the variance of h per token. γ = 0 ignores cost, as in classical adaptive testing. It stops when the SD of h reaches the target, or declares the suite saturated when no remaining task can reduce the variance by 1% and more than 20% of the posterior lies beyond the longest task.

What this builds on

• METR, Measuring AI Ability to Complete Long Tasks and Time Horizon 1.1: the tasks, human-time anchors, horizon definition and public run data.
• BRIDGE (Liu, Gala, Nilaksh, Bahdanau, Reddy, Larochelle, 2026): IRT difficulty is linear in log human time, so human time is a calibrated difficulty scale. HorizonCAT uses that as its anchor. Unlike BRIDGE, it keeps human time fixed and models only the residual as a random effect, so horizons come out in minutes directly and the remaining task uncertainty is carried into every decision. That is one form of the "uncertainty-aware difficulty estimation" BRIDGE lists as future work.
• Adaptive testing for language models (ATLAS, Fluid Benchmarking), static subsets (tinyBenchmarks), and the 30–70% pass-rate filter of Efficient Benchmarking of AI Agents, included here as a baseline.

What is new here

• The target is a horizon in human minutes, not an ability score. Slope uncertainty is integrated out.
• Selection weighs information against real token cost, which varies about 2500× across METR tasks.
• Task difficulty enters as a posterior, not a point estimate.
• A replay back-test on real agent runs reports calibration (interval coverage), not only error.
• Saturation is detected explicitly, with a calculator for how to extend the suite.

Not claimed

That adaptive testing saves cost in general, which is well known, or that difficulty tracks log human time (BRIDGE, METR). There are no results for models without public per-task runs.

Limits

• One run per task. Repeated runs of a task are correlated through u_t and are not modelled.
• Token cost is a median across past agents. A new agent may use far more or fewer tokens.
• The token cost of hypothetical long tasks is extrapolated from tasks of 16 minutes and longer.
• The agent-by-task term v is one shared Gaussian scale. Real quirks may cluster by task family or by scaffold.