The Long-Horizon-Terminal-Bench was introduced on July 9, 2026, by a team of researchers led by Zongxia Li. This benchmark assesses AI agents on 46 long-horizon tasks across nine categories, including software engineering and scientific computing. Unlike traditional benchmarks that focus on short tasks, this new framework emphasizes intermediate progress through dense reward-based grading.
Understanding Long-Horizon-Terminal-Bench
The Long-Horizon-Terminal-Bench addresses a critical gap in AI evaluations by focusing on tasks that span longer durations and require complex problem-solving skills. Traditional benchmarks often assess performance solely on final outcomes, neglecting the value of intermediate steps. This new approach allows for a more nuanced evaluation of agent capabilities.
The benchmark features a range of tasks that require hundreds of episodes and considerable execution time. Tasks are designed to stress long-horizon planning and iterative debugging, moving beyond one-shot problem-solving scenarios. Each task is broken down into fine-grained graded subtasks, providing dense intermediate rewards and partial credit.
Performance Insights of AI Agents
In their evaluation, the researchers tested 15 frontier AI models, revealing significant insights into their performance. On average, agents consumed 9.9 million tokens per task, with each run taking approximately 85.3 minutes across 231 episodes. The findings indicated that even the best-performing model achieved a mere 15.2% pass rate at a partial-reward threshold of 0.95.
