AgentBench

Project overview

ARC-AGI-2 contains 1,000 public training tasks and 120 public evaluation tasks built from small colored-integer grids. Solvers infer a transformation from demonstration pairs and must produce every test output with the correct dimensions and cells. The evaluation set is intended for previously unseen testing, so repeatedly tuning against it would compromise the result.

Project overview

SUPER evaluates whether LLM agents can configure environments, execute code, and answer questions drawn from real machine-learning and NLP research repositories. Its dataset is divided into Expert, Masked, and AutoGen task sets, with code for running agents and scoring results. Local mode executes repository code directly on the machine, while Docker and Modal backends offer safer isolation and concurrent benchmark runs.