Find the open-source solution that fits.

Project overview

AgencyBench V2 covers six capabilities across game, frontend, backend, code, research, and MCP work, with 32 real-world long-horizon scenarios and 138 tasks. A user-simulation agent supplies iterative feedback while a Docker sandbox runs functional and visual checks using rule-based, vision-based, and LLM judges. Scenarios require model and evaluator configuration, and game or frontend cases also need the remote Docker sandbox.

Project overview

VitaBench evaluates agents in real-world service scenarios such as delivery, in-store operations, and OTA, using databases, API tools, and 100 tasks per domain. Configure the models, run the vita command, save simulations under data/simulations, and re-evaluate existing runs when needed. The current release updates datasets, tools, evaluator models, and metrics, so scores should be interpreted alongside the chosen configuration.

Project overview

ARC-AGI-2 contains 1,000 public training tasks and 120 public evaluation tasks built from small colored-integer grids. Solvers infer a transformation from demonstration pairs and must produce every test output with the correct dimensions and cells. The evaluation set is intended for previously unseen testing, so repeatedly tuning against it would compromise the result.

Project overview

SWE-Gym provides 2,400 real software-engineering tasks drawn from 11 Python repositories, together with repository context, executable environments, and test-based verification. Researchers can use it to generate trajectories and train agents or solution verifiers. Reproducing the experiments also involves the published dataset, prebuilt Docker images, and instructions for OpenHands or MoatlessTools.

Project overview

TRAIL is a benchmark dataset with 148 annotated agent execution traces and 841 errors across reasoning, execution, and planning. The traces come from software-engineering and information-retrieval tasks, and the evaluation scripts run a LiteLLM-compatible model on GAIA or SWE Bench splits before calculating scores. It is intended for studying error localization in complex agent workflows.

Project overview

Wallpaper6 Design Skills packages 40 atomic recipes for turning a source image into a seamless tile. Each recipe combines primitives such as symmetry, density, rotation, twist, graphic effects, and lighting effects, with JSON, JSONL, CSV, and TypeScript formats for different consumers. The repository keeps a legacy 64-skill file for compatibility, but new integrations should use the v2 recipes.json recipes.