Compare frameworks for turning web applications into desktop software.
Project overview
This repository compares Electron, Tauri, NW.js, Wails, NodeGui, Neutralino, Flutter, and .NET MAUI for developers choosing a web-to-desktop stack. Its tables cover platform support, front-end compatibility, updater options, project characteristics, package size, memory use, and other practical differences. Benchmark applications run in GitHub CI to provide like-for-like measurements across frameworks and operating systems. The maintainers note that system load and limited runs introduce error, so the numbers are directional rather than absolute rankings.
Create skill evaluations, run benchmarks, and compare performance across models.
Project overview
Waza is a Go CLI for scaffolding Agent Skills evaluation suites, running benchmarks, and comparing outcomes across models or executors. Its readiness checks cover frontmatter compliance, token budgets, evaluation files, and agentskills.io specification rules, and the workflow can be integrated into CI. It is aimed at skill authors and teams that need repeatable quality measurements and regression checks.
Run repeatable evaluations of whether agents discover and use a skill correctly.
Project overview
Skillgrade scaffolds an eval.yaml for a SKILL.md directory, runs repeated tasks, and measures whether an agent discovers and applies the skill as intended. It supports Gemini, Claude, Codex, ACP, OpenCode, and custom command agents, with deterministic or LLM-rubric graders and CLI or browser reports. Node.js 20+ is required, while Docker, API credentials, or a local command depend on the selected executor.
Develop and evaluate web agents through a unified, extensible Gym environment.
Project overview
BrowserGym exposes open-ended browser tasks and benchmarks such as MiniWoB, WebArena, WorkArena, and VisualWebArena through a consistent Gym interface. Researchers can install only the core environment or add benchmark-specific packages, then run agent–environment loops with Chromium controlled through Playwright. It is designed as research infrastructure rather than a consumer automation product, and individual benchmark suites require their own setup.
Compile, benchmark, and optimize agent skills across different models and runtimes.
Project overview
SkVM profiles a model-and-agent-runtime combination, compiles a skill against those capabilities, and then evaluates and improves it through precompilation and runtime-optimization workflows. It is aimed at teams that need the same skill to behave across heterogeneous agents rather than tuning prompts by hand for each runtime. Provider configuration is required, and the bundled benchmark suite lives in a separate data submodule.
Evaluate code patches proposed for real GitHub software issues.
Project overview
SWE-bench evaluates language models on real software issues collected from GitHub by giving each model a codebase and an issue, then testing the generated patch in a reproducible Docker environment. It is useful for comparing coding agents on actual repair work, but local evaluation is resource intensive: the project recommends an x86_64 machine with at least 120 GB of free storage and 16 GB of RAM.
Benchmarks autonomous agents on long-horizon real-world tasks with up to million-token contexts.
Project overview
AgencyBench V2 covers six capabilities across game, frontend, backend, code, research, and MCP work, with 32 real-world long-horizon scenarios and 138 tasks. A user-simulation agent supplies iterative feedback while a Docker sandbox runs functional and visual checks using rule-based, vision-based, and LLM judges. Scenarios require model and evaluator configuration, and game or frontend cases also need the remote Docker sandbox.
Evaluates LLM agents on real-world interactive tasks, tool use, and long-horizon application workflows.
Project overview
VitaBench evaluates agents in real-world service scenarios such as delivery, in-store operations, and OTA, using databases, API tools, and 100 tasks per domain. Configure the models, run the vita command, save simulations under data/simulations, and re-evaluate existing runs when needed. The current release updates datasets, tools, evaluator models, and metrics, so scores should be interpreted alongside the chosen configuration.
Train and evaluate systems on abstract grid-transformation reasoning tasks.
Project overview
ARC-AGI-2 contains 1,000 public training tasks and 120 public evaluation tasks built from small colored-integer grids. Solvers infer a transformation from demonstration pairs and must produce every test output with the correct dimensions and cells. The evaluation set is intended for previously unseen testing, so repeatedly tuning against it would compromise the result.
Evaluate LLM agents across diverse interactive environments and tasks.
Project overview
AgentBench measures how well LLMs act autonomously across eight interactive environments, including database and operating-system tasks. The current repository carries a function-calling version integrated with AgentRL, while the older benchmark remains available as v0.1. Running the suite requires Docker, and Python 3.9 is recommended because several scientific dependencies are pinned to older versions.
Studies active tool discovery for autonomous agents with an MCP retrieval dataset and scripts.
Project overview
MCP-Zero is the open-source code accompanying a paper on active tool discovery and retrieval for autonomous LLM agents. It includes instructions for an MCP-tools dataset covering 308 servers and 2,797 tools, plus code under MCP-tools/build_data for constructing a custom server dataset. The data file must be obtained separately and placed at the path expected by the evaluation code.
Evaluates stateful agents with datasets, graders, rewards, multi-turn cases, and repeatable suites.
Project overview
Letta Evals organizes agent evaluation from dataset and target through extractors, graders, rewards, and stored results. It supports JSONL or CSV data, multi-turn samples, multiple model handles, cached re-grading, deterministic or model-judge graders, and custom agent factories. Runs target Letta Code through a self-hosted or cloud server, so Python 3.11+, a running Letta service, and provider credentials are required.
Run deterministic, cross-model evaluations for agent skills in Docker.
Project overview
skill-optimizer provides both an agent skill for authoring eval suites and a local CLI for executing them in Docker against OpenRouter models. Cases and suites can be repeated across trials to benchmark skill behavior, debug failures, and compare reliability across models. The workbench can also expose hidden services such as MCP servers to the agent during a controlled test.
Run and score multiple model providers on ARC-AGI tasks.
Project overview
This benchmarking toolkit runs ARC-AGI tasks through configurable adapters for providers such as OpenAI, Anthropic, and Gemini, with rate limiting, retries, saved submissions, and scoring built in. It supports both single-task debugging and asynchronous batch runs. Full evaluations require separately cloned ARC-AGI data, and hosted models require their respective API credentials.
Evaluates agent trace debugging across reasoning, execution, and planning errors.
Project overview
TRAIL is a benchmark dataset with 148 annotated agent execution traces and 841 errors across reasoning, execution, and planning. The traces come from software-engineering and information-retrieval tasks, and the evaluation scripts run a LiteLLM-compatible model on GAIA or SWE Bench splits before calculating scores. It is intended for studying error localization in complex agent workflows.
Evaluate whether agents can set up and execute tasks from real research repositories.
Project overview
SUPER evaluates whether LLM agents can configure environments, execute code, and answer questions drawn from real machine-learning and NLP research repositories. Its dataset is divided into Expert, Masked, and AutoGen task sets, with code for running agents and scoring results. Local mode executes repository code directly on the machine, while Docker and Modal backends offer safer isolation and concurrent benchmark runs.
Score organizational AI maturity across ML, generative AI, and agentic readiness.
Project overview
The AAQ skill scores an organization from 0 to 100 across machine learning and analytics, generative AI adoption, and agentic AI readiness, with adjustable weights and five maturity tiers. It is designed for structured benchmarking from public information, so results help frame diligence but cannot reveal capabilities that a company does not disclose.