Design LLM evaluations with tool use, multi-turn dialogue, and model grading.
Project overview
Inspect is an evaluation framework created by the UK AI Security Institute for testing large language models. It includes components for prompt construction, tool use, multi-turn dialogue, and model-graded evaluation, plus more than 200 pre-built evaluations. Researchers can compose these capabilities into tasks and extend the framework through Python packages that add new elicitation or scoring techniques.
Build Python agent workflows with tools, handoffs, guardrails, sessions, and tracing.
Project overview
OpenAI Agents SDK lets Python developers define agents with instructions, tools, guardrails, and delegation between specialized workers. It includes session history, human approval points, tracing, sandboxed long-running agents, and realtime voice support. The framework can use OpenAI APIs or other supported model providers and requires Python 3.10 or newer.
Connect private data to retrieval systems and agentic applications.
Project overview
LlamaIndex OSS supplies connectors, indexes, retrieval, and query components for grounding LLM and agent applications in documents, APIs, databases, and other private data. Teams can start with a bundled package or assemble the core library with selected model, embedding, and vector-store integrations. LlamaParse is a separate commercial platform for document parsing, OCR, extraction, and hosted document agents.
Optimizes AI agents with almost no code changes across frameworks and training methods.
Project overview
Agent Lightning connects existing agents to optimization workflows with minimal code changes by collecting prompts, tool calls, and rewards as structured traces. It can selectively optimize one or more agents built with LangChain, the OpenAI Agent SDK, AutoGen, CrewAI, Microsoft Agent Framework, or plain Python. Supported approaches include reinforcement learning, automatic prompt optimization, and supervised fine-tuning.
Build contextual text and voice assistants across messaging channels.
Project overview
Rasa Open Source combines natural-language understanding with dialogue management for assistants that handle multi-turn conversations and connect to channels such as Slack, Telegram, and Twilio. The classic framework is now in maintenance mode, while newer Rasa agent development is moving toward Hello Rasa and the CALM engine.
Route each conversation to the most suitable specialized AI agent.
Project overview
Agent Squad selects the most suitable specialized agent from each request and preserves conversation context across agents and sessions. It provides Python, TypeScript, and Swift runtimes with replaceable classifiers, storage, retrievers, and tools. The Swift implementation can keep routing, MCP tools, voice, and chat history on Apple devices, while GroundedAgent separates data gathering from response writing so answers stay limited to actual tool results.
Let Claude create, load, and chain new tools during a conversation.
Project overview
Claude Engineer provides CLI and web interfaces where Claude can identify a missing capability, generate the corresponding tool code, load it dynamically, and keep using it during development work. It also manages persistent conversations, context and token usage, tool chaining, and debugging output, with optional Python execution through an E2B sandbox. Its core workflow depends on Anthropic Claude 3.5, and some tools require additional service credentials.
Benchmarks autonomous agents on long-horizon real-world tasks with up to million-token contexts.
Project overview
AgencyBench V2 covers six capabilities across game, frontend, backend, code, research, and MCP work, with 32 real-world long-horizon scenarios and 138 tasks. A user-simulation agent supplies iterative feedback while a Docker sandbox runs functional and visual checks using rule-based, vision-based, and LLM judges. Scenarios require model and evaluator configuration, and game or frontend cases also need the remote Docker sandbox.
Trains multi-turn, multi-task agents with asynchronous reinforcement learning on Ray.
Project overview
AgentRL separates agent reinforcement-learning training from environment deployment. Rollout, actor, and reference workers run on Ray to generate trajectories, update the policy, and maintain a frozen KL baseline, while a controller and task workers manage multi-turn environments over HTTP or gRPC. It is suited to reproducing the paper’s experiments or adding tasks, but training requires Ray, suitable CPU/GPU resources, and model configuration.
Evaluates LLM agents on real-world interactive tasks, tool use, and long-horizon application workflows.
Project overview
VitaBench evaluates agents in real-world service scenarios such as delivery, in-store operations, and OTA, using databases, API tools, and 100 tasks per domain. Configure the models, run the vita command, save simulations under data/simulations, and re-evaluate existing runs when needed. The current release updates datasets, tools, evaluator models, and metrics, so scores should be interpreted alongside the chosen configuration.
Orchestrate multiple coding agents for unattended implementation, testing, and review.
Project overview
Kodo assigns a goal to coding backends such as Claude Code, Cursor, Codex, Gemini, or Kimi and coordinates repeated work cycles with independent verification. It can implement features, run improvement reviews, test software through realistic user workflows, and fix findings from earlier runs, with effort levels controlling how aggressively agents iterate. At least one supported coding-agent backend must be installed.
Build collaborative multi-agent systems in Python or TypeScript.
Project overview
BeeAI Framework supplies Python and TypeScript components for agents, multi-agent handoffs, model backends, tools, and RAG, helping developers assemble applications that reason, act, and collaborate. Requirements can constrain agent behavior consistently across different models, while MCP, ACP, and custom tools extend the system. Examples work with hosted providers and can also run locally through Ollama.
Choose an architecture before creating, evaluating, improving, and packaging reusable AI skills.
Project overview
Skill Conductor manages the lifecycle of an AI skill from design through distribution, requiring authors to choose an architecture pattern before writing `SKILL.md`. It supports creation, improvement, validation, third-party review, trigger-description optimization, and packaging, combining deterministic checks with benchmark tasks and evidence-backed binary quality questions. It is aimed at systematic skill authoring and review rather than a one-off prompt test.
Version an agent’s identity, rules, memory, tools, and skills in Git.
Project overview
Gitagent represents an AI agent as a Git repository, with model configuration, identity, rules, memory, tools, skills, hooks, and workflows stored as reviewable files. You can fork an agent, branch its personality, diff rule changes, and inspect memory history with normal Git operations. The runtime supports OpenAI, Anthropic, and OpenAI-compatible providers, and requires Node.js, Git, and the relevant API credentials.
Train models to retrieve and invoke tools through token generation.
Project overview
ToolGen represents tools as dedicated vocabulary tokens so a model can select tools, call them, and continue generating language within one process. The repository includes models, data for its three-stage training procedure, and evaluation code for research reproduction. Its local inference example also requires a ToolBench key and a StableToolBench deployment.
Styles plain HTML as grayscale hand-drawn wireframes with embedded fonts and 1,745 scribble icons.
Project overview
lofi.css adds a paper-scribble look to semantic HTML through one CSS file with embedded handwriting fonts, grayscale styling, annotations, and a 1,745-icon sprite. It uses no JavaScript, dependencies, build step, or external requests, making it suited to early flow discussions rather than polished UI. Its companion Claude skill can generate constrained mockups from descriptions, notes, websites, or whiteboard photos.
Add tool governance, cost tracking, and audits to AI agents
Project overview
AutoHarness is a lightweight governance layer for existing AI agents. It can wrap an LLM client, check tool calls for risk and permissions, sanitize outputs, record decisions, and manage context and token budgets. YAML rules select different governance levels, while multi-agent profiles, per-call cost attribution, and session persistence cover longer-running workflows.
Evaluates stateful agents with datasets, graders, rewards, multi-turn cases, and repeatable suites.
Project overview
Letta Evals organizes agent evaluation from dataset and target through extractors, graders, rewards, and stored results. It supports JSONL or CSV data, multi-turn samples, multiple model handles, cached re-grading, deterministic or model-judge graders, and custom agent factories. Runs target Letta Code through a self-hosted or cloud server, so Python 3.11+, a running Letta service, and provider credentials are required.