letta-evals

Skill Conductor manages the lifecycle of an AI skill from design through distribution, requiring authors to choose an architecture pattern before writing `SKILL.md`. It supports creation, improvement, validation, third-party review, trigger-description optimization, and packaging, combining deterministic checks with benchmark tasks and evidence-backed binary quality questions. It is aimed at systematic skill authoring and review rather than a one-off prompt test.

Playwright drives Chromium, Firefox, and WebKit through one API for end-to-end testing and general browser automation. It can be used as a test runner or library, through a CLI designed for coding agents, or through an MCP server for LLM-driven workflows. Its VS Code extension also supports test recording, debugging, locator picking, and trace inspection.

FIRERPA combines Android remote desktop, UI automation, OCR and image matching, traffic capture, Frida, proxy networking, and device operations behind one on-device service and Python client. It can work with or without root and expose controls through a browser, APIs, MCP clients, or a natural-language agent. The platform targets automation engineering, mobile security work, and long-running device farms, so its breadth and deployment complexity go well beyond a simple phone mirroring utility.

agent-skills-eval tests an Agent Skill against the same prompts in two modes: with the skill loaded and without it as a baseline. A judge model grades both outputs against defined assertions, producing JSON and JSONL artifacts plus a static HTML report that shows measurable lift, failures, timing, and tool-call behavior. The project includes a command-line runner and a TypeScript SDK, supports OpenAI-compatible chat backends and custom providers, and can run in CI. Users supply both the target model and the judge model.