Project overview
webclaw provides a local-first way to scrape, crawl, map, and extract websites for agents and RAG systems. Its core CLI and MCP workflow can run without an account for many pages, while a hosted API is available for bot-protected sites, JavaScript rendering, asynchronous crawl jobs, search, and production monitoring.
Repository facts
- Primary language
- Rust
- License
- AGPL-3.0
- Repository updated
- Jul 13, 2026
- Default branch
- main
Resource types
CLI appAPI or serverMCP serverSDK
Use cases
Research and knowledge
Platforms
Claude CodeCursorCodex
Runtime
Command lineCloudLocal
Protocols & integrations
Model Context ProtocolREST
Capabilities
Web search
Audience
Developers