Fast, token-efficient web content extraction that converts websites to clean Markdown. Features Mozilla Readability, smart caching, polite crawling with robots.txt support, and concurrent fetching with minimal dependencies.
Fast, token-efficient web content extraction for AI agents - converts websites to clean Markdown.
Existing MCP web crawlers are slow and consume large quantities of tokens. This pauses the development process and provides incomplete results as LLMs need to parse whole web pages.
This MCP package fetches web pages locally, strips noise, and converts content to clean Markdown while preserving links. Designed for Claude Code, IDEs and LLM pipelines with minimal token footprint. Crawl sites locally with minimal dependencies.
Note: This package now uses @just-every/crawl for its core crawling and markdown conversion functionality.
claude mcp add read-website-fast -s user -- npx -y @just-every/mcp-read-website-fast
code --add-mcp '{"name":"read-website-fast","command":"npx","args":["-y","@just-every/mcp-read-website-fast"]}'
cursor://anysphere.cursor-deeplink/mcp/install?name=read-website-fast&config=eyJyZWFkLXdlYnNpdGUtZmFzdCI6eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBqdXN0LWV2ZXJ5L21jcC1yZWFkLXdlYnNpdGUtZmFzdCJdfX0=
Settings β Tools β AI Assistant β Model Context Protocol (MCP) β Add
Choose βAs JSONβ and paste:
{"command":"npx","args":["-y","@just-every/mcp-read-website-fast"]}
Or, in the chat window, type /add and fill in the same JSONβboth paths land the server in a single step. οΏΌ
{
"mcpServers": {
"read-website-fast": {
"command": "npx",
"args": ["-y", "@just-every/mcp-read-website-fast"]
}
}
}
Drop this into your clientβs mcp.json (e.g. .vscode/mcp.json, ~/.cursor/mcp.json, or .mcp.json for Claude).
read_website
- Fetches a webpage and converts it to clean markdown
url
(required): The HTTP/HTTPS URL to fetchpages
(optional): Maximum number of pages to crawl (default: 1, max: 100)read-website-fast://status
- Get cache statisticsread-website-fast://clear-cache
- Clear the cache directorynpm install
npm run build
npm run dev fetch https://example.com/article
npm run dev fetch https://example.com --depth 2 --concurrency 5
# Markdown only (default)
npm run dev fetch https://example.com
# JSON output with metadata
npm run dev fetch https://example.com --output json
# Both URL and markdown
npm run dev fetch https://example.com --output both
-p, --pages <number>
- Maximum number of pages to crawl (default: 1)-c, --concurrency <number>
- Max concurrent requests (default: 3)--no-robots
- Ignore robots.txt--all-origins
- Allow cross-origin crawling-u, --user-agent <string>
- Custom user agent--cache-dir <path>
- Cache directory (default: .cache)-t, --timeout <ms>
- Request timeout in milliseconds (default: 30000)-o, --output <format>
- Output format: json, markdown, or both (default: markdown)npm run dev clear-cache
The MCP server includes automatic restart capability by default for improved reliability:
For development/debugging without auto-restart:
# Run directly without restart wrapper
npm run serve:dev
mcp/
βββ src/
β βββ crawler/ # URL fetching, queue management, robots.txt
β βββ parser/ # DOM parsing, Readability, Turndown conversion
β βββ cache/ # Disk-based caching with SHA-256 keys
β βββ utils/ # Logger, chunker utilities
β βββ index.ts # CLI entry point
β βββ serve.ts # MCP server entry point
β βββ serve-restart.ts # Auto-restart wrapper
# Run in development mode
npm run dev fetch https://example.com
# Build for production
npm run build
# Run tests
npm test
# Type checking
npm run typecheck
# Linting
npm run lint
Contributions are welcome! Please:
npm run dev clear-cache
-t
flag-u
flagMIT