CLI / Rust
DOCRAWL
DOCS-FOCUSED CRAWLER TO CLEAN MARKDOWN
Crawls documentation websites and converts them to clean Markdown while preserving structure. Auto-detects frameworks like Docusaurus, MkDocs, Sphinx, and Next.js. Respects robots.txt and rate limits.
$
cargo install docrawl Features
Auto-detects Docusaurus, MkDocs, Sphinx, Next.js, and more
Converts HTML to Markdown preserving code blocks and tables
Mirrors original folder hierarchy
Respects robots.txt and rate limiting
Sitemap support for comprehensive crawling
Resume capability for interrupted crawls
Polite crawling with configurable delays