doc-scraper
doc-scraper is an open-source Go command-line web crawler for scraping documentation sites and converting their content into clean Markdown. It is intended for developers preparing documentation for retrieval-augmented generation systems and LLM training datasets.
Overview
6 featuresIn the RAG, search & retrieval space, doc-scraper takes a focused approach. It focuses on collecting and cleaning documentation for RAG systems and training datasets. It is built as an open-source project for developers building documentation crawlers, RAG pipelines, and LLM datasets. doc-scraper is open source under the Open Source license. doc-scraper is available on the command line, and it can be self-hosted.
It is developed by Sriram-PR. Among its 6 catalogued features are web crawling, documentation scraping, and markdown conversion.
Summary written by a language model from the project’s public pages.
- Web crawling
- Documentation scraping
- Markdown conversion
- Content cleaning
- RAG ingestion
- Training data export
Built with & integrations
Trust & compliance
Indexing history
1What PulseGate has recorded for this listing
- Indexed4 Sept · 12:28 UTCdoc-scraper verified via MCP Registry (official)Source: MCP Registry (official) · Open
Frequently asked questions about doc-scraper
- What does doc-scraper do?
- Doc-scraper focuses on collecting and cleaning documentation for RAG systems and training datasets. It is catalogued under RAG, search & retrieval on PulseGate.
- Who should use doc-scraper?
- doc-scraper is an open-source project built for developers building documentation crawlers, RAG pipelines, and LLM datasets.
- Is doc-scraper free?
- Yes — doc-scraper is open source under the Open Source license and free to use.
- What platforms does doc-scraper run on?
- doc-scraper runs on the command line. It can also be self-hosted.
- Is doc-scraper still active?
- Unverified. doc-scraper has not been re-checked since it entered the index, so there is no finding either way — and only a positive finding would say otherwise.
- What projects are similar to doc-scraper?
- Similar projects tracked by PulseGate include docscrape, DocHarvest, and WebGlean.docscrapeDocHarvestWebGlean
- Who develops doc-scraper?
- doc-scraper is developed by Sriram-PR.
- Is doc-scraper open source?
- Yes — doc-scraper is open source under the Open Source license.
Similar projects
Closest matches by what these projects do