Skip to content
Back to the index

RobotsTxtDetection

github.comInfrastructure

PulseGate's liveness check found it on 16 Sep 2026; it is registered on Firefox Add-ons and has been in the index since 16 Sep 2026. How this is checked

RobotsTxtDetection is an open-source utility that automatically checks whether websites expose a robots.txt file. It is intended for developers and automation workflows that need to inspect crawler directives across websites.

Inferred · not functionally tested

Open SourceWebEmbeddableCLISelf-hosted
RobotsTxtDetection preview
Visit github.com

Overview

3 features

Purpose: Checking websites for robots.txt files without manually visiting each site.

Inferred · not functionally tested

Audience: developers and web crawlers

Inferred · not functionally tested

Functions: Unknown

Interfaces: API: unknown · MCP: unknown · CLI: indicated (inferred, not tested) · Self-hosting: indicated (inferred, not tested)

Recorded constraints: pricing: open_source · license: Open Source · platforms: WEB · deployment: browser, embeddable, cli, self_hosted

Constraint provenance is unknown; confirm requirements with the publisher.

Record sources: github.com · addons.mozilla.org. These links do not verify the individual claims.

RobotsTxtDetection sits in PulseGate's CLI tools & terminal category. Inferred · not functionally tested: It focuses on checking websites for robots.txt files without manually visiting each site. Inferred · not functionally tested: RobotsTxtDetection is an open-source project aimed at developers and web crawlers. Basis unknown · not verified: The project is open source (Open Source). Basis unknown · not verified: It runs on the web, embeddable surfaces, and the command line, and it can be self-hosted.

It is developed by Werneror, and it first shipped in 2019. Inferred · not functionally tested: Key capabilities include robots.txt detection, website checking, and automated scanning.

Summary written by a language model from the project’s public pages.

Tasks: Inferred · not functionally tested

  • Robots.txt detection
  • Website checking
  • Automated scanning

Topics: Inferred · not functionally tested

Tags
robots-txtwebsite-crawlingweb-scrapingseo-tools

JSON profile · Text profile · Access guide

Built with & integrations

Runs on
BrowserEmbeddableCLISelf-hosted

Trust & compliance

License
Open Source
Public signals
HTTPSPrivacy PolicyTerms of ServiceOpen SourceFree tierStore listing found
Legal
Privacy Policy Terms of Service

Indexing history

1

What PulseGate has recorded for this listing

  1. Indexed16 Sep · 17:00 UTC
    robots.txt Detection seen via Firefox Add-ons
    Source: Firefox Add-ons · Open

Frequently asked questions about RobotsTxtDetection

What does RobotsTxtDetection do?
Inferred · not functionally tested: RobotsTxtDetection focuses on checking websites for robots.txt files without manually visiting each site. It is catalogued under CLI tools & terminal on PulseGate.
Who is RobotsTxtDetection for?
Inferred · not functionally tested: RobotsTxtDetection is an open-source project built for developers and web crawlers.
Does RobotsTxtDetection have a free plan?
Basis unknown · not verified: Yes — RobotsTxtDetection is open source under the Open Source license and free to use.
What platforms does RobotsTxtDetection run on?
Basis unknown · not verified: RobotsTxtDetection runs on the web, embeddable surfaces, and the command line. It can also be self-hosted.
Is RobotsTxtDetection still active?
PulseGate's liveness check found it on 16 Sep 2026.
Who develops RobotsTxtDetection?
RobotsTxtDetection is developed by Werneror.
How long has RobotsTxtDetection been around?
RobotsTxtDetection first shipped in 2019.
Is RobotsTxtDetection open source?
Basis unknown · not verified: Yes — RobotsTxtDetection is open source under the Open Source license.

Similar projects

Closest matches by what these projects do