robots.txt Tester dojito tools

robots.txt Tester

Paste a robots.txt, enter your bot name and some URLs. See what is allowed by Google's rules and by Python's robotparser — they don't always agree.

Examples

Result for Google (RFC 9309)

—

URLGoogle (RFC 9309)Python robotparserGroup used

    Crawl time calculator

    How long will a polite crawl take? Each request takes the server's response time plus your wait before the next one.

    Total time

    —

    Scraping with Python?

    The Polite Web Scraper Kit is a small Python toolkit that reads robots.txt before every request, waits between requests to the same site, caches pages for a day and retries gently. Standard library only, with five worked examples on public APIs and feeds, plus a CC0 dataset.

    Get the kit ($19)

    Check it in Python

    The same check with Python's built-in parser. Remember it uses first-match order, as shown in the Python column above.

    How robots.txt matching works

    1. Pick the group. A robots.txt is split into groups, each starting with one or more User-agent lines. Google takes the name part of your bot's User-Agent (letters, - and _) and looks for a group with exactly that name, ignoring case. All groups with that name are merged. If there is none, the User-agent: * groups apply. If there is no * group either, everything is allowed.

    2. Find the matching rules. Each Allow or Disallow path is matched against the start of the URL path and query. Matching is case-sensitive. * matches any characters and a $ at the end means "the URL ends here". An empty Disallow: blocks nothing.

    3. The longest rule wins. When several rules match, the one with the longest path wins. If an Allow and a Disallow are equally long, Allow wins. /robots.txt itself is always allowed.

    Where Python differs. urllib.robotparser picks the first group whose name appears inside your bot name, uses only the first User-agent: * group, and then returns the first rule in file order that matches as a plain prefix — no wildcards. Rows where the two answers differ are highlighted in the table. If your scraper uses Python's parser, put specific Allow lines above broader Disallow lines in your own files, and treat a "Blocked" from either column as a reason to stay away.

    This tool follows RFC 9309 and Google's published examples, and the Python column is tested against Python 3.12's real parser on 4,025 cases.

    Questions

    Does this tool download robots.txt from my site?

    No. Paste the file yourself. Everything runs in your browser; nothing you type is sent to us or to the site. To see a site's file, open https://the-site.com/robots.txt in a new tab and copy it.

    Why do Google and Python give different answers?

    Google follows RFC 9309: the longest matching rule wins, Allow wins a tie, and * and $ work as wildcards. Python's urllib.robotparser uses the first rule that matches, in file order, and treats * and $ inside a path as plain characters. So a file with Disallow: /shop/ above Allow: /shop/sale/ blocks /shop/sale/ in Python but allows it for Google.

    How do I pick the bot name?

    Use the name your crawler sends in its User-Agent header, like MyScraper/1.0. Google matches only the name part (letters, hyphens and underscores) and needs an exact match. Python checks whether a group name appears anywhere inside your bot name, so a group for 'bot' also catches 'MyBot'.

    Is Crawl-delay a real rule?

    It is not in RFC 9309 and Google ignores it, but Bing, Yandex and many polite scrapers honor it. If a site sets it, wait at least that long between requests.

    Does robots.txt make scraping legal?

    No. robots.txt is a request from the site owner, not a contract or a law. Also check the site's terms of service, copyright and privacy rules. This is general information, not legal advice; ask a lawyer about your own case.

    How fast can I crawl?

    Without other guidance, one request per second to one site, using one connection, is a common polite default. The calculator shows how long a crawl takes at that pace, so you can plan instead of speeding up.

    More free tools