Website Robots.txt Checker
Find mistakes in robots.txt before they cost you traffic. The checker fetches the file, handles missing files and server errors the way Google does, parses it to the robots exclusion standard and reports syntax problems, a Disallow: / that blocks everything, unsupported Noindex lines, crawl-delay, the content type, a byte-order mark and a missing Sitemap line. It shows the groups and their rules, tests common paths for Googlebot and checks that the home page’s own CSS and JavaScript are not blocked.
- Encrypted connection
- No sign-up
- Free to use
How to use Website Robots.txt Checker
- Enter a website.
- Click “Check robots.txt”.
- Read the problems found.
- Test your own URLs with the Robots.txt Tester.
Website Robots.txt Checker features
Standard parsing
RFC 9309 rules.
Blocked assets
CSS and JS of the home page.
Status handling
404 and 5xx explained.
Groups and rules
Per user agent.
Safe fetching
Public addresses only, with size and time limits.
Common paths
Tested for Googlebot.
When to use Website Robots.txt Checker
- Launch checks for new sites.
- Investigating lost traffic.
- Reviewing CMS defaults.
- Before changing crawl rules.
Website Robots.txt Checker FAQ
What happens if robots.txt returns 500?
Google stops crawling the site until the file is reachable again.
Is a missing robots.txt a problem?
No – crawlers then assume everything is allowed.
Why not block CSS and JavaScript?
Google renders pages like a browser; blocked files make pages look broken to it.
Does robots.txt remove pages from search?
No. It blocks crawling; use noindex to keep pages out of results.
What happens if robots.txt returns a server error?
Google treats a 5xx response as a temporary full block and may stop crawling the site until the file is reachable again.
Can robots.txt hide a page from Google?
It stops crawling, but a blocked page can still appear in results if other sites link to it. Use a noindex directive on a crawlable page to keep it out.
How quickly does Google see robots.txt changes?
Google usually caches robots.txt for up to a day. You can ask for a faster refresh in Search Console’s robots.txt report.
robots.txt without surprises
One wrong line in robots.txt can hide a whole site from search engines. Checking it after every change is cheap insurance.
Remember that robots.txt is public: do not use it to hide sensitive URLs.
robots.txt tells crawlers which paths they may fetch. A single mistake such as “Disallow: /” can remove a whole site from search. The checker fetches the file, handles missing files and server errors the way Google does, and parses it according to the robots exclusion standard (RFC 9309).
Common problems are reported: syntax errors, rules outside a user-agent group, Noindex lines (Google no longer supports them), crawl-delay (ignored by Google), the wrong content type and a byte-order mark at the start of the file. A sitemap line pointing to your XML sitemap is recommended.
Remember that robots.txt controls crawling, not indexing. To keep a page out of search results, allow crawling and use a noindex meta tag or header instead.
Keep robots.txt short. Block only areas that waste crawl budget, such as internal search results or endless filter combinations, and never block CSS or JavaScript files that search engines need to render pages. After every change, test important URLs to make sure they are still allowed, and keep a copy of the previous version so you can roll back quickly.