1
robots.txt
Communicates crawler-access directives and may reference sitemap locations.
Free crawl discovery check
Check robots and sitemap
Review public robots.txt and a declared same-host sitemap.
This bounded public check shows supported evidence only. Review the result before making a site change.
What does a robots.txt and sitemap checker do?
This checker reviews public crawl and discovery files. robots.txt can communicate crawler-access rules, while XML sitemaps help search engines discover intended URLs. Healthy files do not guarantee indexing.
How to read crawl discovery signals
Each supported result maps to observed evidence, a status, why it matters, and next-step guidance. Unavailable evidence remains not checked rather than being inferred.
2
XML sitemap
Lists intended URLs or sitemap indexes for discovery.
Robots.txt vs XML sitemap
Robots.txt communicates crawler-access directives and can reference sitemap locations. An XML sitemap lists intended URLs for discovery. Neither file guarantees rankings or indexing.
Check, fix, recheck
- 1. Check the current public state.
- 2. Understand the observed evidence and update the responsible CMS, host, CDN, DNS, or template setting.
- 3. Recheck the same public URL to verify that the observed signal changed.
Fix → recheck
1. Check
Run the current public URL.
2. Fix
Use the observed evidence to update the responsible setting.
3. Recheck
Confirm the public signal after caches or deployments settle.
Why continuous monitoring can matter
Crawl-discovery files can change during site releases.
Migrations, redesigns, plugin updates, CDN changes, and deployments can alter robots or sitemap behavior. For important client sites, monitoring reduces reliance on manual spot checks.
Save and monitor this website
For agencies
One-off diagnosis. Repeatable client-site oversight.
The free checker is useful after a change. Agencies can use supported Pulse monitoring workflows when crawl-discovery signals need repeatable attention across client sites.
What does Disallow: / mean?
A broad Disallow: / rule can communicate a site-wide crawl restriction to participating robots parsers.
When to use Search Console or a full crawler
Use Search Console for submitted sitemap and URL-inspection evidence; use a full crawler for exhaustive internal URL, canonical, noindex, and link analysis.
More detail
Check now. Monitor what matters over time.
Use the bounded evidence above to take the next appropriate action, then return to the same public URL after a change.
Recheck this website