Skip to content
— ✱ FREE TOOL · ENGINEERING · NO SIGNUP

Who is training on you?

See which AI training crawlers your robots.txt actually blocks — and which it doesn't.

We fetch your robots.txt, ai.txt, /.well-known/ai-training, and X-Robots-Tag header, then classify each of 18 known AI crawlers as allowed, blocked, or not mentioned.

Try:
How this works
  • What we check: robots.txt, ai.txt (LLMTXT), /.well-known/ai-training, and the X-Robots-Tag HTTP header — cross-referenced against the 2026 registry of 18 AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, Applebot-Extended, Meta-ExternalAgent, Amazonbot, and more).
  • What you get: per-vendor breakdown of who is allowed vs. blocked, plus a drop-in robots.txt block that opts you out of every AI training crawler.
  • What we don't do: no authentication, no port scanning, no AI inference. Four parallel HTTP requests + DNS resolution. ~2–4 seconds.
  • Probe ID: all requests carry the user-agent VexocoreAiCrawlerAudit/1.0 with a link to our responsible-disclosure policy.
  • Privacy: anon scans live 90 days. Sign in and your scans persist forever in your dashboard. We never sell your data; see our Privacy Policy.