AI crawler log check: which AI bots actually visit your site?
Drop in a web server access log and see how often GPTBot, ClaudeBot, PerplexityBot and other AI crawlers visited, which pages they read and whether they were blocked. The log stays in your browser.
How do I know if AI crawlers visit my website?
Look in your web server's access log for requests whose user agent contains an AI crawler name such as GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot. This tool does that count for you: choose an access log file and it lists each AI crawler, how many pages it requested, the pages it read most and how many requests were refused.
Is my log file uploaded anywhere?
No. The file is read and counted by your own browser on this page. Nothing from the log is sent to SeoGauge or anyone else, and nothing is stored after you close the tab.
Which log formats work?
The Apache and nginx combined format (the default on most servers), and logs with one JSON object per line such as Cloudflare Logpush or Caddy. The log must include the user agent, so the short "common" format without it cannot be used. Files up to 40 MB are read; for larger logs, use the most recent part.
An AI crawler is allowed in robots.txt but never shows up. Why?
Being allowed only means the crawler may visit. New or rarely linked sites are crawled seldom, a firewall or bot-protection service may refuse the request before it reaches your server, and CDN-cached pages may never appear in the origin log. Check your CDN's own logs and bot settings, and make sure the pages are linked from your sitemap.
See how your site scores on SEO, AEO and GEO.
Measure your website — free