How to See Which Bots Are Crawling Your WordPress Site

Open Google Analytics right now and look at your traffic. What you’re seeing is a filtered version of reality. By design, Google Analytics strips out bot traffic and shows you only human visitors. The bots that visit your site — sometimes hundreds of times a day — are invisible in the data you look at every week.

This matters for two distinct reasons. First, you can’t make informed decisions about your content strategy, server performance, or security posture without knowing what’s actually hitting your site. Second, some of those invisible bots are doing things you’d actively want to stop — harvesting your contact information, scraping your content, scanning for vulnerabilities, or probing for weaknesses.

This is how you see them. All of them.

Why Your Analytics Show a Fraction of Your Real Traffic

Google Analytics works by loading a JavaScript snippet on every page. When a real browser loads your page, it executes the JavaScript and sends a tracking event. Bots don’t execute JavaScript — they request the HTML and move on. So the JavaScript never fires and the visit is never recorded.

The result: your analytics shows you human traffic. Your server logs show you everything. A 2026 analysis of AI crawler detection across 50+ sites found that over 40% of “traffic” on content-heavy sites wasn’t human — it was AI bots, scrapers, and automated systems that standard analytics reported as zero. The site owners had no idea.

The fastest-growing bot category in 2026 is AI crawlers. GPTBot alone grew 305% between May 2024 and May 2025, climbing from ninth to third among crawlers tracked by Cloudflare. ClaudeBot (Anthropic), PerplexityBot, Google-Extended, Bytespider (ByteDance/TikTok), and dozens of others are systematically visiting WordPress sites — indexing content for AI training pipelines and answer engines. A single bot can hit a site thousands of times in a day.

The Two Categories of Bots on Your Site

Before looking at how to see them, it’s worth understanding what you’re looking for. Bots fall into two meaningful categories for WordPress site owners — and the response to each is completely different.

The AI crawler category (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) is a judgment call that depends on your content strategy. Allowing them means your content may appear in AI-generated answers — which can drive awareness. Blocking them means your content stays exclusively on your site and in traditional search. Neither choice is wrong; the important thing is making it deliberately rather than by default.

The bad bot category — scrapers, contact harvesters, vulnerability scanners — deserves active attention. These are the bots covered in Good Bots vs Bad Bots: What’s the Difference? and the ones most directly connected to the spam calls, junk emails, and robocalls covered in Why Your WordPress Contact Page Is a Spam Magnet.

Method 1: Server Access Logs — The Most Complete Picture

Your server access log records every request made to your site — human or bot, JavaScript-capable or not. It’s the most complete bot visibility tool available, and it’s almost certainly accessible through your hosting control panel right now.

Where to find your logs:

Most WordPress hosting control panels expose access logs directly. In cPanel, look under Metrics → Raw Access or Logs → Access Log. In Plesk, find them under Websites & Domains → Logs. In RunCloud and similar managed hosting panels, logs are typically under the server or application settings. On WP Engine, Kinsta, and other managed WordPress hosts, logs are usually downloadable from the user dashboard or accessible via SSH.

What a log entry looks like:

56a41047@7dcae6ad.com - - [13/Jun/2026:09:14:22 +0000] "GET /contact/ HTTP/1.1" 200 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
774924ae@bd316356.com - - [13/Jun/2026:09:31:08 +0000] "GET /about/ HTTP/1.1" 200 "-" "Mozilla/5.0 AppleWebKit/537.36 (compatible; GPTBot/1.0; +https://openai.com/gptbot)"
d39b0396@d87d492d.com - - [13/Jun/2026:10:02:51 +0000] "GET /contact/ HTTP/1.1" 200 "-" "python-requests/2.31.0"

The user-agent string at the end of each line identifies the visitor. Googlebot is Google’s crawler. GPTBot is OpenAI’s. python-requests/2.31.0 — no recognized bot name, just a generic Python HTTP library — is almost certainly a scraper or automated tool, not a legitimate crawler.

Searching your logs for specific bots:

If you have SSH access, you can grep your access log directly for specific user-agent strings:

# Find all GPTBot visits
grep "GPTBot" /var/log/apache2/access.log
# Find all contact page visits by any bot
grep "/contact/" /var/log/apache2/access.log | grep -v "Mozilla/5.0 (Windows\|Macintosh\|Linux x86_64)"
# Count visits per user agent (top 20)
awk '{print $12}' /var/log/apache2/access.log | sort | uniq -c | sort -rn | head -20

If you’re on shared hosting without SSH access, download the raw log file and open it in a text editor, then use Find to search for bot user-agent strings. It’s less elegant but gives you the same data.

Method 2: WordPress Plugins — Easier, No SSH Required

If direct log access isn’t available or isn’t comfortable, several WordPress plugins bring bot visibility into the admin dashboard without requiring any command-line work.

WP Mayor’s review of LLM Bot Tracker is worth reading if you want a detailed walkthrough of what the dashboard reveals — most site owners who install it report being surprised by the volume and frequency of AI crawler visits they had no idea were happening.

Method 3: Google Search Console — For Googlebot Specifically

Google Search Console provides detailed Googlebot-specific data that server logs can’t match. Under Settings → Crawl Stats, you can see how often Googlebot crawls your site, which pages it visits most, what response codes it’s getting, and how long pages take to load for the crawler.

This is the right tool for diagnosing crawl budget issues — situations where Googlebot isn’t crawling your important pages because it’s spending time on low-value content. It won’t show you other bots, but for the specific question of “is Google seeing my site correctly,” Search Console is more useful than raw log analysis.

Bing Webmaster Tools provides equivalent data for Bingbot. Both are free and worth having connected to your WordPress site regardless of your bot monitoring approach.

What to Do With What You Find

Once you can see which bots are crawling your site, three responses apply depending on what you find.

For AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended): Make a deliberate decision. If you want to block them, add the appropriate User-agent directives to your robots.txt file. Nexter WP’s 2026 crawler reference provides copy-paste robots.txt rules for each major AI crawler with their verified user-agent strings. If you want to allow them, do nothing — they’ll continue crawling.

For SEO tool bots (AhrefsBot, SemrushBot, MajesticBot): These crawl your site as part of competitor research by other site owners. Blocking them is optional — they don’t directly harm your site, but they do consume crawl budget and server resources. Many site owners block them via robots.txt or .htaccess.

For scrapers, contact harvesters, and unknown bots: This is where the action matters most. Generic user-agent strings like python-requests, curl, libwww-perl, or blank user agents are almost always automated tools without legitimate purpose. Bots hitting your contact page repeatedly with these strings are likely harvesting your contact information.

The Connection Between Bot Visibility and Contact Protection

Seeing which bots are crawling your site is diagnostic. It tells you what’s happening. What you do with that information is where protection actually lives.

For search engine bots and AI crawlers, robots.txt and content strategy are the tools. For contact-scraping bots, the most effective response isn’t blocking the bot after it’s found — it’s ensuring that even if it reads your pages, it finds nothing worth harvesting.

This is what WhoKnew Shield™ does. The free version automatically detects and applies dual-layer obfuscation to every email address across your entire WordPress site — no shortcodes, no manual configuration. When a scraper reads your contact page HTML, the email addresses aren’t readable patterns anymore. Pro extends this to phone numbers and physical addresses, adds rotating-key encryption that changes every 15 minutes, and includes Scraper Trap™ honeypots that catch and block bots the moment they probe your pages.

The WhoKnew Intelligence™ network — included in Shield Pro — maintains a community blocklist of over 15 million known bad actors blocked network-wide. When a known scraper visits your site, it’s blocked before it reads a single page. The combination of passive protection (encryption that makes contact data unreadable) and active blocking (honeypots and IP intelligence that stops scrapers in the act) is what separates a site that’s defended from one that’s simply harder to scrape.

For the full picture of what bad bots do once they’ve identified your site as a target, What Is Contact Scraping and Is It Happening to You? covers the detection methods in depth. For what to do if your contact information is already in circulation, What to Do When Your Email Is Already on Spam Lists covers the remediation path. And for why robots.txt alone doesn’t stop the bots that are actually a threat, What robots.txt Does (And Why Bad Bots Ignore It) covers the gap directly.

Who knew that the traffic report you check every week is missing more than half the story — and that the half you can’t see includes the bots reading your contact page right now.