Key takeaways from this article:
- More than half of all internet traffic — 51% — is now generated by automated bots, not humans. That figure comes from Imperva’s 2025 Bad Bot Report, which analyzed 13 trillion bot requests across thousands of domains worldwide.
- Of that 51%, roughly 14% is legitimate bot traffic: search engine crawlers, uptime monitors, and similar tools you want on your site. The remaining 37% is malicious — scrapers, credential stuffers, spammers, and DDoS tools that are actively working against you.
- Good bots and bad bots are not always visually distinguishable from each other. Both make HTTP requests, both read your HTML source, and some bad bots specifically impersonate good ones to avoid detection.
- The critical difference for WordPress site owners is behavioral: good bots cooperate with your preferences (robots.txt, crawl rate controls), while bad bots are designed to ignore them. Your robots.txt file is completely invisible to contact scrapers.
- Understanding this distinction changes how you think about WordPress security — and specifically why contact information protection requires a different approach than anything designed to stop cooperative crawlers.
When most WordPress site owners think about bots, they picture search engine crawlers — Googlebot indexing their content, Bingbot checking their pages, the familiar agents that SEO plugins are designed to work with. That mental model is accurate as far as it goes. It just captures about 14% of the automated traffic hitting your site.
The other 37% — the majority of all bot traffic — is doing something else entirely.
According to Imperva’s 2025 Bad Bot Report, which analyzed 13 trillion automated requests across thousands of domains worldwide, bots now account for 51% of all global web traffic — the first time automated activity has outnumbered humans in a decade. The split breaks down as approximately 14% good bots and 37% bad bots. That 37% has risen for six consecutive years, driven largely by AI tools that have made bot creation dramatically cheaper and faster.
For WordPress site owners, the practical question isn’t whether bots are on your site — they are, constantly, starting within hours of a new site going live. The question is which ones are doing what, and what you can actually do about the ones working against you.
What Good Bots Actually Do
Legitimate bots serve functions you depend on, whether you realize it or not. The most important category for any WordPress site is search engine crawlers — Googlebot, Bingbot, and their equivalents from other search engines. These bots systematically visit your pages, read the content, follow internal links, and report back to the search index so your content can appear in search results. Without them, your site is invisible to organic search. Google and Bing crawlers together account for more than 60% of all known good bot traffic, according to HUMAN Security’s 2026 bot analysis.
Beyond search crawlers, the good bot category includes uptime monitors that check whether your site is responding, SEO tools that audit your page structure and link profile, social media preview generators that fetch page metadata when someone shares your URL, and a growing category of AI crawlers from companies training large language models. That last subcategory is new, fast-growing, and increasingly contested — Cloudflare reports that over 80% of its customers have chosen to block AI crawlers entirely, citing server load and the zero-click problem where AI tools serve your content in answers without sending users to your site.
The defining characteristic of all genuinely good bots is cooperative behavior. They identify themselves honestly through their user-agent string. They read and respect your robots.txt file. They honor crawl rate limits. They operate within the conventions that the web’s cooperative crawling infrastructure was built on.
What Bad Bots Actually Do
Bad bots share the same basic mechanism as good ones — they make HTTP requests to your server, receive HTML responses, and process the content. The difference is entirely in intent and behavior.
Data Harvesters
Read your HTML source and extract email addresses, phone numbers, and physical addresses using regex pattern matching. Feed the data into spam pipelines within hours of collection.
Login Attack Bots
Use harvested email addresses to attempt logins across thousands of sites with known leaked password combinations. Contact scraping enables credential stuffing by supplying the email list.
Content Thieves
Copy your article text, product descriptions, and pricing data for republication on competitor or spam sites. Often operate in bulk across thousands of sites simultaneously.
Bad bots don’t cooperate. They don’t identify themselves honestly. They don’t read robots.txt — or they read it and ignore it anyway. The contact scrapers that harvest email addresses from WordPress sites aren’t search crawlers operating under a cooperative framework. They’re harvesting tools designed to extract data as efficiently as possible, and your preference that they not be there is simply irrelevant to them.
This is not a theoretical distinction. Imperva’s data shows malicious bot traffic has increased for six consecutive years, rising from 32% of all internet traffic in 2023 to 37% in 2024. Akamai’s 2025 research recorded a 300% surge in AI-powered crawler traffic, much of it operating in a grey zone between legitimate indexing and aggressive data extraction. Some bad bots are now sophisticated enough to rotate IP addresses, mimic human browsing behavior, and specifically impersonate legitimate bots — with HUMAN Security finding that 5.7% of requests presenting as known AI crawler user agents were actually spoofed.
Some bad bots present themselves as Googlebot or other legitimate crawlers specifically to avoid detection and blocking. A bad bot claiming to be Googlebot in its user-agent string looks cooperative on the surface. The actual Googlebot can be verified through reverse DNS lookup — if the PTR record doesn’t resolve to a googlebot.com or google.com domain, it’s not Googlebot regardless of what it claims. Most contact scrapers don’t bother with this level of impersonation because they don’t need to — they’re not trying to get past robots.txt. They’re just reading your HTML and leaving.
The Grey Area: AI Crawlers
The Grey Area: AI CrawlersBetween the clearly good and clearly bad categories, a third category has emerged that doesn’t fit either label cleanly. AI crawlers from companies like OpenAI (GPTBot), Anthropic (ClaudeBot), Google (Google-Extended), and ByteDance operate at massive scale — Cloudflare’s AI Insights dashboard recorded approximately 50 billion AI crawler requests per day from its network alone by late 2025. They’re not malicious in the traditional sense — they’re not harvesting contact data or attempting logins. But they consume significant server resources, provide zero-click answers that reduce referral traffic, and operate in a legal and ethical grey zone around content licensing that’s still being resolved.
How you handle AI crawlers is a strategic decision rather than a security one. The robots.txt directives for blocking specific AI crawlers (User-agent: GPTBot / Disallow: /) are honored by most legitimate AI companies, which makes this one area where robots.txt does work as intended — because these are cooperative actors with reputational incentives to respect site owner preferences. The majority of Cloudflare customers have chosen to block them. Others welcome the indexing in hopes of AI citation traffic. There’s no single right answer.
Why This Distinction Matters for WordPress Specifically
WordPress sites receive disproportionate bot attention for a simple reason: the platform is predictable. Scrapers know where WordPress themes put contact information — footers, widget areas, contact pages, about pages — and they know how to navigate a WordPress site efficiently because the structure is consistent across tens of millions of installations. A bot trained on WordPress conventions can extract contact data from a new site in seconds.
The tools most WordPress site owners use to manage bot behavior — robots.txt, security plugins, firewalls — are built primarily for cooperative bots. Robots.txt works on Googlebot because Google has agreed to respect it. It does nothing to a contact scraper that has no incentive to cooperate. As we’ve covered previously, the file that legitimately controls search crawler behavior is simply invisible to adversarial bots.
This is why contact protection requires a fundamentally different approach. Contact scraping isn’t stopped at the door — it’s stopped at the source. If your email address doesn’t exist as readable text in your HTML source code, a scraper can read every byte of your page and extract nothing useful. That’s the protection that operates at the right layer — not the bot’s behavior, but the data’s accessibility.
WhoKnew Shield™ operates at the source code layer — encoding, encrypting, and optionally rendering contact information as images so that the data doesn’t exist as extractable text regardless of which bot is reading the page. The Scraper Trap™ honeypot system adds active detection: hidden elements that legitimate crawlers never interact with but that scrapers reliably trigger, identifying and permanently blocking them. Download the free version from WordPress.org and see what contact information currently exists as readable text in your site’s HTML.
Good bots respect your preferences and should be welcomed — they’re how your content gets found. Bad bots ignore your preferences and need to be addressed at the layer where they operate: the data they’re trying to extract. robots.txt is the right tool for the first problem. Source-level contact protection is the right tool for the second. Using one where the other is needed is why most WordPress sites remain vulnerable to contact scraping despite having security plugins installed.
Further Reading
- What robots.txt Does (And Why Bad Bots Ignore It) — why the most commonly used bot management tool doesn’t stop the bots that matter most.
- What Is Contact Scraping and Is It Happening to You? — how harvesting bots find and extract contact information from WordPress HTML.
- Why Wordfence Recorded Zero Attacks While WhoKnew Shield™ Caught 491+ — what happens when you add source-level contact protection alongside a traditional security plugin.
- The WordPress Security Layer Nobody Installs — how contact protection sits outside what firewalls and security plugins are designed to cover.
- Beyond the Firewall: Your Firewall Is Watching the Wrong Door — the reconnaissance phase of attacks and why stopping it upstream changes the threat picture entirely.
Who knew that the tool most WordPress site owners rely on to block bad bots — robots.txt — works perfectly on the bots that were never the problem, and does nothing to the ones that are.
