Key takeaways from this article:
- robots.txt is a request, not a firewall. It tells cooperative crawlers — like Googlebot — what to index. It has no technical mechanism to stop anything.
- Contact scrapers don’t read robots.txt. They’re not search crawlers operating on good-faith conventions — they’re harvesting tools that skip the file entirely and go straight to your page source.
- A 2025 Duke University study of 130 bots found that AI search crawlers and data harvesters rarely check robots.txt at all. A 2025 industry report found 13.26% of AI bot requests ignored robots.txt directives outright — up from 3.3% the previous year.
- Having robots.txt configured correctly is still worth doing for legitimate SEO reasons. It just doesn’t protect your contact information from the bots that are actually threatening it.
- The only protection that works against contact scrapers operates at the HTML source level — encoding, encrypting, or rendering contact information so it doesn’t exist as extractable text regardless of whether a bot is supposed to be there or not.
If you’ve ever looked into protecting your WordPress site from bots, robots.txt comes up immediately. Add a disallow rule, the thinking goes, and the bots won’t come. It sounds simple. It’s also, for the specific threat of contact scraping, almost completely beside the point.
robots.txt does something real and useful — just not what most WordPress site owners think it does. Understanding the actual scope of it changes how you think about what protection your site does and doesn’t have.
What robots.txt Actually Does
robots.txt is a plain text file that lives at the root of your website — accessible at yourdomain.com/robots.txt — and contains instructions for automated crawlers about which parts of your site they should and shouldn’t access. A typical entry looks like this:
User-agent: *
Disallow: /wp-admin/
This tells all crawlers not to index your WordPress admin area. Googlebot, Bingbot, and other legitimate search engine crawlers read this file before crawling your site and follow its directives. That’s genuinely useful — it keeps administrative pages out of search indexes, prevents duplicate content issues from being indexed, and gives you control over what appears in search results.
The critical phrase in all of that is “legitimate search engine crawlers.” robots.txt works because Google and other major search engines have agreed to respect it as a convention. That agreement is entirely voluntary. There is no technical mechanism in robots.txt that prevents any bot from ignoring it. The file is just text. Any program that requests your pages can simply skip reading it and proceed directly to your content.
The official name for robots.txt is the Robots Exclusion Protocol. The word “protocol” here means a convention — an agreed-upon behavior — not a technical enforcement mechanism. Cloudflare’s documentation puts it plainly: “robots.txt files express your preferences. They do not prevent crawler operators from crawling your content at a technical level.”
Why Bad Bots Don’t Read It
Legitimate search crawlers read robots.txt because they’re operated by companies that have reputational and legal incentives to behave well. Google’s business depends on being trusted by webmasters. Ignoring robots.txt would create enormous friction with the sites Google needs to index. The convention holds because compliance serves the crawler’s interests.
Contact scrapers and data harvesters operate under completely different incentives. They’re not building a search index — they’re harvesting email addresses, phone numbers, and physical addresses to sell to spam operations. Their business depends on extracting data as efficiently as possible. Reading a text file and then skipping pages it says to avoid directly reduces the amount of data they collect. There’s no upside to compliance and a clear downside.
The research bears this out. A 2025 Duke University study analyzed the behavior of 130 self-declared bots over 40 days and found that certain categories of bots — specifically AI search crawlers and data harvesters — rarely check robots.txt at all. A 2025 industry report tracking bot traffic found that 13.26% of AI bot requests ignored robots.txt directives outright, up from 3.3% just a year earlier. And that’s only the bots that identified themselves — anonymous scrapers that don’t declare their user-agent skip robots.txt at rates that are impossible to measure precisely but are understood to be much higher.
The Two Types of Bots You’re Actually Dealing With
Understanding why robots.txt works for some bots and not others requires understanding the difference between cooperative and adversarial crawlers.
Cooperative Crawlers
Googlebot, Bingbot, legitimate SEO tools. Read robots.txt. Follow its directives. Operated by companies with legal and reputational incentives to behave well. robots.txt works on these.
Adversarial Scrapers
Contact harvesters, spam bots, data brokers’ collection tools. Don’t read robots.txt. Skip it entirely. Operated anonymously with no reputational stake. robots.txt does nothing to these.
The bots that are a threat to your contact information are exclusively in the second category. A bot that’s harvesting email addresses to sell to spam lists has no reason to identify itself, no reason to read your robots.txt file, and no reason to respect it if it did. It requests your page, reads the HTML source, extracts every contact pattern it finds, and moves on to the next site. Your robots.txt file is irrelevant to that process.
What robots.txt Is Still Good For
None of this means robots.txt is useless — it means it’s useful for specific things that don’t include stopping contact scrapers.
robots.txt is genuinely valuable for:
- Keeping admin pages out of search indexes. Your WordPress dashboard, login pages, and internal tools shouldn’t appear in Google results. robots.txt handles this reliably because Googlebot respects it.
- Preventing duplicate content from being indexed. Paginated archives, tag pages, and parameter-heavy URLs can create duplicate content issues. robots.txt lets you exclude them cleanly.
- Controlling crawl budget on large sites. Telling search crawlers where to focus their attention matters for large sites with thousands of pages.
- Blocking specific legitimate crawlers. If you don’t want your content used for AI training by a specific company whose crawler respects robots.txt, the disallow directive is the right tool.
These are real SEO and site management benefits. Configure robots.txt correctly. Just don’t mistake it for security against the bots that are actually threatening your contact data.
What Actually Stops Contact Scrapers
If robots.txt doesn’t stop contact scrapers, the question becomes: what does?
The answer is protection that operates at the level where the scraping actually happens — the HTML source code. A scraper reads your page source and extracts contact patterns. The only way to defeat that is to ensure that contact patterns don’t exist in your source code as readable text, regardless of whether the scraper was supposed to be there.
This means one or more of the following:
- HTML entity encoding — scrambles your contact information in the source code so the characters that appear in the HTML don’t form a recognizable email or phone pattern, while the browser renders them correctly for human visitors. This stops basic scrapers but can be decoded by more sophisticated ones.
- Site-specific encryption — uses an encryption key unique to your site so that even if a scraper captures the encoded data, it can’t be reversed without the key. The contact information doesn’t exist in any usable form in the HTML.
- Image rendering — converts your contact information to a PNG image. No text exists in the HTML at all. There is literally nothing for a regex pattern to match.
- Active honeypot traps — hidden elements invisible to human visitors that scrapers interact with, identifying and blocking them before they reach your real contact data.
WhoKnew Shield™ combines all of these layers in a single plugin — dual-layer encoding in the free version, site-specific encryption, Silent Contact™ image rendering, and Scraper Trap™ honeypots in the Pro tier. It also auto-detects contact information across your entire site so there are no gaps from pages you forgot to protect. Download the free version directly from WordPress.org — it works immediately without any shortcodes required.
Think of robots.txt as a “no soliciting” sign on your front door. It works on delivery companies and legitimate visitors who respect the convention. It does nothing to stop someone who was never going to respect it in the first place. Contact scrapers are that second category. The only protection that works is making sure there’s nothing worth taking — which means your contact information doesn’t exist as readable text in your HTML source code.
Further Reading
- What Is Contact Scraping and Is It Happening to You? — how scrapers work and what the downstream consequences look like.
- Why Most WordPress Contact Protection Plugins Are Useless — why most protection approaches fail for the same reason robots.txt does.
- Beyond the Firewall: Your Firewall Is Watching the Wrong Door — why firewalls and robots.txt both address the wrong threat for contact protection.
- The WordPress Security Layer Nobody Installs — the reconnaissance phase of spam and why it requires a different category of protection.
- Where Robocallers Get Your Number (And How to Stop Them) — the path from your WordPress site to your phone’s spam folder.
Who knew that the file WordPress site owners rely on most to block bad bots is the one bad bots are least likely to read.
