Key Takeaways
- The bot abuse problem has gotten bad enough that major hosting providers are now blocking entire categories of requests based on User Agent string patterns — and legitimate SEO tools like Ahrefs, SEMrush, and Screaming Frog use the exact same string format that malicious bots do.
- SiteGround confirmed in a direct support exchange that its firewall now blocks any request containing
Mozilla/5.0 (compatible;— a pattern used by thousands of legitimate crawlers, because that same pattern has been so heavily exploited by malicious bots that blocking it wholesale was the practical solution. - This is a blunt-force defense against a genuine problem: hosting infrastructure is absorbing billions of bot requests, and pattern-matching on User Agent strings is one of the few interventions that scales across millions of hosted sites simultaneously without requiring per-site configuration.
- The same bot abuse problem driving hosting-level blocking is the reason your WordPress site’s contact information needs protection at the application layer — by the time a malicious scraper reaches your server, your hosting provider may or may not stop it, but the contact data sitting in plain HTML is exposed regardless.
- The fix for blocked SEO tools is to contact both your hosting provider and the tool vendor — but the fix for contact data sitting exposed in your own source code is something you control directly, independent of what any hosting firewall does or doesn’t catch.
If your Ahrefs crawl has been returning errors lately, or SEMrush is telling you it can’t access pages that load fine in a browser, you’re not imagining it and you haven’t done anything wrong. Something changed at the infrastructure level, and it’s happening to WordPress site owners running legitimate SEO tooling across multiple major hosting providers. Understanding what’s actually going on — and why — is more useful than a quick workaround, because this isn’t going away.
What’s Actually Happening
The bot problem has reached a scale that’s difficult to overstate. SiteGround is one of the largest WordPress hosting providers in the world, powering millions of sites globally — and their own published blog post on the subject describes the situation directly: in the early years of AI and scraping bots, traffic was “often so aggressive that we had to kill the requests in order to not let them overload our servers.” WP Engine reported mitigating 75 billion bot requests in 2025 alone. Cloudflare’s Q1 2026 analysis found that AI training crawlers like ClaudeBot make 20,583 crawl requests for every referral they send back — Google sends one referral per five crawl requests, by comparison.
Against that backdrop, hosting providers started making infrastructure-level decisions about which requests to allow and which to block — and those decisions are being made by pattern-matching on User Agent strings, because that’s the only tool that scales across millions of hosted sites simultaneously. The problem is that the User Agent string format most legitimate SEO crawlers use — a standardized, decades-old convention — is the same format that malicious bots adopted because it’s a legitimate-looking string that tends to get through basic filters.
75B
20,583
2.5M+
A Real Exchange With SiteGround Support
The following is an actual support conversation that illustrates exactly how this is playing out in practice. A WordPress site owner using a third-party SEO crawling tool found their crawler blocked and contacted SiteGround support. The response from SiteGround’s technical team was explicit:
SiteGround Support — June 30, 2026“We noticed a very large number of malicious requests using User Agents that contain the Mozilla/5.0 (compatible; string, so such requests are blocked on our servers… in order to avoid similar issues in the future, I would advise you to contact the remote service and ask them to change the User Agent used to send these requests and omit the Mozilla/5.0 (compatible; string.”
When asked whether the specific User Agent could be whitelisted to allow legitimate use, the response was equally direct: “I am afraid that it will not be possible to whitelist this User Agent, as our System does not allow this.”
That last line is the detail worth sitting with: SiteGround’s current implementation doesn’t allow whitelisting individual User Agents as an exception to this rule. The block is at the infrastructure level, applied uniformly, and the only resolution available is to ask the tool vendor to change their User Agent string — something outside the site owner’s control entirely.
Firewall block response
Whitelisting not supported statement
It’s Not Just SiteGround
This is an industry-wide response to an industry-wide problem, not a quirk of one hosting provider’s configuration. Across the hosting landscape, providers are making similar choices through different mechanisms. Analysis of hosting provider responses to the bot surge shows IONOS returning HTTP 429 “Too Many Requests” responses to certain crawlers on shared hosting; Cloud86 explicitly rate-limiting Ahrefs and SEMrush crawlers to two requests per minute; and Cloudflare’s broader platform implementing multiple tiers of bot control that have caught legitimate SEO tools as collateral damage in user-configured rule sets.
The Sledgehammer Problem
What’s happening at the hosting layer is a sledgehammer solution — genuinely necessary at the scale hosting providers are operating, but with unavoidable collateral damage on legitimate tooling that happens to share the same User Agent pattern as malicious traffic. This is the inherent limitation of pattern-matching at the string level: a filter that’s specific enough to catch all the bad bots catches legitimate crawlers too, and a filter that’s permissive enough to let legitimate crawlers through lets many bad bots through with them.
A hosting firewall blocking everything with Mozilla/5.0 (compatible; in its User Agent is the digital equivalent of a security guard stopping everyone wearing a hoodie. It catches most of the people you want to stop. It also stops some people you didn’t want to stop.
Why This Connects Directly to Contact Data Protection
The same underlying dynamic — bots operating at a scale that forces blunt-force infrastructure responses — is precisely the reason contact information sitting in plain HTML on a WordPress site needs protection at the application layer rather than relying on hosting-level defenses alone. A hosting firewall blocking User Agent patterns will catch some malicious scrapers and miss others, depending entirely on whether the scraper’s User Agent happens to match the pattern being blocked. A scraper sophisticated enough to use a clean, non-flagged User Agent string gets through hosting-level pattern filters entirely.
The approach Shield takes is the opposite of pattern-matching: rather than trying to identify malicious requests by what they say they are (a User Agent string), it makes the contact data itself unreadable regardless of who’s asking, and catches bots through behavioral traps that good bots never trigger and bad bots invariably do. A hosting firewall watches the door. Shield watches what’s behind it.
❌ Sledgehammer: Pattern Matching
- Block everything with
Mozilla/5.0 (compatible;in User Agent - Catches most bad bots — but also Ahrefs, SEMrush, Moz, and hundreds of legitimate tools
- Sophisticated scrapers using clean User Agents get through entirely
- Site owner has no control over the rule or its exceptions
✓ Surgical: Behavioral Detection
- Make contact data unreadable in the HTML regardless of who’s asking
- Catch bots by what they do (triggering honeypots) rather than what they claim to be
- Good bots — Googlebot, Ahrefs, SEMrush — never trigger honeypot traps
- Site owner controls the protection directly at the application layer
What You Can Do Right Now About Your Blocked SEO Tools
- 1
Contact your hosting provider’s support directly and ask them to confirm whether your specific tool’s User Agent or IP address is being blocked at the firewall level — and whether any whitelisting options are available on your plan.
- 2
Contact the SEO tool vendor simultaneously and explain the specific User Agent string being blocked. Ask them whether they have an alternative User Agent format or IP address list they can provide for whitelisting. This is the resolution path SiteGround’s support explicitly recommended.
- 3
Don’t assume your SEO data is accurate in the meantime. If your crawler has been periodically blocked, crawl data from the affected period may be incomplete — particularly for any analysis of page accessibility or internal linking that depends on successfully reaching every URL.
- 4
Consider running SEO crawls through a non-blocked environment temporarily while the tool vendor updates their User Agent — some tools allow crawling from a local instance rather than their cloud infrastructure, which may use a different User Agent string that isn’t currently flagged.
Further Reading
- Beyond the Firewall: Your Firewall Is Watching the Wrong Door — why network-level defenses and application-level contact protection address different parts of the same problem.
- What Is a Scraper Trap Honeypot and How Does It Catch Bots? — the behavioral detection approach that stops scrapers without pattern-matching on User Agent strings.
- Why Most WordPress Contact Protection Plugins Are Useless — why single-layer defenses at any level leave gaps that more sophisticated bots exploit.
Who knew the thing that finally broke your SEO audit tool was the same bot epidemic your contact page has been quietly feeding for years.
