How to Prevent Bots from Scraping Your Website: A Layered Defense Guide

Most scraping isn't stopped by a single tool. You need layers: basic barriers that deter casual scripts, challenges that raise the cost for determined scrapers, and behavioral signals that expose automation even when it mimics human traffic. The final layer is evidence collection — detailed, session-by-session proof you can submit to Google and Meta for refunds.

Understand what you're up against

Scrapers range from simple curl scripts to full browser automation frameworks like Playwright, Puppeteer, and Selenium. Basic bots identify themselves in the user-agent string. Advanced ones rotate residential IPs, spoof headers, and run real browser engines with stealth plugins that hide automation markers. No single check catches all of them.

BotRefund runs 106 independent browser, network, device, and behavioral checks per session. Each check produces one piece of evidence — not a verdict. The system cross-references signals and feeds the complete pattern into an AI model that reaches 99% accuracy by corroboration, not by any single rule.

Layer 1: Basic barriers that cost almost nothing

1. Declare intent with robots.txt. It won't stop malicious bots, but it tells compliant crawlers where they're welcome and gives you a policy baseline for abuse reports.
2. Rate-limit by IP and session. Set thresholds that allow human browsing but throttle rapid-fire requests. Apply stricter limits on login, search, and API endpoints.
3. Block known bad IP ranges. Maintain a deny list of data-center ASNs, VPN exit nodes, and previously flagged addresses. Update it weekly.
4. Require valid TLS and HTTP/2. Many low-end scrapers still speak HTTP/1.1 or skip certificate validation. Rejecting them costs nothing and filters noise.

Layer 2: Challenges that raise the scraper's cost

1. Serve JavaScript challenges. Require the client to execute a small script that computes a token. Headless browsers without full JS engines fail silently.
2. Deploy CAPTCHAs selectively. Show them only when risk signals accumulate — unusual velocity, missing cookies, or fingerprint anomalies. Blanket CAPTCHAs hurt conversion.
3. Use honeypot fields and trap links. Add form fields hidden via CSS (not display:none) and links humans never see. Submissions that fill them are automated.
4. Enforce referrer and origin checks. Reject requests that lack expected headers or come from unexpected origins, especially on state-changing endpoints.

Layer 3: Browser fingerprinting and behavioral signals

This is where automation frameworks betray themselves. Even when Playwright runs a real Chromium binary, the initialization scripts it injects leave detectable inconsistencies.

Playwright init script detection

Automation tools often patch or hide browser APIs to avoid detection. Those patches can break when the browser is probed from another angle — for example, an API behaves differently inside an iframe versus the top frame. BotRefund's Playwright Init Scripts check looks for this mismatch. A normal browser runs standard APIs as designed; an automated browser often reveals the patch when checked from a different context.

Scrollbar width leak

Scripts can send clicks and scrolls, but they struggle to reproduce the varied timing, hesitation, and micro-movements of real people. The Scrollbar Width Leak check flags sessions where scrolling behavior is too uniform or where the reported scrollbar dimensions don't match the interaction pattern.

Clean context iframe

Automation tools often modify browser APIs globally. When a clean iframe is created, those modifications may not propagate correctly, creating a detectable inconsistency between the parent and iframe contexts.

Pointer and motion behavior

Human mouse movement has tremor, curvature, and variable speed. Bots often move in straight lines, at superhuman speed (<1ms), or snap to grid-aligned coordinates. Ghost clicks — click events without the preceding human intent sequence — are another reliable signal.

Layer 4: Collect evidence that ad platforms accept

Blocking isn't enough if you're paying for the traffic. Google and Meta issue invalid-activity credits only when you submit structured evidence: click IDs (GCLIDs, fbclids), timestamps, session recordings, and signal-by-signal reasoning. BotRefund formats reports in the exact structure platform reviewers expect. Across 2,500+ audits, 83% of clients recover funds.

1. Capture every click ID. Store GCLID, fbclid, msclkid, and other attribution parameters alongside the session record.
2. Record session replays. Visual proof of non-human behavior (no scrolling, instant form fills, linear mouse paths) is persuasive to reviewers.
3. Document the signal chain. List each independent check that fired, why it matters, and how it corroborates others. Raw rule hits get rejected; correlated patterns get approved.
4. Submit within the platform's window. Google typically allows 60 days; Meta's window varies. Automate the claim generation so you never miss a deadline.

Common mistakes that leave gaps

• Relying only on robots.txt or IP blocks. Determined scrapers ignore both.
• Using a single CAPTCHA vendor. Solver farms specialize in specific CAPTCHA types. Rotate or combine.
• Treating one anomaly as proof. Privacy tools, corporate proxies, and unusual devices create false positives. Always cross-check.
• Not preserving attribution before changing campaigns. If you pause or restructure before exporting click IDs, you lose the evidence trail.
• Assuming server logs are enough. Server-side data misses client-side behavior — mouse movement, scroll depth, browser API consistency — that distinguishes sophisticated bots.

Verification: How to know it's working

1. Run a controlled test: deploy a known automation script (Playwright with stealth plugin) against a staging page instrumented with your detection.
2. Confirm the session is flagged and the evidence panel shows multiple independent signals (Playwright init script, pointer behavior, scrollbar leak, etc.).
3. Verify the exported report includes click IDs, session recording link, and a signal-by-signal explanation.
4. Submit a test claim to Google or Meta (or use their invalid-traffic reporting tools) and confirm the evidence format is accepted.
5. Monitor the false-positive rate: check sessions flagged as bot that came from known human sources (internal team, verified customers). Adjust thresholds if needed.

Key facts

Metric	Detail	Source
Independent detection checks per session	106+	S1
Bot detection accuracy	99% via AI corroboration	S1
Brands audited	2,500+	S2
Client refund recovery rate	83%	S2
Ad budget lost to bot clicks (typical)	Up to 20%	S2
Report format	Refund-ready, accepted by Google and Meta	S2
Negotiation experience	2,500+ audits, direct platform engagement	S2

Limitations and when this advice doesn't apply

• DDoS-scale volumetric attacks require edge/CDN mitigation (Cloudflare, Akamai, DataDome). This guide covers scraping and click fraud, not network-layer floods.
• Zero-day browser exploits that perfectly mimic human behavior may evade fingerprinting until signatures update.
• Internal tools and testing scripts will be flagged unless allowlisted by IP, user-agent, or authentication token.
• Privacy-focused users (Tor, hardened browsers, anti-fingerprinting extensions) can trigger signals. Always cross-check before blocking.
• Non-ad traffic — if you don't run paid campaigns, the refund layer is irrelevant; focus on Layers 1-3.

FAQ

Does robots.txt actually stop scrapers?

No. It only instructs compliant crawlers. Malicious bots ignore it. Treat it as policy documentation, not enforcement.

Which CAPTCHA should I use?

Rotate between two providers (e.g., hCaptcha and Turnstile) and trigger them only on risky sessions. Blanket CAPTCHAs reduce conversions by 10-30% on some sites.

Can't scrapers just use residential proxies and real browsers?

Yes, but they still need automation to scale. The automation framework (Playwright, Puppeteer, Selenium) leaves fingerprints in browser APIs, timing, and interaction patterns that client-side checks detect.

How long does it take to get a refund from Google or Meta?

Typically 2-6 weeks after submitting a complete, well-structured claim. Incomplete claims get rejected and reset the clock.

What if I don't run ads — do I still need Layer 4?

No. Layer 4 is for recovering ad spend. If you only care about content protection and server load, Layers 1-3 are sufficient.

How often should I update IP blocklists?

Weekly at minimum. Data-center ranges and VPN exit nodes change daily. Automate pulls from reputable threat-intel feeds.

Will these defenses break legitimate traffic from corporate networks?

They can. Corporate proxies, VPNs, and security appliances sometimes strip headers or modify TLS fingerprints. Monitor false positives and allowlist known partner ranges.

Further reading and comparison sources

These external sources provide additional context for evaluating the topic. Their inclusion is not an endorsement.

• r/webscraping on Reddit: What are the best practices to prevent ...
• How Do I Stop Bots from Crawling My Website? - Anura.io
• Web Scraping Protection: How to Prevent Web Scraping - DataDome

How BotRefund can help

BotRefund installs a lightweight script that runs 106+ independent browser, network, device, and behavioral checks on every session. It detects automation frameworks like Playwright through init-script inconsistencies, scrollbar leaks, and clean-context iframe mismatches — signals that server logs miss. Each session gets a signal-by-signal evidence panel and a session replay. The platform assembles refund-ready reports formatted for Google and Meta reviewers, including click IDs, timestamps, and corroborated reasoning. Across 2,500+ audits, 83% of clients recover ad spend. You need to add the script to your pages and connect your ad accounts for attribution capture; the system does not block traffic directly — it supplies the evidence you use to claim refunds and feed suppression lists.