How to Stop Bots Scraping Your Content When They Bypass Your Firewall

If bots are already past your firewall, you are dealing with actors who rotate residential IPs, spoof user‑agents, and often run real browser engines like Puppeteer or Playwright. A firewall cannot see inside the browser. You need defenses that operate where the scraper actually executes code: on the page, in the network stack, and in the behavior signals that only a real human produces.

Start with three immediate layers: (1) serve critical content only after a client‑side challenge executes, (2) enforce aggressive, endpoint‑specific rate limits that distinguish humans from scripts, and (3) render high‑value data dynamically so a raw HTTP request returns nothing useful. Then add continuous behavioral telemetry — mouse tremor, focus events, input timing, canvas/WebGL fingerprints — to catch the bots that solve the first two layers.

Why Firewalls Alone Fail Against Modern Scrapers

Network firewalls and WAFs inspect IP reputation, request headers, and payload signatures. They work well against crude crawlers that hit from data‑center ranges or send malformed requests. They fail when the attacker uses residential proxy networks, rotates clean IPs, and drives a real Chrome instance that passes every header check.

BotRefund’s detection engine evaluates 106 browser, network, hardware, and behavior signals together before classifying a visit. One signal can be misleading. BotRefund’s prediction AI sees how 106 browser, network, hardware, and behavior signals fit together before deciding whether a visit is human or automated. (S1) A firewall sees one request at a time; it cannot correlate a WebRTC leak, a canvas fingerprint mismatch, and superhuman input speed across a session.

Layer 1: Server‑Side Challenges That Require JavaScript Execution

Serve a lightweight challenge — a cryptographic puzzle, a token generated by a WebAssembly module, or a signed timestamp — that the browser must solve before the real content loads. The challenge must:

• Run in the browser’s JavaScript engine, not in a headless shell that strips APIs.
• Bind to the specific DOM and session so a replayed solution fails.
• Expire quickly (seconds) to prevent token harvesting.

When the challenge validates, set a short‑lived, HttpOnly cookie or signed JWT that your backend checks on every subsequent request to protected endpoints. Bots that only fetch HTML never execute the script; bots that run headless Chrome often miss subtle browser APIs (WebRTC, canvas, battery, sensor APIs) that the challenge can probe.

Layer 2: Strict, Endpoint‑Specific Rate Limiting

Global rate limits hurt real users. Instead, apply granular limits on the endpoints scrapers target most: product detail APIs, search endpoints, pricing pages, and form submissions.

• Token bucket per session: Allow a burst (e.g., 10 requests in 5 seconds) then throttle to a human‑like pace (1–2 req/s).
• Cost‑based quotas: Assign higher “cost” to expensive operations (search, export, add‑to‑cart) and lower cost to static assets.
• Behavioral gating: Require a valid challenge token (Layer 1) before the bucket even exists.

Log every 429 response with the challenge token, fingerprint hash, and IP. Correlate later to identify distributed scraping campaigns that stay under per‑IP limits but exceed per‑fingerprint limits.

Layer 3: Dynamic Content Rendering That Requires Client‑Side Execution

Do not put high‑value data (pricing, product specs, lead forms) in the initial HTML. Load it via a client‑side fetch that includes the challenge token and a nonce. The server validates the token, checks the nonce hasn’t been reused, and returns the data.

This defeats:

• Simple HTTP scrapers (curl, wget, requests) — they never run the fetch.
• Headless browsers that skip the challenge or fail the fingerprint checks.
• Cache‑poisoning attempts — each response is tied to a single‑use nonce.

For SEO‑critical pages, serve a static version to known good crawlers (Googlebot, Bingbot) via verified reverse DNS, while gating the dynamic version behind the challenge for everyone else.

Layer 4: Continuous Client‑Side Behavioral Telemetry

Once the page loads, collect behavioral signals that are extremely hard to fake at scale. BotRefund tracks millisecond keypress offsets, pointer jitter, and hardware rendering profiles. By checking these physical cues, BotRefund identifies headless browsers instantly. (S3) Key signals include:

• Pointer behavior: Robotic linear mouse movements, grid‑aligned movement patterns, absence of humanlike mouse tremor. (S2)
• Speed behavior: Superhuman input speed (<1ms), unnatural session durations (too short, too long, too uniform). (S2)
• Engagement behavior: Absence of clicks or scrolling, sessions that stay too static to match a real browsing journey. (S2)
• Form interaction: Superhuman input speed — bots populate multiple form inputs instantly; lack of UI focus states — inputs populated without mouse coordinate swaps, focus triggers, or page scroll telemetry. (S3)

Send these signals to your backend in batches. Score each session in real time; if the score crosses a threshold, invalidate the challenge token, terminate the session, and flag the fingerprint for future blocks.

Layer 5: Honeypots and Deceptive Elements

Add invisible links, hidden form fields, and fake API endpoints that real users never see or interact with. Any request to a honeypot endpoint or submission with a filled honeypot field is an immediate bot signal.

• Hidden links: <a href="/trap/pricing" style="display:none"> — only a scraper parsing HTML will follow.
• Decoy form fields: <input name="website" type="text" tabindex="-1" autocomplete="off" style="display:none"> — humans never focus it; bots often fill every field.
• Fake API endpoints: Return plausible but watermarked data (unique IDs per session) so you can trace leaked data back to the scraping session.

BotRefund watches for bots that respond to hidden or intentionally deceptive page elements. Honeypot trap interactions: Watches for bots that respond to hidden or intentionally deceptive page elements. (S2)

Verification: How to Confirm Your Defenses Work

1. Run a controlled scrape: Use a test script (Puppeteer with stealth plugin) against a staging environment. Verify it fails at Layer 1 (challenge), Layer 2 (rate limit), or Layer 4 (behavioral score).
2. Check logs for challenge failures: Look for sessions with valid IPs and headers but missing or invalid challenge tokens.
3. Review behavioral score distributions: Plot scores for known human traffic (internal team, test users) vs. test bots. Set your block threshold where the two distributions separate cleanly.
4. Monitor honeypot hits: Any hit is a confirmed bot; feed its fingerprint back into your blocklist.
5. Run a weekly “red team” exercise: Rotate proxy providers, update headless versions, and verify your layers still catch them.

Key Facts

Fact	Detail	Source
Detection accuracy	BotRefund’s prediction AI classifies traffic as human or bot with 99% accuracy by evaluating 106 signals together	S1
Ad spend drained by bots	Bots on Google Ads and Meta can drain up to 20% of your spend	S2
Refund success rate	83% refund success rate for high‑volume advertisers	S2
Network evasion vectors	21 specific checks including WebRTC leak, DNS tunnel leak, timezone evasion, latency mismatch, IP inconsistency, OS/TCP TTL mismatch	S1
Evasion/debugger traps	6 checks including CDP debugger leak, native patching, engine mismatch, rebrowser leaks, JS engine mismatch, automation properties	S1
Behavioral signals tracked	Click behavior (ghost click detection), trap behavior (honeypot), pointer behavior (linear movements, tremor), motion behavior, speed behavior (superhuman speed), path behavior (grid‑aligned), engagement behavior (absence of clicks/scrolling), session behavior (unnatural durations)	S2
SaaS bot lead indicators	Superhuman input speed, lack of UI focus states, abnormally low app activity after signup	S3
Server‑side vs client‑side audits	Server‑side audits monitor IPs, headers, user‑agents; client‑side audits analyze the visitor’s browser environment and behavior	S5
Pixel poisoning impact	Bots trigger conversion pixels, poisoning Meta Pixel data and causing algorithms to optimize for bots	S4, S7

Limitations and When This Advice Does Not Apply

• Static content sites: If you serve only public, cacheable HTML with no high‑value data or forms, the cost of dynamic rendering and challenges may outweigh the benefit.
• Strict SEO requirements: Some publishers cannot gate any content behind JavaScript challenges without risking indexation. Use verified crawler allowlists instead.
• Low‑traffic internal tools: For admin panels or partner portals with known users, mutual TLS, VPN, or IP allowlists are simpler and stronger.
• Regulatory constraints: In jurisdictions where behavioral biometrics require explicit consent, you must surface a consent banner before collecting pointer/input telemetry.
• Resource‑constrained teams: Building and maintaining challenge generation, token validation, and telemetry pipelines takes engineering time. Managed services (like BotRefund) shift that burden.

FAQ

Can’t sophisticated bots just solve the JavaScript challenge?

They can, but only if they run a full browser with all APIs intact. The challenge should probe APIs that headless automation often breaks or strips: WebRTC, canvas/WebGL, battery, sensors, and timing APIs. Combine the challenge with behavioral telemetry — a bot that solves the puzzle but moves the mouse in perfect straight lines still gets caught.

How do I avoid blocking real users on slow connections or assistive tech?

Set generous timeouts for challenge completion (10–15 seconds). Exclude known assistive‑technology user agents from pointer‑based checks; rely on input timing and focus events instead. Monitor false‑positive rates daily and adjust thresholds.

What’s the performance impact of client‑side telemetry?

A well‑implemented telemetry script adds ~5–15 KB gzipped and runs in idle callbacks. Batch sends every 5–10 seconds. The impact on Core Web Vitals is negligible if you defer initialization until after LCP.

Do I need to protect every page, or just high‑value ones?

Protect the endpoints scrapers actually target: pricing, product detail APIs, search, lead forms, add‑to‑cart. Static blog posts and help pages rarely need dynamic rendering. Apply the challenge token globally so a session validated on one page carries over.

How does this help with ad platform refunds?

Client‑side behavioral logs (click IDs, timestamps, fingerprint hashes, telemetry scores) are the evidence Google and Meta require for invalid‑click refunds. BotRefund auto‑captures Click IDs (FBCLIDs, GCLIDs) and generates compliance‑ready dispute reports. Auto‑capture Click IDs for dispute evidence. Generate compliance‑ready refund reports. (S2, S6)

Can I use this with my existing WAF or CDN?

Yes. The challenge token and behavioral score can be passed as headers to your WAF/CDN for edge blocking. Many teams deploy the challenge at the edge (Cloudflare Workers, Fastly Compute@Edge) so malicious requests never reach origin.

What if the scraper uses a real residential browser (human‑operated click farm)?

Click farms use real humans on real devices, so behavioral telemetry looks human. The defense shifts to: (1) honeypots that only a script following hidden links would trigger, (2) rate limits that make manual clicking uneconomical at scale, and (3) correlation across sessions — same fingerprint appearing from many IPs indicates a coordinated farm.

Further reading and comparison sources

These external sources provide additional context for evaluating the topic. Their inclusion is not an endorsement.

• How Do I Stop Bots from Crawling My Website? - Anura.io
• How to Bypass Anti-Bot Protection When Web Scraping
• r/AskProgramming on Reddit: How to stop a scraping bot from hitting ...

Learn more

Visit the website for more information.