How to Set Up Rate Limiting to Stop Bots

Rate limiting stops bots by capping how many requests a single IP can make in a set window—usually per minute—and returning HTTP 429 Too Many Requests once the cap is crossed. It is a blunt, cheap, and effective first filter for scraper bursts, credential-stuffing runs, and form spam. But it works best when you combine the per-IP cap with path-level rules (protect login, signup, and checkout) and IP reputation (flag datacenter ranges), then tune the thresholds from your logs instead of guessing.

This guide gives you the setup as five ordered steps, with a verification check after each one. Plan for about an hour of work: thirty minutes to configure, thirty minutes to watch and tune.

What rate limiting actually stops

Rate limiting is a traffic cop, not a bot detector. It does not care why a request arrives, only how often. That single rule catches the loudest bots first.

It stops these well:

• Scrapers pulling hundreds of pages per minute
• Brute-force and credential-stuffing runs on login
• Form-spam and fake signup bursts
• Comment spam and vote faking

It does not stop these:

• Patient scrapers that stay under your threshold
• Botnets spread across thousands of residential IPs
• Bots that make only a few requests per session
• A human attacker already holding a valid login

That is why the question is "rate limiting to stop bots," not just "rate limiting." A cap catches volume. To catch the quiet ones, you add reputation and behavior checks. These steps build the first layer; Step 5 shows you how to see the rest.

How rate limiting works: three algorithms in plain terms

You do not need to write the algorithm yourself. Most servers, CDNs, and gateways expose it as a config option. Knowing which one you are using matters because each behaves differently under a burst.

• Fixed window. Counts requests per minute (or per chosen period). Simple and cheap, but a burst near the end of one minute can bleed into the next, letting a bot double its effective rate.
• Sliding window. Counts over a rolling window, such as the last 60 seconds. Smoother and avoids the double-burst problem at the cost of a little more storage.
• Token bucket. Allows a set burst size, then refills at a steady rate. Best for APIs where a short conversation needs several fast calls before settling down.

Pick by workload: login and signup get a strict sliding window; API endpoints get a token bucket with a generous burst; blog pages can use a simple fixed window because you barely care if one bursts.

Step 1: Choose where to enforce the limit

You have three realistic places, and they are not mutually exclusive.

• Web server or load balancer — nginx limit_req, HAProxy, or your gateway. Fastest, catches traffic before the app sees it. Good first choice.
• CDN or WAF — Cloudflare Rate Limiting, AWS WAF, and similar. Easy to configure, protects the origin from floods, and can use managed IP reputations.
• Application layer — middleware such as express-rate-limit or django-ratelimit, often backed by Redis. Most control, and you can scope by user ID or API key. The catch: the app must be running to enforce it, so it will not stop a flood that crashes the origin.

A useful rule: put the hard cap at the CDN or load balancer, and the smart cap (path-specific, user-scoped) at the app layer. Each layer covers the other's blind spot.

Verify: send a slow loop of requests through each layer and confirm they pass. Then confirm a fast loop trips only the layer you intended.

Step 2: Pick a starting threshold

Do not guess. Read your actual usage first. Check analytics or logs for the 95th percentile of requests per IP per minute across a normal week, then set the cap at about double that. Tighten in small steps afterward.

Sensible starting points for most sites:

• Public pages and blog: 120–300 requests per minute per IP
• Login and signup: 5–10 per minute per IP (humans rarely exceed 2)
• Search: 20–60 per minute per IP
• API without auth: 60–120 per minute per IP
• API with auth: 10–30 per minute per API key

These are starting values, not gospel. Your real threshold comes from observing your own traffic, which is exactly what Step 5 does.

Verify: pull the 95th percentile from a week of logs and confirm your cap is roughly double it.

Step 3: Configure the cap and the 429 response

In nginx you define a shared zone keyed by IP with a rate, then attach it to the location you want to protect. A typical login rule looks like: a limit_req_zone named login using $binary_remote_addr at rate=10r/m, then limit_req zone=login burst=5 nodelay inside the login location. The burst lets a few extra requests pass before rejection; nodelay returns 429 immediately instead of queueing.

Three settings matter most:

• Rate — the sustained cap, such as 10 requests per minute.
• Burst — how many extra requests you tolerate before rejecting.
• nodelay — reject immediately rather than queueing the overflow.

Always return the standard status: 429 Too Many Requests. Add a Retry-After header (for example, 60 seconds) so well-behaved clients back off on their own. For browsers, you can also serve a lightweight challenge page instead of a bare 429, but only after the same IP keeps crossing the threshold.

Log every 429. You need the client IP, the path, whether the IP passed a reputation check, and the user-agent family. That log is the evidence you read in Step 5.

Verify: fire 30 requests in 10 seconds from one IP. You should see all pass up to the burst, then a stream of 429s.

Step 4: Add path-level rules and IP reputation

A blanket cap on the whole site will break normal browsing. Aim the limits at the paths that matter most.

• /login, /signup, /register — highest fraud value, lowest legitimate volume. Strictest cap.
• /checkout, /cart — billing friction, large damage if abused. Keep a moderate cap.
• /search, /api/* — easy scrape targets. Medium cap, tighter for unauthenticated clients.
• Everything else — keep a generous blanket cap so ordinary page views never trip it.

IP reputation changes who even reaches the counting step. Most CDNs and WAFs flag datacenter and hosting ranges, anonymizers, and known VPN egress. Bots rotate through those ranges constantly. A simple rule: if the IP is flagged as a datacenter and the path is /login, start counting at one tenth of the normal cap. That blocks automation without touching home and office users, who usually sit behind residential-looking IPs.

Verify: from the same IP, hit a public page and then /login. The public page should pass; login should trip the cap far sooner.

Step 5: Watch the debug console and tune with evidence

This is the step most guides skip, and the one that separates a working setup from a support nightmare.

After you deploy, watch the rejected-request logs for at least 24 to 48 hours. Look for two things.

False positives. Real users who hit a 429. Check their behavior: normal mouse movement, scrolling, time on page, and click sequences. If a flagged session behaves human, your threshold is too tight. Raise it by about 50 percent and re-observe.

Bots that slip through. Sessions with superhuman input speed, no pointer movement, no scrolling, or unrealistically fast form fills. These are the quiet bots your cap missed. Their behavior is the evidence you use to tighten the relevant path rule.

Keep a simple mental model: a single anomaly is not a verdict. One 429, one fast form fill, one odd session—any of these could be a genuine person behind a corporate proxy or a privacy tool. Act only when several independent signals agree: the same IP keeps tripping the cap, and its behavior stays consistent with automation, such as robotic pointer paths or sub-millisecond inputs.

In practice, a debug console helps here more than raw logs. It shows, for any flagged session, which checks fired and why. Instead of guessing at a threshold, you read the breakdown of what looked bot-like and what looked human, then adjust the one rule that mattered.

Verify: pick the ten most frequent rejected IPs and open each session's check breakdown. If most look human, loosen the rule. If most look automated, tighten it.

Common mistakes when tuning rate limits

Limitations: when a 429 is not a bot verdict

Rate limiting is a blunt instrument. It will occasionally flag a real person—someone on a corporate NAT, on hotel Wi-Fi, or using a privacy browser that routes many users through one IP. A single 429 is not proof of a bot. Conversely, a patient bot that stays under the cap is invisible to it.

Treat the cap as a temporary throttle, not a permanent ban. Keep 429 blocks short until you have evidence from several independent checks. The first layer catches volume; the second layer—reputation and behavior—catches precision.

Key facts at a glance

FAQ

How many requests per minute should I allow per IP?

Start at 120–300 for public pages and 5–10 for login, then read your logs. Your real threshold comes from the 95th percentile of your own traffic, doubled as a safety margin.

What HTTP status should rate limiting return?

429 Too Many Requests, with a Retry-After header so clients know when to try again.

Should I ban an IP permanently after it hits the limit?

No. Start with a short block. Permanent bans should wait for multi-signal evidence, because shared IPs also carry real users.

Will rate limiting stop a distributed botnet?

Not alone. Thousands of residential IPs each make a few requests, so no single IP trips the cap. Add IP reputation and behavior checks on top.

Where should I enforce the limit?

At the CDN or load balancer for the hard cap, and at the app layer for path- and user-scoped rules. Defense in depth beats either one alone.

How do I know if I blocked a real user?

Open the flagged session's behavior breakdown. Mouse movement, scrolling, and natural timing point to a human; robotic paths and sub-millisecond inputs point to a bot.

Do I need Redis or a database to count requests?

For a single server, in-memory counters are fine. For multiple servers, use a shared store like Redis or a CDN rule so counts agree across nodes.