Bot User Agents and HTTP Headers: Which Detection Signals Actually Work

Bots typically reveal themselves through HTTP headers in three recurring patterns: a User-Agent string that names an automation tool (the clearest being “HeadlessChrome” from Puppeteer, Selenium, or Playwright), a User-Agent that is empty or malformed, and a set of headers that contradict each other — like a Chrome User-Agent paired with missing Sec-CH-UA client hints or an Accept-Language list no installed browser would generate. The most useful signal is the third one: not any single header, but the mismatch between headers a real browser would send together.

The decision rule that matters: ask whether the header story holds together, not whether one field looks bot-like. A real Chrome session sends a Chrome User-Agent, matching client hints, consistent fetch metadata, and an Accept-Language header that reflects system languages. Automation tools borrow pieces of that story but rarely copy every piece at once. That gap is what server-side detection looks for.

What bot user agents actually look like

You will see three families of bot user agents in your logs.

Automated browser tools. Puppeteer, Selenium, and Playwright ship with headless Chromium by default. Their User-Agent typically contains the literal substring “HeadlessChrome” — for example, Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/120.0.0.0 Safari/537.36. Operators can override this string, so treat it as a strong hint, not proof.

Scripts and libraries. curl, Python's requests, Node fetch, and Go's HTTP client send plain User-Agents that name the tool. These are trivial to spot and trivial to fake. They show up in scraping, API probing, and health checks as well as fraud.

Named platform crawlers. Googlebot, Bingbot, and social platforms have their own User-Agents. They are legitimate crawlers, but attackers can copy those strings. Verifying a crawler means checking its reverse-DNS and IP range, not the header.

HTTP headers that hint at automation

Beyond the User-Agent, four header groups do most of the work.

• Accept-Language. Real browsers send a list built from system languages, often with quality weights, like en-US,en;q=0.9,fr;q=0.8. Bots frequently omit it entirely or send a single language with no weights.
• Sec-CH-UA and client hints. Chrome and Edge send structured client hint headers that list brand, version, and platform. Automation tools usually omit them or send values that do not match the User-Agent.
• Sec-Fetch-* metadata. Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-Dest, and Sec-Fetch-User tell a server how a request was initiated. Browsers send these consistently; many bots omit them or send wrong values — for example, claiming same-origin for a request that must have been cross-site.
• Accept-Encoding and Connection. Real browsers support gzip, deflate, and brotli. Some automation stacks send only gzip or nothing. Connection: keep-alive appears everywhere, so it is the least useful field.

A fourth group deserves attention: how the User-Agent combines with these headers. A HeadlessChrome string with consistent Sec-CH-UA and Accept-Language is more likely the operator's deliberate attempt. A HeadlessChrome string with missing client hints is the default automation profile.

Decision criteria: which header signals to trust

Weight each header with three questions before you act.

1. Does a legitimate user ever produce this pattern? Privacy browsers, fingerprinting blockers, corporate proxies, and travel networks strip or rewrite headers. If a signal appears in genuine traffic, treat it as suspicious rather than certain.
2. How hard is the signal to fake? Any header can be forged by a determined operator. Client hints and Sec-Fetch metadata are slightly harder to forge consistently because a server can cross-check them against the User-Agent.
3. Does the signal correlate with something else? The real value comes from correlation. A HeadlessChrome UA plus missing mouse movement plus a form submitted in under a second is a compelling story. Any single line item is weak.

In practice, the signals rank like this:

Signal	Trust level	Reason
HeadlessChrome substring in UA	High when confirmed	Automation tools use it by default; operators must actively strip it.
Header contradiction (UA vs Sec-Fetch vs client hints)	High	Hard to align every header consistently.
Missing Accept-Language or client hints	Medium	Privacy tools, old browsers, and enterprise proxies also omit them.
Empty or malformed User-Agent	Medium	Legitimate health checks and monitoring tools do this too.
Named crawler UA out of context	Low alone	Copying a Googlebot string is trivial; needs IP verification.

A practical detection rule for header analysis

Follow this sequence when you review your server logs.

1. Collect the full header set. Log User-Agent, Accept-Language, Sec-Fetch-*, and Sec-CH-UA for every request, not just the IP.
2. Flag exact automation substrings. Look for HeadlessChrome, PhantomJS, python-requests, curl, and similar names.
3. Check for contradictions. A Chrome UA with no Sec-CH-UA, or a viewport size that does not match the request's user agent family, is a useful signal.
4. Never block on a header alone. Use headers to focus your attention, then verify with behavior: did the visitor move the mouse, scroll, pause, and advance through fields like a person?
5. Rate-limit instead of block when in doubt. A soft challenge (slowing response, adding a proof-of-work step) slows cheap automation without harming genuine users.

The common mistake: treating one header as proof

Because a header is easy to log, teams tend to trust it too far. The clearest failure is blocking or refunding based on a user agent alone. Bot detection documentation makes the point directly: a single anomaly is not a bot verdict. Privacy tools, travel networks, corporate proxies, and unusual devices can produce unexpected behavior for real people. If you block every session that sends an odd header, you lose those visitors to competitors who bother to check.

Modern bot operators exploit exactly this over-reliance. Fraud networks route traffic through residential proxies, which present legitimate consumer IP addresses and defeat location filters. They also use AI generators to simulate human mouse curvature, click intervals, and scrolling, leaving header-based checks looking at a normal surface. The header may be clean while the behavior behind it is machine-made.

The correction is to treat header signals as one of several evidence types and demand corroboration before you take action.

Key facts about bot detection signals

The table below pulls the relevant facts from BotRefund's detection documentation and related guides.

Fact	Detail	Source
Automated browser tools	Puppeteer, Selenium, and Playwright load sites and fill forms automatically, producing identifiable header and behavior patterns.	Affiliate lead fraud guide
Residential proxies	Bot operators spread traffic across consumer-owned IPs to bypass geolocation firewalls, so IP plus header checks lose power.	Affiliate lead fraud guide
AI behavior mimicry	Fraud networks use AI to simulate human mouse curves, click intervals, and page scrolling, defeating simple pattern rules.	Ad fraud trends guide
Single anomaly is evidence, not verdict	Privacy tools, travel, corporate networks, and unusual devices can produce unexpected behavior; one mismatch is not a conclusion.	Console Debug Evaluator
Corroboration model	Detection cross-checks browser, network, device, and behavior evidence before classifying a visit as bot or human.	Console Debug Evaluator

Limitations: when header checks fail

Headers are the weakest layer of bot detection, and they fail in predictable ways.

• Full spoofing. A motivated operator can copy every header from a real browser. Nothing in the header layer proves the client actually executed JavaScript, painted pixels, or accepted cookies.
• False positives from privacy tools. Users with fingerprinting blockers, strict privacy settings, or enterprise proxies often send simplified headers that resemble bots.
• Cache and CDN rewriting. Content delivery networks may modify headers before they reach your origin, hiding automation signals or adding their own.
• AI-driven botnets. As noted in the ad fraud trends report, modern botnets use residential proxies and AI-generated telemetry, so the HTTP surface can look entirely human.

If your traffic is low-volume or low-stakes, header checks are a reasonable first filter. If you run paid ads, lead forms, or affiliate payouts, you need a second layer: behavioral evidence from the client side.

Terminology you may see

• User-Agent (UA) — the header that describes the client, including browser, version, and OS.
• Client hints (Sec-CH-UA) — a newer group of headers that announce browser brand, version, platform, and model.
• Sec-Fetch-* — headers that describe how a request began: navigation, same-origin resource, or cross-site.
• Headless browser — a real browser engine without a visible window, commonly used for automation and scraping.
• Residential proxy — a network of real consumer IPs used to make bot traffic appear local and legitimate.
• Behavioral telemetry — data about mouse movement, scrolling, clicks, and timing that distinguishes human from scripted sessions.

FAQ

Can bots fake a real Googlebot user agent?

Yes. Copying the string is trivial. Verify Googlebot by reversing the IP against Google's published ranges, not by trusting the header.

Why do some bots leave the User-Agent empty?

Simple scripts and libraries omit it. Some privacy tools also strip it, so an empty header is a flag to investigate, not a conclusion.

Is HeadlessChrome always a bot?

Not always. Teams use headless browsers for testing, PDF generation, and monitoring. The correct response is close attention, not blocking.

What is the most reliable server-side header check?

A combination mismatch: a User-Agent claiming Chrome with client hints and Sec-Fetch metadata that a real Chrome session would produce. One field can be spoofed; a full contradictory set is harder to fake.

Do privacy tools trigger bot detection?

They can. Privacy browsers, corporate networks, and unusual devices produce unexpected header behavior. Good detection systems treat a single anomaly as evidence, not a verdict.

How do modern bots pass header checks?

By borrowing from real browsers, routing through residential proxies, and generating human-like telemetry. That is why behavioral correlation matters more than any header.

Further reading and comparison sources

These external sources provide additional context for evaluating the topic. Their inclusion is not an endorsement.

• User agent strings to HTTP signatures - methods for AI agent ...
• What Is a User Agent? A 2026 Guide for Developers, Scrapers, and ...
• 2025: List of User-Agent strings | DeviceAtlas

Learn more

Visit the website for more information.