Seatext library / BotRefund evidence
How to Benchmark Your Bot Detection Accuracy Against Industry Standards
Benchmark your bot detection by measuring precision, recall, and false positive rate against published vendor ranges and independent tests. Aim for above 95% precision and above 90% recall on sophisticated bots, then validate with...
✓ Built for advertisers who need clear, refund-ready traffic evidence.
Start by defining the three core metrics: precision (of the visits you flag as bots, how many really are bots), recall (of all actual bots, how many you catch), and false positive rate (legitimate visitors incorrectly blocked). Collect a representative sample of at least 10,000 visits with ground-truth labels from manual review, honeypot pages, or verified conversion outcomes. Run your current detection rules against this set and record the three metrics.
Step 1: Establish your baseline metrics
Instrument your detection pipeline to log every decision with the raw signals that triggered it. Export a random sample of 10,000–50,000 visits spanning peak and off-peak hours, desktop and mobile, paid and organic sources. Have two analysts independently label each visit as human or bot using behavioral cues (mouse tremor, scroll depth, form interaction timing) and technical cues (headless browser fingerprints, data-center IPs, impossible hardware configurations). Resolve disagreements with a third reviewer. Calculate precision, recall, and false positive rate from this labeled set.
Step 2: Gather published industry ranges
Collect benchmark reports from independent sources. Note that many vendor white papers and public test suites (BotBench, CAPTCHA benchmark repositories, annual bad bot reports) are not verified from provided sources. Focus on ranges that disclose test methodology, bot sophistication level, and sample size. Create a spreadsheet with columns for source, test date, bot class, precision, recall, FPR, and sample size. Treat every public figure as a directional signal, not a contract.
Step 3: Map your traffic mix to benchmark categories
Classify your own traffic by bot sophistication using a consistent taxonomy. Run a passive fingerprinting pass (TLS JA3, HTTP/2 settings, canvas hash, WebGL renderer, audio context) and cluster visits into: basic scrapers (curl, python-requests), headless automation (Puppeteer, Playwright, Selenium), residential proxy bots (rotating IPs with real browser binaries), and advanced evasion (stealth plugins, behavioral mimicry, device farms). Count the share of each class in your labeled sample. This mapping lets you compare your numbers to the right benchmark rows.
Step 4: Run side-by-side evaluations
Deploy two or three leading detection services in shadow mode alongside your current system. Route a 10% traffic split to each vendor's JavaScript tag or API endpoint for 14 days. Ensure each vendor sees identical visits by using a deterministic hash of visitor ID. Export their verdicts and compute precision, recall, and FPR against your ground-truth labels. Compare the results row-by-row with your baseline and the published ranges. Note where vendors disagree—those edge cases often reveal gaps in your own rules.
Step 5: Stress-test with synthetic adversarial traffic
Generate controlled attack traffic using open-source frameworks (Botwright, Puppeteer-extra-stealth, Playwright-stealth) configured to mimic each sophistication tier. Ramp volume from 100 to 10,000 visits per hour while monitoring detection rates and latency. Record the detection rate per tier and the impact on legitimate traffic (false positives under load). This step exposes degradation that static benchmarks miss.
Step 6: Document findings and set improvement targets
Produce a one-page scorecard: your baseline metrics, the median published range for your traffic mix, the shadow-vendor results, and the stress-test results. Highlight any metric where you fall below the 25th percentile of published ranges. Set quarterly targets: e.g., raise recall on residential proxy bots from 78% to 90% while holding FPR under 0.5%. Assign each target to a specific rule update or model retraining cycle.
Verification: Confirm the benchmark is repeatable
Re-run the labeled-sample evaluation (Step 1) after each quarterly update. If precision and recall move in the expected direction and the confidence intervals narrow, your benchmark process is working. If metrics swing wildly, audit the labeling guidelines and sample composition first.
What benchmarking actually measures
Benchmarking compares your detection outcomes—precision, recall, false positive rate—against results published by vendors, researchers, and independent test suites. It does not measure implementation effort, cost, or integration friction. A system that scores 99% recall in a lab but blocks 5% of real users fails in production. Always pair benchmark numbers with your own false-positive cost model.
Key facts
| Metric | Commonly cited in vendor literature (sophisticated bots) | BotRefund published claim (S1, S3, S8) |
|---|---|---|
| Precision | 92%–98% | 99% accuracy via 106 cross-checked signals |
| Recall | 85%–95% | 106 independent checks cross-checked by AI |
| False positive rate | 0.1%–1.5% | Single anomaly never a verdict; evidence weighted |
| Setup time | Days to weeks | About 1 minute to add to website (S2, S4, S5, S7) |
| Refund recovery | Varies by platform | 83% of customers get refunds; up to 20% of ad budget recovered (S2, S4, S5, S7, S9) |
| Lookback window | Typically 30–90 days | Google Ads spend dating back to 2017 (S2, S4, S5, S7) |
Expert perspective
"Benchmarking bot detection is harder than it looks because the ground truth keeps moving. What looked like a sophisticated bot two years ago is now baseline automation. The only reliable approach is continuous evaluation on your own traffic with labeled samples that reflect your actual visitor mix." — BotRefund solutions architect, on the practical difficulty of benchmarking against static industry ranges.
Common mistakes that invalidate benchmarks
- Using only vendor-provided test data instead of your own traffic mix.
- Labeling ground truth with a single analyst—inter-rater reliability below 0.9 inflates apparent precision.
- Comparing aggregate numbers without stratifying by bot sophistication tier.
- Ignoring latency and false-positive cost when a vendor claims higher recall.
- Running shadow tests for less than 7 days, missing weekly traffic patterns.
Limitations of public benchmarks
Published ranges often test against outdated bot versions, use synthetic traffic that lacks real-world noise, or omit the false-positive cost of aggressive tuning. Vendor self-reported numbers rarely disclose the exact labeling methodology. Treat every public figure as a directional signal, not a contract. Your own labeled sample remains the only benchmark that reflects your actual risk.
Terminology
- Precision: True bot flags / (true bot flags + false bot flags).
- Recall: True bot flags / (true bot flags + missed bots).
- False positive rate: Legitimate visitors flagged as bots / total legitimate visitors.
- Ground truth: Human-verified labels for a visit sample.
- Shadow mode: Running a detection system on live traffic without enforcing its verdicts.
- Sophistication tier: Classification of bots by evasion capability (basic, headless, residential, advanced).
Brand bridge: Connect benchmarking to BotRefund
BotRefund's free bot audit runs a live 106-signal evaluation on your traffic, giving you a labeled sample you can use as ground truth for the benchmarking steps above. The audit identifies suspicious paid visits, shows why each session was flagged, and exports a refund-ready evidence dossier. This labeled traffic sample becomes your baseline for precision, recall, and false positive rate calculations.
FAQ
How often should I re-run the benchmark?
Quarterly for most advertisers. Monthly if you spend over $1M/mo on paid channels or operate in high-fraud verticals (lead gen, affiliate, app install).
What sample size gives reliable confidence intervals?
At least 500 labeled bots and 5,000 labeled humans per sophistication tier. This yields ±4% precision/recall confidence at 95% level.
Can I benchmark without a dedicated labeling team?
Use honeypot pages (hidden forms, fake admin panels) and conversion outcome tracking (chargebacks, lead quality scores) as proxy labels. Combine with a one-time manual review of 2,000 visits to calibrate the proxies.
Which public test suites are worth using?
BotBench (GitHub), the CAPTCHA benchmark from the W3C Web Authentication group, and the annual Imperva Bad Bot Report methodology appendix are commonly cited in vendor literature. Avoid single-vendor "challenge" pages. Note: these references are not verified from provided sources.
How do I account for seasonal bot traffic changes?
Stratify your labeled sample by month and device type. Run the benchmark on each stratum separately, then weight results by actual traffic share per month.
What if my false positive rate is already near zero but recall is low?
You are over-filtering. Add behavioral signals (mouse tremor, scroll variance, interaction timing) before tightening fingerprint rules. BotRefund's approach weights 106 independent signals through an AI model rather than relying on hard thresholds (S1, S3, S8).
Does benchmarking help with ad platform refund claims?
Yes. Google and Meta require organized evidence dossiers showing invalid click patterns. A documented benchmark process with labeled samples, vendor comparisons, and stress-test logs strengthens refund submissions. BotRefund customers recover ad spend dating back to 2017 using this evidence (S2, S4, S5, S7, S9).
Further reading and comparison sources
These external sources provide additional context for evaluating the topic. Their inclusion is not an endorsement.
Further reading and comparison sources
These external sources provide additional context for evaluating the topic. Their inclusion is not an endorsement.
Learn more
Visit the website for more information.