Test Your Bot Detection System with Real Traffic

You test with real traffic by logging detection decisions for a sample of visitors, manually verifying a subset of flagged and unflagged sessions, and computing accuracy metrics from the verified labels.

Why Real‑Traffic Testing Matters

Real traffic reflects the actual behavior of your users and attackers. Synthetic tests can miss edge cases like privacy tools, corporate networks, or travel‑related device anomalies that still produce legitimate sessions. A detection system that looks perfect on lab data may block real customers when privacy extensions strip fonts or corporate proxies rotate IPs. BotRefund notes that privacy tools, travel, and corporate networks can produce unexpected behavior for genuine people, so each signal is kept as evidence—not a verdict (S1). Testing on live traffic surfaces these ambiguities before they cost revenue.

Key Components of a Testing Pipeline

A practical pipeline needs three parts: data collection, human verification, and metric calculation. Each part should be repeatable so you can track improvements over time. Data collection captures every detection decision with context. Human verification assigns ground‑truth labels to a representative sample. Metric calculation turns those labels into precision, recall, and F1 scores you can compare across releases. BotRefund uses 106 independent checks across browser, network, device, and behavior layers, then cross‑checks them before an AI model weighs the complete pattern (S1). Your pipeline should mirror that diversity: collect signals from every layer, not just one.

How to Collect and Label Traffic Data

Follow these steps to build a labeled dataset from live traffic.

1. Enable logging. Record each detection decision (bot vs. human) along with timestamps, IP, user‑agent, and any signal scores. Store the raw signal vector so you can re‑score later.
2. Define a sample window. Choose a recent period (e.g., last 7‑14 days) and export the logged decisions for that range. Exclude known test IPs and internal traffic.
3. Build a stratified sample. Pull 5‑15% of total sessions, but oversample flagged sessions so you get enough true bots for recall estimation. Example: if you have 100,000 sessions and 2,000 flagged, take all 2,000 flagged plus a random 8,000 unflagged for a 10,000‑session sample (10%).
4. Tag sessions for review. Mark the sampled sessions as "verified" in your labeling tool. Assign each to a reviewer with a checklist (see next section).

Worked example — sampling plan for a mid‑size e‑commerce site

• Total sessions in 14‑day window: 250,000
• Flagged by detector: 7,500 (3%)
• Target sample size: 12,500 (5%)
• Stratification: all 7,500 flagged + 5,000 random unflagged
• Reviewers: 2 analysts, 6,250 sessions each
• Estimated review time: 45 seconds per session → ~78 hours total

Manual Verification Best Practices

Review both flagged and unflagged sessions. Look for clear signs of automation (straight mouse paths, sub‑1ms clicks) and also check privacy tools or unusual devices that could be mistaken for bots. Document any ambiguity and treat it as "unknown" rather than forcing a label. BotRefund's Empty Font Canvas check, for instance, flags a mismatch between claimed device and graphics output, but notes that virtual machines and spoofed profiles can create similar mismatches for legitimate users (S1).

Verification checklist per session

Check	What to look for	Decision
Mouse path	Natural curves, hesitation, micro‑jitter vs. straight lines or grid‑snapping	Human / Bot / Unknown
Click timing	Intervals > 100ms, variable vs. <1ms or perfectly periodic	Human / Bot / Unknown
Scroll behavior	Variable speed, pauses to read vs. instant jump or no scroll	Human / Bot / Unknown
Form interaction	Keystroke dynamics, corrections, paste events vs. instant fill	Human / Bot / Unknown
Device signals	Consistent hardware, GPU, font list vs. empty canvas or mismatched specs (S1)	Human / Bot / Unknown
Network context	Residential ISP, consistent geo vs. data‑center IP, VPN, proxy ports (S3)	Human / Bot / Unknown
Session flow	Multi‑page journey, referrer logic vs. direct landing + immediate exit	Human / Bot / Unknown
Privacy tools	Known extensions (Privacy Badger, uBlock) that may strip signals	Note only — do not label bot

If two reviewers disagree, a third breaks the tie. Sessions marked "unknown" are excluded from metric denominators but logged for later analysis.

Calculating and Interpreting Detection Metrics

Use standard classification metrics: precision (fraction of flagged sessions that are truly bots), recall (fraction of actual bots you caught), and F1 score (balance of the two). Track false positives and false negatives separately to understand where your model may be over‑ or under‑confident. BotRefund claims 99% accuracy from corroboration across 106 checks, not from any single signal (S1). Your metrics should reflect the same principle: report per‑signal contribution if possible.

Metric‑tracking template (per evaluation cycle)

Metric	Formula	Current value	Target	Notes
Precision	TP / (TP + FP)	—	> 95%	Low precision = blocking real users
Recall	TP / (TP + FN)	—	> 90%	Low recall = bots slipping through
F1 Score	2 * P * R / (P + R)	—	> 0.92	Balance metric
False Positive Rate	FP / (FP + TN)	—	< 1%	Critical for revenue impact
False Negative Rate	FN / (FN + TP)	—	< 5%	Critical for ad‑spend protection
Unknown rate	Unknown / Sampled	—	< 10%	High unknown = checklist gaps
Sample size	Flagged + Unflagged reviewed	—	> 5% of traffic	Stratified as described
Cycle date	—	—	Quarterly	Or after model change

Export this table as CSV each cycle. Plot trends to catch regressions early.

Integrating Feedback into Your Detection System

Feed the verified labels back into your training pipeline. Adjust thresholds, add new signals, or retrain models based on which types of errors dominate. Re‑run the test after each iteration to measure progress. If false positives cluster on privacy‑tool users, add a "privacy‑tool present" feature and down‑weight anomaly signals for those sessions. If false negatives cluster on headless Chrome, add the JS engine mismatch check (S4) or monitor sync anomaly (S7) to your signal set. BotRefund's pipeline sends each signal into a prediction AI that weighs the complete pattern instead of trusting a raw rule (S1). Mimic that: let a model combine signals, don't hard‑code thresholds.

Limitations and Edge Cases

Real‑traffic testing cannot guarantee 100% coverage. High‑value traffic may be sparse, and some bot families mimic human behavior closely. Also, privacy tools can produce signals that look like bots; treat them as evidence, not verdicts. Common labeling ambiguities and how to resolve them:

• Privacy‑extension user flagged as bot. Empty font canvas or missing GPU data. Resolution: check extension list, mark unknown if extension explains anomaly.
• Corporate proxy rotates IPs mid‑session. Network signals disagree. Resolution: verify corporate ASN, mark human if device/behavior consistent.
• Traveler on hotel Wi‑Fi with carrier‑grade NAT. Geo‑IP mismatch, shared IP. Resolution: check device fingerprint stability, mark human if consistent.
• Sophisticated bot with human‑like mouse replay. Path and timing look real. Resolution: look for absence of micro‑tremor (S2), superhuman click speed (S2), or honeypot interaction (S2).
• Session with no clicks or scrolls. Could be bot or idle human. Resolution: check session duration, referrer, and whether page has interactive elements. Mark unknown if indeterminate.

Document every "unknown" decision with the reason. Review unknowns quarterly to see if new signals resolve them.

Glossary of Common Terms

False positive: A legitimate session incorrectly labeled as a bot.
False negative: A bot session incorrectly labeled as human.
Signal: An individual data point (e.g., hardware fingerprint, mouse jitter) used in detection.
Ground truth: The verified label assigned by human reviewers.
Stratified sample: A sample that preserves the proportion of flagged/unflagged sessions from the population.
Corroboration: Multiple independent signals agreeing on the same classification (S1).

FAQ

What is a false positive?

A false positive occurs when your system flags a real user as a bot, potentially blocking legitimate traffic.

How often should I run the test?

Run a full verification cycle quarterly or after any major model update to ensure performance stays stable.

Can I use synthetic traffic instead of real traffic?

Synthetic traffic is useful for stress‑testing, but real traffic is essential for validating accuracy against actual user behavior.

What metrics should I track?

Focus on precision, recall, F1, false‑positive rate, and false‑negative rate to get a complete picture.

How do I share results with my team?

Export a summary report (CSV or PDF) that includes the metrics, sample size, and any recommended rule changes.

What if my unknown rate exceeds 10%?

Revise your checklist. Add specific checks for the ambiguity patterns you see most often (privacy tools, corporate proxies, travel).

How many reviewers do I need?

At least two per session for tie‑breaking. For 10,000 sessions, two reviewers at 45 seconds each need ~125 hours total.

Can I automate the verification?

Partial automation helps (e.g., auto‑label known honeypot hits), but human judgment is still required for ambiguous cases.

Key Facts

Fact	Source
BotRefund uses 106 independent checks to decide human vs. automated.	S1
Accuracy comes from corroboration, not a single browser tell; BotRefund claims 99% accuracy.	S1
Free bot audit can be added to a website in about one minute, no credit card required.	S2
Case study: BotRefund identified 19% fake leads and saved sales pipeline quality.	S6

Further reading and comparison sources

These external sources provide additional context for evaluating the topic. Their inclusion is not an endorsement.

• IP Bot Detection Test: Detect & Block Bots | IPQS | IPQS
• Bot Detection 101: How to Detect (and Beat) Bot Traffic - Stytch
• Bot Detection | How to Detect Bots in 2023 | Radware

Further reading and comparison sources

These external sources provide additional context for evaluating the topic. Their inclusion is not an endorsement.

• IP Bot Detection Test: Detect & Block Bots | IPQS | IPQS
• Bot Detection 101: How to Detect (and Beat) Bot Traffic - Stytch
• Bot Detection | How to Detect Bots in 2023 | Radware

Learn more

Visit the website for more information.