How to Calculate Precision and Recall for Your Bot Detection System

Quick answer: the two formulas you need

Precision = True Positives / (True Positives + False Positives). Recall = True Positives / (True Positives + False Negatives). In bot detection terms: precision tells you what share of blocked traffic was truly automated; recall tells you what share of all automated traffic you blocked. Both require a confusion matrix with four counts: true positives (bots correctly flagged), false positives (humans incorrectly flagged), false negatives (bots that slipped through), and true negatives (humans correctly passed).

Step 1: Collect a labeled sample of traffic

You cannot compute precision or recall without ground truth. Start by pulling a representative sample of visits from your logs — at least a few thousand sessions across different times, campaigns, and device types. Label each visit as bot or human. You can label manually (reviewing session recordings, mouse paths, challenge results) or use a trusted third-party verification set. BotRefund, for example, uses 106 independent checks including Empty Font Canvas and Suspicious Ports to build a reliable picture of whether a visit is human or automated, then feeds those signals into an AI model that weighs the complete pattern instead of trusting a raw rule.

Step 2: Run your detection system on the sample

Feed the same sample through your bot detection pipeline. Record the system's decision for each visit: flag as bot or allow as human. Do not adjust thresholds yet; you want the raw output at your current operating point. If your system outputs a score, pick the threshold you currently use in production.

Step 3: Build the confusion matrix

Create a 2x2 table. Rows = actual class (bot, human). Columns = predicted class (bot, human). Count the four cells:
True Positive (TP): actual bot, predicted bot.
False Positive (FP): actual human, predicted bot.
False Negative (FN): actual bot, predicted human.
True Negative (TN): actual human, predicted human.

Step 4: Calculate precision and recall

Precision = TP / (TP + FP). Recall = TP / (TP + FN). Write the numbers down. Example: if you flagged 1,200 visits as bots and 1,050 were truly bots, precision = 1,050 / 1,200 = 87.5%. If the labeled set contained 1,500 real bots and you caught 1,050, recall = 1,050 / 1,500 = 70%.

Step 5: Compute confidence intervals

Point estimates are noisy. Use Wilson score intervals or bootstrap resampling to get 95% confidence bounds for each metric. This tells you whether a 2% precision drop after a threshold change is real or sampling variance.

Step 6: Sweep thresholds to see the trade-off

If your detector outputs a continuous score, repeat steps 2–4 at multiple thresholds. Plot precision vs. recall (PR curve) or precision and recall vs. threshold. Choose an operating point that matches your cost structure: blocking a real user (false positive) usually costs more than letting a bot through (false negative) for ad fraud, but the reverse may be true for account takeover.

Step 7: Validate on a hold-out set

Never tune thresholds on the same data you used to measure. Split your labeled data before step 2. Use the first split for threshold selection, the second for final precision/recall reporting. If you have multiple traffic sources (paid search, organic, direct), validate per source — bot mixes differ.

Common mistake: using accuracy instead of precision/recall

Accuracy = (TP + TN) / (TP + FP + FN + TN). When bots are rare (e.g., 5% of traffic), a dummy model that labels everything human scores 95% accuracy but 0% recall. Always report precision and recall for the minority class (bots).

Common mistake: labeling bias

If your labeled set over-represents obvious bots (headless Chrome, data-center IPs), recall will look inflated. Include stealthy bots — residential proxies, human-in-the-loop click farms, session replay scripts — in proportion to their real prevalence.

Common mistake: ignoring false-positive cost

A 99% precision claim means 1 in 100 blocked users is human. At 1M visits/month with 10% bot rate, that's ~1,000 real users blocked. If each blocked user is worth $50 LTV, that's $50k/month in false-positive loss. Quantify this before you celebrate high precision.

Verification step: run a live A/A test

Deploy the new threshold to 1% of traffic behind a feature flag. Compare conversion rate, bounce rate, and support tickets against the control for two weeks. If human metrics dip, your false-positive rate is higher than the labeled sample suggested. Roll back or adjust.

Key facts from BotRefund's detection approach

Limitations of precision/recall for bot detection

Precision and recall assume a static ground truth. In reality, bot operators adapt. A model with 90% recall today may drop to 60% next month without retraining. The metrics also ignore latency: a detector that takes 500ms per request may hurt page speed more than the bots it catches. BotRefund addresses this by sending each signal into a prediction AI that evaluates the complete picture across browser, network, device, and behavior evidence, identifying a visit as bot or human with 99% accuracy — but that accuracy claim depends on continuous model updates and corroboration across 106 checks, not a single rule.

Another limitation: precision/recall don't capture financial impact. A bot that clicks ads costs you money; a bot that scrapes content may not. Weight your confusion matrix by estimated revenue loss per bot type if you need a business-aligned metric.

Terminology cheat sheet

• True Positive (TP): Bot correctly identified as bot.
• False Positive (FP): Human incorrectly identified as bot (false alarm).
• False Negative (FN): Bot incorrectly identified as human (missed detection).
• True Negative (TN): Human correctly identified as human.
• Precision: TP / (TP + FP) — of those you called bots, how many were bots.
• Recall: TP / (TP + FN) — of all real bots, how many you caught.
• F1 Score: Harmonic mean of precision and recall = 2 * (P * R) / (P + R).
• PR Curve: Plot of precision vs. recall across all thresholds.
• Confusion Matrix: 2x2 table of actual vs. predicted classes.

FAQ

How much labeled data do I need?

At minimum, 500–1,000 labeled visits per class (bot/human) for a rough estimate. For confidence intervals under ±3%, aim for 2,000+ per class. If bots are rare, oversample them in your labeling set and weight the metrics accordingly.

Can I use my ad platform's invalid click reports as ground truth?

Only as a weak signal. Google and Meta's invalid click filters are conservative — they miss sophisticated bots. Treat platform reports as a lower bound on recall, not ground truth.

What if I don't have any labeled data?

Start with a honeypot: add invisible links or form fields that humans never see. Visits that interact are bots with near-certainty. Use those as positive labels. For negatives, sample high-engagement sessions (long dwell, multiple pages, conversions) and spot-check a few dozen manually.

How often should I recompute precision and recall?

Monthly at minimum. Weekly if you're actively tuning thresholds or seeing bot mix shifts (new proxy providers, seasonal click farms). Automate the labeling pipeline so it's not a manual fire drill.

Should I optimize for precision or recall?

Depends on your cost asymmetry. For ad fraud: false positives (blocking real users) waste ad spend and hurt conversion rates — optimize for precision first, then raise recall until false-positive cost equals bot-cost savings. For account takeover or scraping: missed bots are far costlier — optimize for recall.

What's a good precision/recall target?

There's no universal number. BotRefund's system achieves 99% accuracy through corroboration across 106 independent checks, but that's an aggregate across browser, network, device, and behavior signals. A single-signal detector (e.g., only user-agent checks) might hit 60% recall at 80% precision. Measure your current baseline, then improve incrementally.

How do I explain these metrics to stakeholders?

Use concrete scenarios: "At our current threshold, for every 100 visits we block as bots, 87 are actually bots (precision). Of all bots hitting our site, we catch 70% (recall). Moving the threshold to catch 85% of bots would drop precision to 72%, meaning we'd block 28 real users per 100 flagged visits."