How to Validate a Device Group Before You Block It

Use a chi-square test to compare the device group’s click/error ratio with your broad site average. If the p-value is below 0.05 and the group has at least 30 events, the pattern is unlikely to be random, so the block is worth serious review. This article walks through that validation process step by step.

A device group is a traffic segment such as one iOS version, one Android model, or one browser on a specific operating system. Ad platforms may flag these groups automatically when behavior looks automated. The problem is that small samples create false flags. A handful of bad clicks can make a normal group look fraudulent. You need enough evidence before you block.

What counts as evidence in a device group

Evidence means repeatable patterns, not one bad lead. As BotRefund’s Meta Ads invalid traffic guide puts it: “Bot traffic and form spam tend to leave repeatable technical and behavioral patterns: unusually fast form completion, identical field structures, sudden placement-level spikes, or conversion events with no meaningful page engagement.”

Apply that idea to a device group. Three errors out of ten clicks is a signal to investigate, not a reason to block. Thirty errors out of three hundred clicks, with the same pattern repeating over several days, is a much stronger case. The evidence needs two parts: a statistical difference from normal traffic and a behavioral reason to believe the difference is automated.

The chi-square test in plain terms

A chi-square test compares what you observed with what you would expect if the device group behaved exactly like the rest of your traffic. If the difference is large enough, the test returns a p-value below 0.05. That means the difference is unlikely to be random.

Here is the process in plain numbers:

1. Pick one outcome: clicks that turn into conversions, clicks that turn into errors, or clicks per impression.
2. Find the broad site average for that outcome. Use the rest of your traffic as the baseline, not the whole site including the device group.
3. Calculate the expected count for the device group. Multiply the site average by the device group’s clicks.
4. Compare observed and expected counts with the chi-square formula: sum of (observed - expected)² / expected for each category.
5. Check the p-value. If it is below 0.05, the group is statistically different.

Example (illustrative): your site average error rate is 5%. A device group has 200 clicks and 18 errors. Expected errors are 10. Observed errors are 18. The chi-square contribution for errors is (18-10)² / 10 = 6.4. The contribution for non-errors is (182-190)² / 190 = 0.34. Total chi-square is 6.74. With one degree of freedom, the p-value is below 0.05. The device group is statistically different. All expected counts are above 5, so the chi-square approximation is reliable here.

Minimum sample size

Use at least 30 events in the device group. Some analysts prefer 50. The exact number matters less than avoiding decisions on tiny counts. Chi-square is also less reliable when any expected count is below 5. If your expected count is below 5, wait for more data or use Fisher’s exact test, which works better with very small samples.

Step-by-step: validate a device group before blocking

Before you start, export device group data for the last 14 to 30 days. Choose one outcome metric and calculate the site average. Then follow these steps:

1. Pull the device group’s clicks and outcome count for the same period.
2. Calculate the expected outcome count using the site average.
3. Run the chi-square test using a spreadsheet, calculator, or statistical tool.
4. Check the p-value. If it is 0.05 or higher, the difference could be random. Do not block.
5. Check the sample size. If the group has fewer than 30 events, wait for more data.
6. Review behavior patterns in the flagged group: bursts at unusual hours, no scrolling, no field corrections, identical field structures, or near-instant bounces.
7. Block the group only if the statistical test and the behavioral review both point the same way.
8. Document the evidence and the date. This helps if you later ask the ad platform for a refund.

Verify the next step

After you block a device group, watch the next 7 to 14 days. Did the site-wide error rate improve? Did conversions from other groups stay stable? Did the blocked traffic reappear under another device label? If nothing changes, remove the block. A good block changes the metric that made you suspicious.

Common mistakes that produce false blocks

• Blocking on fewer than 30 events. A tiny sample can look extreme by chance.
• Using the wrong baseline. Compare the device group with the rest of your traffic, not with a blend that includes the group itself.
• Treating statistical significance as proof of fraud. It only proves the group is different.
• Using only click rate. Bots can click once and leave. Conversion or error rates are usually stronger signals.
• Ignoring placement. Device groups that come mostly from the Meta Audience Network can show high click-through rates and near-instant bounces because of the placement, not the device.
• Blocking before checking session behavior. A landing page change or a bad creative can make a device group look broken without any bot involvement.

What to check after you block

Blocking is not the final step. It is an experiment with a clear prediction: the problem metric should improve. If it does not, the block was probably wrong.

• Check the device-level breakdown for the blocked group. Did the suspicious clicks stop?
• Check overall conversions. A sudden drop without an improvement in error rate means you may have blocked real users.
• Check for reappearing traffic. Bots often rotate user agents or device strings, so the same behavior may show up under a new device label.
• Check the refund path. If you have session-level evidence, keep it. It is the basis for contesting invalid clicks with Google or Meta.

Limitations and when this test does not apply

A chi-square test is a decision aid, not a verdict. It tells you that a device group is different from the baseline. It does not tell you why.

• Bot traffic often arrives in bursts. The chi-square test assumes independent events, so a burst can inflate significance. If the traffic is clustered in one hour, treat the result with caution.
• Device group definitions change. An OS version becomes obsolete, and a model stops being sold. Revalidate blocks on a regular schedule.
• This test is for ad traffic and invalid-traffic decisions. It is not the right standard for endpoint security, conditional access, or network access control. Those systems have their own evidence requirements.
• If the expected count is below 5, the chi-square approximation can be misleading. Use an exact test or collect more data.

Key facts at a glance

Terminology

• Device group: a traffic segment defined by device type, operating system version, browser, or model.
• Invalid traffic: clicks or impressions that are not the result of genuine user interest.
• Chi-square test: a statistical test that compares observed counts with expected counts.
• p-value: the probability that the difference happened by chance. A p-value below 0.05 means the difference is unlikely to be random.
• Pixel poisoning: bot traffic triggering conversion events and making the ad platform optimize toward bots rather than real buyers.

FAQ

What minimum data should a device group have before I consider blocking it?

Use at least 30 events in the device group, and avoid relying on the chi-square result if any expected count is below 5. More data is better, especially for high-traffic groups.

Can I use click-through rate instead of error or conversion rate?

You can, but clicks alone are a weaker signal. A bot can click once and leave. Outcomes such as form submissions, errors, or conversions give you more evidence about whether the traffic can actually do what a human would do.

What if the p-value is below 0.05 but the sample is tiny?

Do not block. A tiny sample can produce a significant result by chance. The minimum count exists to prevent that bias. Wait for more data.

Does a significant chi-square test prove the device group is bots?

No. It proves the group is statistically different from the baseline. You still need behavioral evidence: timing bursts, no scrolling, identical field structures, or other repeatable patterns.

How long should I test before blocking?

A 14 to 30 day window is a reasonable starting point. Shorter windows are more likely to be distorted by a single spike or a campaign change.

What should I do if the block does not change performance?

Remove the block. Then look for another explanation, such as a placement issue, a creative problem, or a landing page bug. The block was meant to fix a measurable problem.