How to spot bad survey data before you pay for it

Buy 200 responses from any panel and some of them will be worthless. That is not a scandal, it is arithmetic: people are being paid a small amount to answer questions they did not choose, and a minority will optimise for speed.

The question is not whether bad responses arrive. It is whether you notice, and whether you pay for them.

The three ways a response goes bad

Speeding. Somebody completing a twelve-screen study in twenty seconds did not read it. This is the easiest to catch and the most common.

The naive check, a fixed time threshold, is worse than it looks. Reading speed varies enormously and a fast, genuine respondent gets punished. A better check is per-item: did they act on this card faster than it could physically be read, and how often?

Straight-lining and pattern answering. Picking the first option every time, or alternating, or in a card sort dropping every card into the pile in exactly the order it was presented without ever going back. That last pattern, sometimes called grab and dump, is invisible in the final data: the sort looks complete and the piles look plausible. It only shows up in the behaviour, in the fact that nothing was ever revised.

Screener fraud. Claiming to be a purchasing manager at a firm of 500 to qualify for an incentive. This one is largely the panel's problem to catch, and a good panel does catch most of it, along with duplicates and bots.

What actually works

Measure behaviour, not just answers. The strongest signals are in how a response was produced. Time per item rather than time overall. Whether somebody revisited a decision. Whether their grouping is suspiciously aligned with the order things were shown in. A response with plausible answers and impossible behaviour is the one you most want to find.

Ask a question you already know the answer to, sparingly. An attention check works, but every one you add lengthens the study and irritates the honest majority. One is usually enough, and phrasing it as a genuine question rather than "select strongly agree to show you are paying attention" avoids teaching people to look for the trap.

Never let the tool decide silently. A quality flag should be a signal, not a verdict handed down without appeal. You want to see the flagged responses, understand why, and view the result with and without them. If excluding twelve rushed responses changes your ranking, that is worth knowing before you present it. If it changes nothing, your finding is robust and you can say so.

The billing question, which is the one that matters

Detecting bad data is only half of it. The other half is who pays.

In most setups you buy N completes, the panel delivers N completes, and the ones you throw away are your problem. You end up over-ordering to compensate, which is a tax on the honest respondents.

The arrangement worth having is two filters, neither of which costs you:

Before they start, the panel turns away fraud, duplicates and anyone failing your screener. They never reach your study and never reach your bill.

After they finish, your own checks read how the response was actually made. A rejected one goes back to the panel as a quality reject, is replaced, and is not charged.

That is how SortedResearch works, and it is why the pricing page says you are only charged for responses that pass. It also aligns the incentives properly: a tool that bills you per response collected has no reason to be strict.

How much is normal

Reject rates vary by study length, incidence and panel, but a well-run study on a decent panel typically sets aside a small single-digit percentage. If you are seeing 20 percent, something is wrong upstream: the study is too long, the screener is too tempting to game, or the incentive is attracting the wrong people.

A rate of zero is not good news either. It usually means nothing is being checked.

Before you buy anything

Pilot with five. Look at the actual responses, not the summary. You will learn more about your questionnaire in ten minutes than from any amount of review.
Time it yourself. If it takes you six minutes knowing exactly what you meant, it will take a stranger longer, and length drives every quality problem there is.
Check your screener is not a treasure map. If the qualifying answer is obvious from the question, you have taught people how to qualify.
Know the replacement policy before you pay, not after you find the speeders.