Survey respondents and quality checks
Every survey respondent who finishes is checked for quality automatically, and given one of four verdicts: Passed, Held for review, Rejected or Not judged. Rejected responses are left out of your results by default, and a rejected panel response is not charged. The Respondents view on the Results tab lists everyone who completed, the doubtful ones first, so you can review them and remove any you don't want.
The Respondents view
Open Results, then Respondents. It lists the survey's live completed responses, one row each, sorted with Rejected first, then Held for review, Not judged and Passed. Each row shows:
- Response: the respondent's number.
- Quality: the verdict, with the names of any checks that fired underneath.
- From: Panel or Own link.
- Time: minutes from start to finish.
- Completed: when they finished.
If new responses land while you have the list open, a banner offers the fresh list rather than reshuffling the one you are reading. Test-link responses are not listed here.
When a panel order fills, a banner above the Results tabs says so, and its Review button opens this view.
The checks
Each check has a level: reject (enough on its own), hold (holds the respondent for review) or note (shown for you to judge). The names below are the ones you see on the Respondents view.
| Check | What it means | Level |
|---|---|---|
| Speeding (most screens) | Answered most screens faster than their words can be read | Reject |
| Speeding (some screens) | Answered a good share of screens too fast to read | Hold |
| Grid straightlining | Same answer down every row of most grids (needs at least two grids of five or more rows) | Hold |
| Failed an attention check | Didn't give the answer an attention check asked for | Reject |
| Picked a made-up option | Chose an answer you marked as made up | Reject |
| Ticked everything on a screening question | Ticked every option of a question that decides who takes part | Hold |
| Ticked nearly everything on a screening question | Ticked nearly all of those options | Hold |
| Contradicted themselves | Answered two linked questions in ways that can't both be true | Hold |
| Junk text | Keyboard mashing or repeated letters, or one word to every open question | Hold |
| Copied text | A text answer word for word the same as another respondent's | Hold |
| Nearly copied text | A text answer nearly the same as another respondent's | Note |
| Pasted text | Pasted a text answer in, or typed it faster than anyone can type | Hold |
| Answer pattern | Picked the same position every time on shuffled questions, or zig-zagged down grids | Hold |
| Unbelievable number | Gave a number outside the range you set as believable | Hold |
| Possible repeat respondent | The same network and browser already completed this survey | Note |
| Came back after being screened out | Was screened out, then returned on the same device and qualified | Hold |
Most checks run on their own. Four need you to set something up, in the Quality checks section of a question in the builder:
- This is an attention check: say which answer it asks for (on a grid, which row and which answer).
- Made-up options: add an answer that doesn't exist, a brand nobody has heard of, say, and tick it.
- Believable range: the lowest and highest believable number.
- Must agree with: link two questions that should give the same answer, the opposite answer (a reversed scale), or an age and birth year that add up.
What you set up for checking never reaches the respondent's browser, so it can't be read off the page.
The "screening question" checks only look at multiple choice questions that decide who gets in: one whose answers lead to a screen-out, or that a rule or quota reads to screen people out. A heavy buyer ticking every brand on an ordinary question is telling the truth, so that isn't counted.
How the verdict is decided
- Rejected: any reject-level check fires, or two or more hold-level checks fire together.
- Held for review: exactly one hold-level check fires.
- Passed: nothing at hold or reject level fired. The two notes never tip anyone on their own or alongside a hold; they are there for you to judge, because honest people sometimes word things alike or share a mobile network.
- Not judged: the person didn't complete (only completes are judged), or the checks couldn't run. Someone is never stopped from finishing because judging failed.
Test-link respondents are judged the same way, so you can try the checks yourself; the repeat-respondent checks only compare live respondents.
For 48 hours after someone finishes, they are checked again against everyone who has finished since, so two copies of the same text that arrived at the same moment are still caught. A second look can only make a verdict worse, and at most to Held for review.
Timings come from our server
Speed is measured from when our server showed each screen to when it received the answers, never from a count kept in the respondent's browser, which a script could fake. A screen's time is judged against how long its words take to read, and very short screens aren't timed at all. The one thing the browser does report is pasting, so Pasted text can hold a respondent but never rejects one alone.
What a rejected response costs
- A rejected panel response is reported to the panel as a quality reject: you are not charged for it, and it doesn't count toward your order, so recruiting carries on to fill its place.
- A rejected response from your own link doesn't count toward your target either.
- Any quota place it held is given back, so someone else can fill it.
A held panel response is paid as a complete, but the panel's final approval is held back for a few days so you still have time to remove it.
Removing a respondent
Editors can click Remove on any row. Removing a response deletes it and all its answers from every result, chart and download (and from the report the next time it is written), and it can't be undone.
- A panel response can be removed while its order is paused. It is rejected at the panel, so you aren't charged for it, and someone is sent in its place. Recruiting restarts in about an hour, so remove anything else you want gone before then.
- If the order is already closed, the panel can't accept a rejection any more. You'll be warned that removing it won't get your money back or a replacement, and asked to confirm.
- Someone still taking the survey can't be removed until they finish.
Who sets the levels
The levels and numbers above (the reading speed, how many grids, how alike two answers must be) are set by SortedResearch for every survey, not per study. What you control per survey is the set-up on your own questions, and which verdicts you include with the Quality filter on the results (see survey results).