Online sampling fraud is an evolving challenge in market research. While many reputable panel companies have systems in place to recruit quality respondents (e.g., speeders, straight liners, red herrings), the opportunity for fraud has expanded faster than their methods to catch or prevent it. Fraud can be completely unnoticed if your data are not thoroughly examined; and if left unchecked, fraud can lead to invalid results and misleading insights.
What is being done to combat fraud outside of the panel companies? According to the 2022 GRIT Business and Innovation report, fewer than half of all market research suppliers invest in any fraud detection services or processes. This should be a concerning statistic for anyone who relies on the integrity of their data to drive their most important business decisions. Even if you trust your panel partner to deliver quality completes, it’s a common industry practice for panels to work with 3rd party suppliers to help with fieldwork, and those 3rd party panels may not have the same quality standards. It always pays to check the data!
Here are 3 red flags to help you identify fraud in your next online survey:
1. Questionable Open-Ends. While some panel providers perform their own quality checks on open-ends, this can be a very subjective process. Here are some of the typical (and not so typical) indicators of fraud:
-
-
-
- Language: Responses that are inarticulate, illegible, or illogical are clear indicators of someone who either doesn’t have the ability to provide a valuable response or the inclination to do so.
- Content: Nonsensical answers can be easy to identify and remove, but vague answers are a little tougher to spot. Keeping the context of your study in mind and the specificity of the question will help determine whether or not a particular verbatim comes from someone who is knowledgeable about the subject matter. This can be especially apparent when surveying healthcare professionals about technical or medical topics – something that is specifically tied to their training.
- Length: Short comments, which can vary based on the question that was asked, are easy to spot visually in a data set and worth examining closer as they could be throw-away answers from a respondent who is trying to get through the survey as quickly as possible.
- Repetition: Repetition can happen for multiple reasons. If there are batches of responses that look exactly the same, this could be the result of one person taking the survey multiple times, which may result in similar grammatical or spelling errors across multiple records. Or, it could be because respondents are using Google to find the “correct” answer and copy/paste the results directly into the survey. Often this will be from an article that appears at the top of a Google search that is generally aligned with the survey topic: e.g., a Wikipedia page.
-
-
2. Getting a Surge of Completes in a Short Amount of Time. Throughout fieldwork, it’s important to monitor the speed at which respondents are completing your survey. If there is a sudden surge of completions, you may have to determine if this is the result of a large release of invitations or if it could be a sign of fraudulent respondents (or bots) taking the survey all at the same time.
3. Questionable Demographic Trends. Whenever possible, it’s helpful to have a hypothesis for what your demographic breakdown could look like before you launch a survey. In some cases of noticeable fraud, respondents take the same path through the survey each time once they learn that it works (i.e., bypasses any terminations). Often, these will lead to unexpected or strange demographic profiles. For instance, if 20% of your respondents are from Alaska, or if nearly everyone is in the 45-50 age range, ask yourself, “does this align with my target population or sample targeting efforts?”
To ensure the integrity and validity of your research results, make sure your insights partner is taking the appropriate measures to not only deter fraud but catch it if it happens. Reach out to Cue Insights to learn how our fraud-busting practices can deliver quality insights from your next research study.
Have respondent fraud concerns or questions? Email denise.burns@cueinsights.com to schedule a consultation.
