“Perfect” is an emotion commonly felt when a researcher finishes crafting a question. The question is clear, the answer choices provided inclusive and mutually independent, there is no way this question will not return invaluable data! Unfortunately, this is sometimes just not the case. It is for this reason pretesting is an incredibly important, and surprisingly simple part of the survey process.
It is a very common misconception that pretesting a survey is a long and expensive process, and while it certainly can be, it does not have to be. Any testing that can be done prior to a full survey lunch is a very good thing so don’t think of it as all or nothing but as levels of added value. As a general rule five people will find 95% of the biggest issues. So start small…and cheap.
To help validate the cost and time a pretest takes remember that just five people could potentially save thousands of dollars, especially when you have outgoes in the thousands that are common in large quantitative studies. In most situations, most of people will have the same problems with the survey.
It is very true that adding more people might identify some additional smaller issues, but it also makes pretesting more time consuming and costly. It is for this reason that you should always do the initial pretest with no more than a handful of individuals to see what kind of issues you are dealing with before you add unnecessary time and cost to a project.
Often more important than the number of individuals that pretest a survey is how they pretest the survey. In order for a pretest to be as effective as possible the conditions the final survey will be administered under should be replicated as closely as possible. Your pretest should always be conducted via the same methodology as survey is intended to be conducted in.
If issues are found in your pretest you should of course take the time to talk with your pretesters to find out exactly where things went wrong. Did they not understand the question? Were they unable to find a single correct answer choice? Chances are the same individuals who identify the issue could be the ones who provide a solution so do not be afraid that you will have to completely redesign your survey because it tested poorly.
Ideally, after the issues flagged in the initial pretest are corrected another pretest should be conducted but make sure you use common sense here. If the issues were minor or if the fixes were simple then you can proceed straight to full launch, confident that the survey will provide solid data that translates into true insights and you can be the hero that all researchers fantasize about being.