Common Pitfalls of Survey Design (And How to Avoid Them)

ux design - common pitfalls survey design

Welcome to the second part of the Beginner’s Guide to Writing Great Surveys.  The first part of this guide explained how to draft a series of survey questions that will provide you with the data that you most need, based on your goals. In this article you will learn some of the common survey pitfalls and how you can avoid them.

| Article written by Rachel Gita Schiff, founder and principal consultant at schiffresearch.com, research methods and evaluation enthusiast, data collection champion.  

Think back over surveys you have taken in the past. Have you taken a survey that was fun? Satisfying? Did you feel important after completing a survey?

For one project, I conducted surveys asking people about their home energy use. Maybe not the sexiest subject you might think of. But our target audience were households with extremely low energy bills. I asked questions about their schedule and habits, and understanding of home energy use. Almost every person I talked to was excited to share their tips and tricks, and thanked me for taking to time to listen to them.

This audience felt valued for being asked about a topic they cared about. My client got great information on how homeowners can save on their bill. This golden survey experience is exactly what you want to create.

One of the most common mistakes in survey writing is moving too quickly to question development, without investing some time and reflection into investigating what you, and your team, really need to know. Without that reflection, researchers often end up with surveys that are too long, biased, or confusing.

Survey Mistake #1: Too Long

If the survey is too long, people won’t finish. It can be challenging enough to get a click through on your survey link, to have long surveys and lots of dropouts means you are losing responses from people that you could have collected with a better written and tested survey

Surveymonkey found that surveys taking longer than 7-8 minutes have a huge jump in drop out rates. Keep your survey focused and as short as possible.

Survey Mistake #2: Biased questions

Surveys with biased questions lead to responses that may not actually be valid. Asking “Do you value eating healthy during the day?” primes the respondent to say yes, whether or not that is a major factor in their life, because it’s so obvious that they should be saying yes. That is bias. Biased questions make the data less useful overall, so remove or adapt them by all means necessary.

Survey Mistake #3: Confusing questions

Confusing questions are frustrating for your audience. Think about what you know about your audience. Of course, you can’t expect to know everything about them – you are sending them a survey to learn more after all. But you may need to know who qualifies and who does not (and then screen out those who don’t), the best way to reach your audience, and their comfort level with specific lingo or terminology used in the survey.

Here are some examples of confusing questions:

    • How happy or unhappy are you with the class schedule and curriculum? (It’s impossible to give an answer when the question asks two different things – schedule and curriculum)
    • Where do you prefer to buy red wine? (another potentially impossible to answer if you don’t buy red wine)

_ _ _

Let’s dig into some additional strategies to avoid common survey pitfalls:

Use best practices

Consider the Three Pillars of Great Survey Design. Getting clear on your goals for the survey will help you shorten that survey and cut all unnecessary questions.

Have a sample plan

Based on your goals and your target audience you can develop a plan for who exactly to survey and when. For example, if you are surveying current users, you should have a sense of how many in total there are. Good practice is often to survey only a sample of the total. One reason for this is to not overburdened your target audience. People already get so many requests for surveys and you can bet that you will be surveying again in the future. So keep it down to a manageable sample for now.

Share benefits

  • If you have the opportunity to provide incentives, such as a Starbucks or Amazon gift card, this is a great way to thank your survey respondents for their time and show how much you value their feedback.
  • One benefit that many survey writers miss is that people who participate are often interested in the results, so considering a plan to share some summary of the data with your users. This is a great incentive that can also encourage people to participate in future surveys!

Test your survey

Testing a survey, even by giving it to a friend to read, will help avoid biased and confusing questions. Time yourself and your friend to see how long the survey will take to complete. As you test and check your survey questions, also make sure that the final survey links work and the format looks as intended.

  • There are many other in-depth ways to test your survey, such as having an expert review. If you are writing a survey for a specific group, for example if you are writing a survey for baseball fans and dont know all the ‘ins and outs’ of the game, it may be helpful to ask for a review from a sports blogger or commentator to make sure that your examples make sense. When you progress to writing more complicated surveys, it might be helpful to have a survey expert or researcher review your survey and discuss statistical reliability and validity.

> Too Long, Didn’t Read? 

Survey Limitations

  • the survey is only as good as your questions
  • often unable to answer the ‘why’ of behavior, only the ‘what’
  • one point in time data collection.
  • statistical limitations

Best Practices for Survey Writing

  • Use the Three Pillars of Great Survey Design
  • Keep it short & clear
  • Keep it consistent
  • Then test, test, test
  • Want people to fill out your survey? Here’s some survey-asking etiquette.

_ _ _

Have a survey or want feedback from a dedicated community? Join Survey Share on Facebook, post your survey, and get good karma for answering other designers’ surveys.