Writing-Good-Survey-Questions

How to Write Good Survey Questions

To write effective survey questions, you should always consider your respondents’ point of view. From question type, length, order and framing, many factors can influence their choices (intentionally or unintentionally).

Before we start, as a rule of thumb all your questions should contribute to your research aims. So, don’t ask any unrelated questions just because you want the data. Respondents tend to disengage of you ask more than they expected to be asked. You can read more about managing respondent expectations in our article on Creating a Survey (Content and Language).

You may also want to be aware the difference between Open Ended Questions and Close Ended Questions.

 

If you don’t have time to read the whole article, skip to the tldr (too long didn’t read) summaries.

 

Question Length

Good survey questions are short and simple. Longer questions can cause survey fatigue and will encourage respondents to satisfice or leave the survey. If you need to use longer questions, do so strategically.

Note: Satisficing is where a respondent only satisfies the minimum requirements necessary to achieve a goal. In the case of surveys, satisficing would be submitting a response, regardless if whether their answers reflected their opinion.

 

Tldr: Keep questions as short as possible.

 

Number of Questions

Don’t ask too many questions. The longer a survey is, the worse your completion and response rates will be.

 

Tldr: Keep surveys as short as possible.

 

Recall

When developing questions and answers, it best to keep respondents in mind. Questions that require respondents to recall information can be mentally-taxing and produce inaccurate results or encourage response bias.

Recall to past questions

Respondents may find it hard to recall previous answers, especially in larger projects with multiple pages. So, if you do need them to remember a previous answer, make sure the question was not asked too long ago.

 

Tldr: If they’re on questions 15, don’t ask them to remember their answer for question 2.

 

Recall to past events

It’s best to only ask questions that have occurred recently for respondents. The further in the past an event took place, the less likely a respondent is to recall their experience clearly.

E.g. Feedback on a product or service purchased last week will be more reliable than feedback on one purchased last year.

It’s always best to ask yourself whether respondents are likely to have an answer readily available to your question?

 

Tldr: Only ask what you’re sure they’ll remember.

 

Context

Each question should only focus on one subject and contain all the context respondents will need to answer. Any specifics they need to address in their answer should be raised within the question itself. E.g. behaviours, events, dates and times.

Bad Question: What is your income?

As you can see, the question above is far to general. Who’s income? What time frame?

Good Question: What was your yearly household salary in 2016?

This questions is more specific in what information is needed from the respondent.

A well written question means the same thing to every respondent.

 

Tldr: Questions should be specific.

 

Loaded words/ phrases

There are certain words that trigger emotional responses, create bias or cause offence. These loaded words and phrases influence a respondent’s answer choices and skew your results.

The language you use should be as neutral as possible and should be free of any social or cultural connotations.

 

Tldr: Use neutral language.

 

Leading questions

Leading questions are those loaded with emotional or persuasive language to sway respondents towards an answer. To create a survey that produces accurate and representative results, you should avoid leading respondents.

 

Example:

Bad Question: Do you have any concerns about your line manager?

This wording frames the department as having problems. Before the respondent has had a chance to evaluate whether they’ve experienced something negative, they’re lead towards that conclusion.

Good Question: Describe your working relationship with your line manager.

This version does not sway the respondent towards an answer. Instead, good survey questions allow them to give an organic response.

 

Tldr: Don’t lead respondents towards an answer.

 

Intent and Meaning

We recommend getting a third party to proof your survey before you launch it. They should evaluate the intent and meaning of every question and give you some feedback on if they were influenced.

In a perfect world, you’d launch a pilot version of the survey first. Then you could compare your expectations against your findings and make improvements.

 

Tldr: Triple check how questions may be interpreted.

 

Order

Question Order

Before you use the builder to create a survey, put together a list of questions and answers. This way you can work out any bumps before messing around with software features. Especially if you’re intending to employ features like skip logic, as respondent paths can become complicated.

Group your questions by topic and create a clear logical structure for your survey. Begin with the more simple and interesting questions to build rapport between you and the respondent.

Leave the more complex, demographic or sensitive questions to the end of your project. This also applies to questions requiring they give personal information (e.g. email addresses). By the time your respondents reach them, they’re more likely to complete your survey than leave.

 

Tldr: Logical question order. Simple and interesting questions first. Demographic and sensitive questions last.

 

Order Effects

The order of your questions can influence respondent’s answer choices. This is to say, an answer a respondent just gave may influence the answer they’re about to give.#

E.g. When answering Question B, respondents are likely to be influenced by what they chose in previous question. Of course, this only applies if the questions are related in some way.

There are two types of order effects: Contrast Effects and Assimilation Effects.

Contrast Effects: The order of questions results in great differences in respondent answer choices.

Assimilation Effects: The order of questions results in answer selections becoming more similar between respondents.

Unfortunately, there is no way to predict how the order of your questions will influence respondents. The best we can do is keep this information in mind when creating a survey.

 

Tldr: A respondent’s answer can be influenced by their answer to a previous question.

 

Answer Order

The order of your answers can also affect a respondent’s choices. This most common cause of this order effect is when the order of answers never changes.

 

Example:

You have a set of multiple choice questions, where the answer choices for each is ‘Yes’ and ‘No’.

 

Question 1:

A: Yes

B: No

Question 2:

A: Yes

B: No

Question 3:

A: Yes

B: No

 

The repeated order of question answers can cause respondents to satisfice and repeatedly choose an answer in the same position. I.e. they only choose answer B every time.

It could also manifest as acquiescence response bias, where respondents only choose the positive answer for each question.

 

You can read more about satisficing in our article on Response Bias.

 

One solution to this would be to randomise the answer option order for each question. This does not eliminate the order effect, but does ensure that the bias is spread randomly amongst the answers.

 

Here’s KwikSurveys’ help doc for Randomizing the Answer Order.

 

Order effects are also occur with rating scales (Strongly Disagree – Strongly Agree), where the answers are always in the same order.

These answers cannot be randomized as it would cause friction for respondents, who expect them to be in some order. However, you are able to reverse the order of answers for each alternating scale.

 

Example:

Question 1: Strongly Disagree – Strongly Agree

Question 2: Strongly Agree – Strongly Disagree

 

Tldr: Don’t use the same answer order for every question. Keep respondents on their toes.

 

Question Types

Surveys are already tedious enough without respondents answering the same question type for five pages. By using a few varying question types, you create a survey that’s more interactive and reduces the chance of respondent disengagement.

Tldr: Use a range of question types to keep the survey fresh.

 

Double Barrelled Questions

Double Barrelled Questions are those where two questions are asked in the space of one. You should avoid doing this at all costs. Each question should focus on collecting one specific set of data, as not to confuse respondents.

Tldr: Only ask one question at a time.

 

Question Dichotomy

Avoid writing survey questions that ignore ideological dichotomies:

Example:

“Do you think basketball players as being independent agents or as employees of their team?”

This type of question frames the choices as two mutually exclusive answers, where only one can be true at any time. But there may be respondents who believe both the above statements are true.

Tldr: Allow respondents to make their own assessments. Don’t create a ‘this or that’ scenario.

 


 

To produce the best results, you have to focus on writing good survey questions. And that means they have to clear, concise and specific.

It’s also important to understand that question text should be as neutral as possible. That is to say, not to lead respondents to a particular answer or way of thinking. This way your results are as accurate as possible.

We’ve also have some tips to help you write good answer answers, so why not take a look.

Share this post

Share on facebook
Share on google
Share on twitter
Share on linkedin
Share on pinterest
Share on print
Share on email