SurveyMonkey SMART SURVEY DESIGN User Manual

2008

SMART SURVEY DESIGN

This guide provides information on writing successful and effective survey questions, creating survey flow and layout, calculating response rates, tips for increasing response rates, and the pros and cons of online surveys. (Plus an appendix of links and works cited for additional help in survey design.)

Smart Survey Design

Table of Contents & Summary

I. Introduction: Relevancy and Accuracy…..………………………pg. 4

 Be familiar with the question  Know the objectives  Know the kind of information needed  Pay attention to wording style, type, and question sequence  Pay attention to survey length and make the survey interesting  Collect the data in a reliable and valid way to ensure accuracy

A. Question Intent……………………………………………………pg. 5

 Use legible questions  Use relevant questions  Use painless and quick to answer type questions

B. Good vs. Bad Questions………………………………………….pg.6

1. Be brief

2. Be objective

♦ Avoid leading questions ♦ Avoid loaded questions ♦ Avoid built in assumptions

3. Be simple

♦ Use words that are simple and direct ♦ Avoid technical jargon ♦ Adopt the same definitions throughout ♦ Avoid negatives or double negatives ♦ Avoid double-barreled questions

4. Be specific

II. Survey Question Types & Structure……………………………….pg. 9

A. Question Types…………………………………………………….pg. 9

1. Types of survey questions and examples

♦ Choice One Answer, Multiple Choice, & Open Ended ♦ Ranking questions ♦ Matrix-Rating scales

 Likert scales  Semantic Differential scales

♦ Balanced vs. Unbalanced-Rating scales

2. Additional things to consider:

♦ Avoid sensitive questions ♦ Memory questions ♦ Subjective questions ♦ Narrative questions

Page 1

Smart Survey Design

B. Question Sequence…………………………………………. pg. 14

 Opening questions  Question flow  Location of sensitive questions

C. Survey Layout………………………………………………... pg. 15

1. Basic guidelines

♦ Introduction ♦ Body of the survey design ♦ End of survey or thank you page

2. Layout for coding/editing and identification

♦ Identification ♦ Numbering ♦ Instructions ♦ Fonts & formats

D. Pre-Test………………………………………………………… pg. 18

 Evaluate survey‟s competency  Estimate the survey length or time to take the survey  Checklist of concerns to address with the pre-test

III. Survey Response Rates & Pros vs. Cons……………….......pg. 20

A. Survey Response Rate……………………………………….pg. 20

1. Response rate calculation

2. Factors in determining acceptable response rates

♦ The research purpose ♦ Type of statistical analysis ♦ How the survey is administered ♦ Relationship to your respondents

3. Tips for maximizing response rates

♦ Request participation in advance ♦ Allow enough time to complete ♦ Provide survey instructions ♦ Design easy to follow and clear/concise Survey ♦ Send reminder ♦ Offer an incentive

B. Pros & Cons of Online Surveys……………………….......pg. 23

(List continued on pg. 3)

1. Pros: ♦ Same strengths as a paper version

Page 2

Smart Survey Design

♦ Better at addressing sensitive issues ♦ Cost efficient ♦ Faster delivery ♦ Endless design options ♦ Dynamic ♦ Ability to track ♦ Quick response time ♦ Easier to use for complex routing or skip logic ♦ Automatic Randomization of Answer Choices

2. Cons:

♦ SPAM/Privacy ♦ Technical issues ♦ Submitting multiple submissions ♦ Not having an interviewer present to clarify questions

IV. Collecting secure data and privacy best practices……... pg. 26 V. Section 508 & Accessibility ………………………………….. pg. 26

1. Making your Surveys Accessible and 508 Compliant

2. Web Accessibility Survey Design tips

VI. Conclusion…………………………..................................... pg. 30

VII. Appendix 1…………………………..................................... pg. 32

VIII. Work’s Cited ……………………….................................... pg. 34

be copied without the express written consent of SurveyMonkey.

Page 3

Smart Survey Design

I: INTRODUCTION: RELEVANCY AND ACCURACY

Writing solid and unbiased surveys may seem like a daunting and unapproachable task. However, it is not as overwhelming as one may think! Creating well structured, simply written questions will help in collecting valid survey responses. An important goal as a survey author is to construct clear, direct questions and answers using the language that survey participants will understand. While there are no set rules on the wording of these survey questions, there are some basic principles that do work to improve the overall design. The intent of this guide is to provide tips on utilizing those principles for constructing effective surveys that collect accurate and valid data.

Relevancy and accuracy are two ideals that encompass the main outcome of

creating reliable surveys. These two principles work together to write effective survey questions. To achieve relevancy, keep the following factors in mind (Iarossi 2006, 27):

familiar

with the

Questions.

Know

the

Objectives.

Know

kinds of

information

needed.

Relevancy

The kinds of questions a survey author should create are based on two things: the

objectives of the survey and the information to be collected. A goal of the author is to

then turn those research objectives into a set of “information requirements.” From here, one can create questions that will produce that information (Brace 2004, 11-12). An accurate survey is one where the questions collect the data in a reliable and valid way. If the questions ask respondents things they do not know, then it can result in inaccurate data. To enhance the accuracy of respondents‟ answers, take into consideration the following items (Iarossi 2006, 28):

 Address the wording style, type, and question sequence.  Make the survey interesting and notice the survey length or how long it takes to

answer the entire survey.

 When designing a survey, the author should try to put him/herself “in the position of

the typical, or rather the least educated, respondent.” (Moser and Kalton 1971, 320)

Some additional things to consider about the relevancy and accuracy of survey questions are the ways in which the questions are written and their overall length. Writing

Page 4

Smart Survey Design

clear, direct, and brief questions will help the survey respondents to know exactly what you are asking. By making sure that the questions asked do not have more than one possible meaning also helps in preventing respondents‟ confusion. Asking sensitive questions in alternate ways may help to alleviate respondents‟ concerns. For example, many people may feel that income, age, lifestyle habits, etc. are personal and may not want to disclose that information. So when collecting a respondent‟s age, a person may be more willing to indicate what year s/he was born rather than to state an actual age. Finally, take into consideration the capability of your survey participants. Some participants may not be able to accurately answer certain questions. If you are surveying employees, perhaps they cannot recall certain details of a project carried out years ago. Or if you are surveying a consumer product, respondents may not remember specific features about it (“Survey

Planning”).

A. Question Intent:

Well-understood questions increase both the accuracy and frequency of survey respondents‟ answers. While creating survey questions, keep these two fundamental questions in mind in regards to the intent behind the questions and the data you want to collect (Iarossi 2006, 44):

 Will respondents be able to understand the question?  Will respondents be able to answer the question?

The legibility and relevancy of these questions will therefore play a key role on impacting each individual question‟s intent. By sticking to these three following suggestions, you may increase the accuracy and frequency of respondents‟ answers:

1. Use legible questions – Ask questions that read well and are quick and easy to answer. This may help to keep the respondents from jumping to an answer before the question is completely read. Avoid writing questions in a complex structure; sometimes the longer the list of questions/answers, the lower the quality of the data (Iarossi 2006, 44).

2. Use relevant questions – Make sure that all questions asked are relevant to all respondents and the survey‟s purpose. In addition, avoid hypothetical questions (Iarossi 2006, 44).

3. Use painless questions – Questions asked in your survey should require a small amount of effort to answer. Most people prefer to answer and complete surveys quickly without thinking too hard or spend a lot of time. If the survey is too long or becomes tiresome, respondents may adopt strategies to get to the end of the survey as quickly as possible. For example, with repeated rating scale type questions,

Page 5

Smart Survey Design

respondents may get into a pattern of response that does not reflect their actual thoughts. They may simply just click the 1st rating scale every time to answer every question and finish the survey quickly (Brace 2004, 18).

B. Constructing Good vs. Bad Questions:

Each survey question has a unique need. Because of this, there is no universal right

or wrong of “question wording.” However, there are ways to construct good vs. bad ones. The following four criteria can help you when wording and structuring your questions (Iarossi

2006, 30-44):

Be Brief

1.) Be Brief – Keep questions short and ask one question at a time. Longer questions may quickly become confusing, thus resulting in a misread of what you are asking.

Remember: Brevity‟s goal is to create the shortest way to ask a question without losing its intent. It is not always about reducing the length of the question (Iarossi

2006, 30-44)!

2.) Be Objective – As the survey designer, pay attention to the neutrality of the words. This helps to avoid unintentional violation of the survey‟s objectivity. Here are some tips to avoid violating the objectivity (Iarossi 2006, 30-44):

A. Avoid leading questions – Based on their content, wording, or structure, these

kinds of questions may lead a respondent towards a certain answer. According to Iarossi, the following three items aid in the creation of leading questions and survey writers should always try and avoid these (33-35):

 Failure to give equal weight to all options.  The actual set of options offered acts as a source of information.  The actual list of options provided will influence the respondents, meaning

the options that appear in the beginning of a long list have the “primacy

effect” and have a higher likelihood of being selected.

A good way to deter the “primacy effect” is to make the answer choices appear in a random order every time the survey opens up to a new respondent. This can

Objective

Simple

Specific

Page 6

Smart Survey Design

be accomplished with the “randomized answers” option as a Professional subscriber in SurveyMonkey.

Example of Leading Question Bias:

Example: We have recently upgraded SurveyMonkey‟s features to

become a first class tool. What are your thoughts on the new site?

Replace with: What are your thoughts on the upgrades to SurveyMonkey?

B. Avoid loaded questions – This type of answer bias works through emotionally

charged items like words, stereotypes, or prestige images. When creating the survey, avoid words that may “cater to the respondent‟s ego or contort the respondent‟s pride.” This may result in pushing the respondent towards a particular answer (Iarossi 2006, 30-44).

C. Avoid built in assumptions – When creating survey questions, avoid questions

that assume the respondent is familiar with the specifications asked within the questions (Iarossi 2006, 30-44).

3.) Be Simple – The survey should use language that is simple in both words and phrases. The following are some helpful points to remember for question simplicity (Iarossi 2006, 30-44):

 Use words and expressions that are simple, direct, and familiar to all

Respondents.

 Avoid technical jargons or concepts.  Adopt the same definitions throughout the form.  Avoid Negative or Double Negative Expressions. The use of universal words

or “absolutes” like “always” or “never” may cause the respondents to avoid answering a question. These tend to extend the question to an extreme. In addition, try not to use words such as “only” or “just.” These could be perceived negatively worded by the respondents (“Writing Survey Questions”).

 Avoid using Double-Barreled questions. Double-Barreled questions split

questions into more than one part, idea or meaning. The answer choice for each part might have separate meanings to the ideas presented within the one question. These may lead the respondent to answer only one aspect of it; ultimately leading him/her to fail in answering both aspects of the question. (“Writing Survey Questions”).

Page 7

Smart Survey Design

Example of a Double-Barreled Question:

Example: “How useful do you find SurveyMonkey‟s Help Center Topics and the email

support center?”

Replace with:

Question 1: “How useful do you find SurveyMonkey‟s Help Center Topics?” Question 2: “How useful do you find SurveyMonkey‟s email support?”

4.) Be Specific – Ask precise questions. Avoid things that are too general, too complex or undefined. Stay away from using words like “often,” “usually,” “generally,” etc. Each person‟s thought process is different and some people may infer a different meaning. “Often” to one person may be once or twice a week and to someone else it could be a few times a month. Do not use abbreviations and spell words out when necessary so it will not lead to potential confusion for the respondents (Iarossi 2006,

30-44).

Page 8

Smart Survey Design

II: SURVEY QUESTION TYPES AND SURVEY STRUCTURE

A. Question Types:

The types of survey questions used in a survey will play a role in producing unbiased

or relevant survey responses. As the survey designer, consider the types of questions to use and when it is appropriate to use them. Question types range from open-ended (comments to essays) to closed-ended (yes/no, multiple choice, rating scale, etc). In the end, it is the question types that determine what type of information is collected.

You may also want to consult with a statistician if you are unsure as to where to

begin (e.g. www.StatisticallySignificantConsulting.com) at the design phase of your study. This may help ensure that your data and research questions will lend themselves well to statistical analysis (Creech, Steve 2007). To view examples of the types of questions SurveyMonkey offers, please visit our homepage and access the “types of questions” example link.

1. Open-Ended Types: Open-ended questions are those that allow respondents to answer in their own

words. In an online survey, textboxes are provided with the question prompt in order for respondents to type in their answer. Open-ended questions seek a free response and aim to determine what is at the tip of the respondent‟s mind. These are good to use when asking for attitude or feelings, likes and dislikes, memory recall, opinions, or additional comments. However, there can be some drawbacks to using open-ended questions (Brace

2004, 55-62):

 Sometimes respondents may find it difficult to express their feelings. This can result with respondents answering “I don‟t know” or skipping it.  They do take more time and effort to fill out and at times they can have a

larger skip rate.  In addition, analyzing open-ended comments can be time consuming and difficult.

We have aimed to make that process a bit easier for Professional subscribers by offering a few summary spreadsheet formats in Excel, HTML, or downloading individual questions into a PDF.

2. Closed–Ended Types (Multiple Choice – One Answer or Multiple Answers):

Closed-ended questions are those with pre-designed answers with a small or large set of potential choices. One type of closed-ended question is a “dichotomous” question which allows respondents to choose one of two answer choices (e.g. Yes or No), while

Page 9

Smart Survey Design

another type is the “mutli-chotomous” question, which allows respondents to choose one of many answer choices (Brace 2004, 55-67).

Let‟s consider this example that asks similar information but uses different question types to ask for the information:

Question 1: Have you used SurveyMonkey in the last year to send out a survey?

 Yes  No

Question 2: How many times in the last year have you used SurveyMonkey to send out a

survey?

 Zero  Once  Twice  Three or more times

Question 3: If you had to estimate, how many times have you used SurveyMonkey to send

out a survey? (Please write in an estimate number) __________

In this example, Question 3 (the open-ended question type) provides the most detail.

Here a respondent can type in the actual number or the estimated number of times s/he has used the tool. The first type, Question 1 example, is the best for setting up a skip logic scenario, consent form, or for basic/general information gathering. For closed-ended questions, like Question 1 and 2, it is important to create answer choices that cover all possible answers that are expected from that question. If you make the choices mutually exclusive (or unable to all be true at the same time), then the participant is able to choose the best possible answer (“Survey how to”).

2. Ranked or Ordinal Questions:

Ranking questions are best to use when all the choices listed should be ranked according to a level of specification (e.g. level of importance). If you have a question in which you need the respondents to indicate what items are the “most important” to “least important,” then you can set up a ranking question (Waddington 2000). In this example on page 11, you can create a Rating type question in SurveyMonkey and set the “forced

ranking” option. This allows only one row (answer choice) to have that rating applied.

Example: If you choose “survey templates” as your 1

choice, then no other row can be

selected as the 1st choice. The Forced Ranking option won‟t allow it.

Page 10

+ 25 hidden pages

SurveyMonkey SMART SURVEY DESIGN User Manual

Specifications and Main Features

Frequently Asked Questions

User Manual