As we discussed in a previous blog, open-ended questions can help you collect different types of information than closed-ended questions. Open-ended questions can give you insight into someone’s explanation (the why) or their processes (the how). These questions are also useful when you want to solicit creative feedback or provide an opportunity for respondents to offer options beyond what you have listed with a closed-ended question.
People often assume that writing an open-ended question is quick and easy: Ask the question you want people to answer. Well, it’s not that simple. People typically won’t respond to poorly written, directionless open-ended questions, or they’ll provide responses that are off-topic. To increase response rates and ensure that you’re getting relevant feedback, consider the following as you craft your open-ended questions.
Provide the Why
Because open-ended questions take more time and effort to answer, participants are less likely to answer them compared to closed-ended questions. Including an introductory statement that explains why you’re asking the question is one way to encourage higher response rates. Will the feedback you receive help to inform what amenities are offered at a new bus stop? Do the ideas provided by respondents shape your public art plans? Respondents are more likely to answer open-ended questions when they believe their input will contribute to something meaningful.
Open-ended questions can also seem more intrusive because they ask for more detail and require respondents to use their own words. Introductory statements can help reassure participants that their information won’t be misused or simply ignored.
If the data gathered through the open-ended question will not be used, or you cannot articulate how it will be used, do not include it in your survey. Every survey question (open- or closed-ended) should be intentional and purposeful. Asking superfluous questions makes the survey longer, increasing question non-response (they skip the question) and survey attrition (they don’t complete the survey). Asking questions without plans to use the feedback fuels community distrust and undermines the success of future engagement efforts.
Set Parameters
Survey-writers sometimes want unrestrained, unbiased community feedback from respondents, so they pose questions that are too broad. This approach is problematic because vague, “big” questions like “Is there anything else you want us to know?” often translate into non-responses (no data) or irrelevant responses (useless data). When respondents don’t understand what’s being asked or they have to make assumptions about the type of information they should provide, they are more likely to skip the question altogether or write some iteration of “I don’t know.”
For others, broad, open-ended questions become a catch-all, regardless of the survey topic. Recently, I analyzed the comments from a survey that asked residents how they wanted to receive information from the County. The prompt “Please provide any comments regarding [the County’s] communications efforts” resulted in irrelevant, off-topic responses ranging from traffic accident reports to suggestions for preschool recreation programming.
Open-ended questions always have the potential to veer off-topic, but providing clear guidance for respondents can help keep their responses relevant. Question guidance should include anything you want respondents to consider in their response. What is the topic? What aspect of the topic should they focus on? Is there a particular timeframe to consider? Is this about a particular geographic area? Is the question about an individual, a household, a group? By giving respondents a frame of reference, we draw their attention to the question we are actually asking and give them what they need to contribute fully and meaningfully.
Don’t Load or Lead
Writing open-ended questions is a balancing act because, while you need to provide specific parameters to keep respondents focused, you want to be careful not to bias questions by making those specifications too strict. When there is too much specificity, you risk loading the question and unintentionally pushing respondents into answering in a way that may not reflect their perspective, if they respond at all.
A loaded open-ended question looks like this: “What do you like most about the new shopping center on Main Street in Gotham?” The phrasing of this question assumes that respondents like something about the new shopping center. By asking the question this way, respondents may feel obligated to find something they like about the center - even if they don’t like it at all. This biases the results or leads to a higher rate of non-response for the question.
Another way to bias an open-ended question is to lead respondents to a particular response by leaving hints about the type of response you are looking for. It sounds a lot like loading, but leading involves using value-laden words as breadcrumbs to elicit certain types of responses. Consider this prompt: “What do you think of schools teaching the fallacious notion known as Critical Race Theory?”
Notice anything? The word ‘fallacious’ implies that the survey writer has inserted their particular perspective into the question and is looking to confirm their perspective using answers to this question. This is a particularly egregious (and real) example, but leading can be more subtle. Especially when it comes to government engagement survey topics, plenty of words have connotations that can be inadvertently leading if used improperly or left undefined (e.g., gentrification, development, citizen, crime, etc.).
In short, you’ll need to be specific enough that respondents understand what question they should answer but not how they should answer.
Conclusion
Open-ended survey questions are great for capturing qualitative data if they are written well. Unlike public meetings, focus groups, and stakeholder interviews, surveys do not allow respondents to ask follow-up or clarification questions. You’ll want to make sure that any open-ended questions included in the survey are intentional and written in a way that focuses respondents on a particular topic without biasing their responses. Not only will this make your survey more ethically and scientifically sound, but it will also result in more useful qualitative data for decision-making.