If your work involves public engagement, you’ve likely been tasked with drafting a survey (or will be in the future). It’s also likely that you haven’t been trained in survey design, so writing a survey can be daunting. There are so many questions about writing the questions. What do I ask? How do I ask it? What order should these questions be in? How long should this survey be? Will this survey provide the data we need? How do I even start?
This last question is the easiest to answer: Every survey should start at the end. You may have heard that writing a survey requires working backward, but what that entails isn’t always clear.
To work from finish to start, you (and your team) should answer the following three questions before you write a single survey question.
1. What decisions will be made based on the survey results?
Government agencies often rely on survey data when making decisions that affect public funds and have community consequences. Information about behaviors, priorities, preferences, and respondent characteristics can be incredibly useful in the decision-making process. But surveys are only useful in gathering this kind of information if they include the appropriate questions to capture it.
The first task when writing a survey, then, is to list all of the decisions that may be influenced by the survey results. Failing to do so can easily lead to an accidental omission of critical questions. Not only will this list inform what questions you ask in the survey, but it will also help you shape the responses you offer participants if only certain courses of action are available.
Let’s say you’re working on a transit plan and one of the primary goals is to determine a route for the new Bus Rapid Transit line. After an assessment of corridor conditions, you have narrowed down the potential bus routes from three to two. The survey should only ask respondents for their feedback on those two options.
What if decisions have already been made? Answering this question requires both honesty and transparency. If a decision has already been made, clearly say that in your survey. You can do this in two primary ways. The first is to communicate the decision and its justification through other methods - for example, a contextual statement or video that accompanies the survey. The second way is to ask for feedback on the decision. If you’re seeking feedback, make sure that the questions you use reflect that the decision is final. Either way, make it clear to your survey participants how their input will be used.
2. What do you need to report from the survey?
Key stakeholders and decision-makers typically have an idea of what information they want (or need) from public engagement initiatives. Information requirements can come from various sources, including legal processes, funding stipulations, directives from government officials or other decision-makers, or promises previously made to the public. Knowing what needs to be reported helps you determine what questions you need to ask.
If the law requires that someone provides their full name and residential address with their public comment for the official record, you will need to ask for that information. Or let’s say that you’re required to produce a report about the demographics of survey respondents. You’ll need to include questions about racial identity, age, income, and any other measures suitable for meeting those reporting requirements. Someone on the City Council might have requested data on the amount of public support for a particular initiative. You’ll want to include a measure of that support in your survey.
Regardless of who’s asking, you’ll want to ensure that you include the questions necessary to capture these informational requirements.
3. What do you want to learn from this survey?
Aside from what you need to report, consider what else you want to learn from the results. Understanding what decisions will be made is a good starting point, but you have the potential to glean a lot more information from your survey respondents.
Your project may not require collecting demographic information but you want that data to compare responses across different groups or to assess the equity of your engagement efforts. It’s also common to use a question or two in a current survey to plan for later initiatives.
A recent initiative in North Carolina included a public survey about bus ridership. Although the primary focus was on making improvements for current bus riders, the agency took the opportunity to identify why respondents did not ride the bus. Though not directly tied to current ridership, this type of information could help the agency identify other areas for improvement in public transportation.
There’s a balance to strike here, though. Curious minds will want to know everything about everything (I know because I’m one of them. Are you?), but you run the risk of making your survey too long. Every question you ask increases the likelihood that someone will stop responding, so use your survey real estate wisely. Pick the information that will be the most useful to your goals and focus on those. If you could succeed without that info and are merely interested in the results, it’s probably best to leave out those questions.
The Survey-Writing Moonwalk
Once you’ve answered the questions above, you’re ready to work your way backward to a quality survey. By identifying what you need to get from the survey, you have also outlined what you need to ask. Each item on the list represents a question or a set of questions that will be vital for collecting the data you need.
Of course, this method doesn’t write or arrange the questions for you, but it serves as a great blueprint for making these decisions. It can also help to avoid some common survey pitfalls:
Asking unnecessary questions
Asking redundant questions
Asking the wrong questions
Survey writing is a process - one that isn’t always easy. But taking time to think through these three considerations before attempting to draft a survey will make it much easier.