The Problem Isn't The Survey

Someone recently told me they were about to say something controversial:

They don't like engagement surveys.

Even as a self-proclaimed survey nerd, I get it.

Too often, surveys are used as a substitute for actual engagement: we send out a link, collect responses, call it participation, and move on.

And there are plenty of bad surveys out there.

But I'm going to defend the humble survey.

Because when a survey is the right tool, it's a really good tool.

And I think we're sometimes blaming surveys for problems that aren't actually the survey's fault.

Surveys can do a lot with relatively little.

Think about what we're able to do with a survey today.

Someone can respond from a computer or phone. Scan a QR code on a flyer. Fill out a paper copy at the library. Use a tablet at a community event. Follow a link sent by text. With some tools, they can even complete the survey directly through SMS without opening a traditional web survey at all.

We can translate surveys into multiple languages. We can bring tablets and paper surveys into neighborhoods. We can work with trusted organizations to reach people through channels they already use. We can deliberately take a survey to people instead of waiting for them to find us.

And people can participate when it works for them, not just at 6 p.m. on the particular Tuesday we've scheduled a meeting.

That gives us the potential to hear from hundreds or thousands of people, through multiple modes, relatively inexpensively.

For government agencies working with limited staff and budgets and communities with limited time to give us, that's a powerful tool.

Surveys can tell us more than how many people checked a box.

Surveys aren't inherently quantitative.

A well-designed survey can help us understand behaviors, experiences, priorities, perceptions, and preferences.

Closed-ended questions can help us identify patterns and make comparisons. Open-ended questions can give people room to explain an experience, tell us why something matters, or raise something we didn't think to ask.

And we can put those things together.

A survey might tell us how many respondents have experienced a particular problem, whether that experience differs across groups or places, and what respondents want us to understand about that experience.

That's a lot of information from one tool.

But here's where I think some of the criticism of surveys is absolutely fair.

A survey isn't a conversation.

A survey doesn't inherently build relationships. It isn't deliberation. It usually doesn't let us hear something unexpected and immediately ask, Tell me more about that.

Those are real limitations.

But they're limitations of what the method is designed to do, not evidence that the method is bad.

A public meeting doesn't efficiently collect standardized information from 2,000 people. A focus group doesn't give every resident an opportunity to participate on their own schedule. An interview doesn't make it particularly easy to examine patterns across hundreds of respondents.

We don't conclude that those methods are bad because of what they weren't designed to do.

Methods have jobs.

Sometimes we need to probe. Sometimes we need dialogue. Sometimes we need people to work through tradeoffs together. Sometimes we need to explore an issue we don't yet understand well enough to structure into questions.

And sometimes we need to systematically collect information from a lot of people.

The useful question isn't whether surveys are better or worse than interviews, focus groups, workshops, public meetings, or other engagement methods.

It's: What do we need to learn, and is a survey equipped to help us learn it?

A survey is only as good as the decisions surrounding it.

Now imagine an engagement survey that:

• asks questions that won't inform any decision;

• includes confusing or leading questions;

• gets distributed through channels that miss important populations;

• collects demographic information nobody needs;

• treats an open convenience sample as representative of the entire community;

• produces a 90-page spreadsheet nobody planned to analyze; and

• never tells participants what happened with their input.

  
  

That's a bad survey.

But notice what went wrong.

None of those problems was caused by the fact that we used a survey.

They were caused by decisions we made before, during, and after using one.

So here's my defense of surveys:

When we hold up our end of the bargain, look at what this tool can actually do.

We can start with a decision instead of collecting information indiscriminately. We can write questions that measure what we intend to measure. We can intentionally reach beyond the people who usually participate. We can collect quantitative and qualitative information. We can analyze patterns without pretending our respondents necessarily represent everyone. And we can show people what happened because they participated.

None of that happens automatically.

The survey gives us the tool. The methodology determines what we can responsibly do with it.

So what does holding up our end of the bargain look like?

1. Start with the decision, not the questions.

Before writing a question, ask:

What decision will this information help us make?

Not: What would be interesting to know?

Not: What did we ask last time?

Not even: What do we want to ask the community?

What decision are we trying to make, and what information do we need to make it?

Then work backward:

Decision → Information Needed → Questions

That order matters.

Otherwise, it's remarkably easy to create a perfectly respectable survey that produces a pile of perfectly respectable data that nobody actually needs.

2. Make every question earn its place.

“Keep your survey short” is common advice.

I'd make the rule slightly different:

Make your survey as long as it needs to be—and not one question longer.

For every question, ask what you'll learn from the answer.

Better yet, ask what you'll do with it.

If respondents answer A, what does that tell you? If they answer B, what changes?

If every possible answer leaves you in exactly the same place, why are you asking?

Participant time is an engagement resource. Spend it deliberately.

3. Asking a useful question and writing a good question are different skills.

Knowing what you need to learn doesn't guarantee that the question you've written will actually measure it.

Question wording matters. So do response options, question order, definitions, and recall periods. Seemingly small choices can affect the answers people are able (or encouraged) to give.

Are you asking one thing at a time? Could reasonable respondents interpret the question differently? Do your response options accommodate the answers people might actually have? Can respondents reasonably know or remember the information you're requesting? Does your wording imply that one answer is more acceptable than another?

And perhaps most importantly:

Does the question actually measure what you think it measures?

These aren't just technical niceties.

A poorly designed question produces poor data, and sophisticated analysis later won't fix it.

4. Distribution is part of survey design.

Too often, distribution becomes the conversation that happens after the questionnaire is finished:

Okay, how do we get people to take this?

I'd move that conversation much earlier.

Who needs to participate should shape how the survey is designed in the first place.

Who do we need to hear from? Where can we reach them? What could make participation difficult? Which languages are needed? Is online participation sufficient? Do we need paper? Tablets? SMS? Community partners? In-person intercepts?

And after responses start coming in:

Who aren't we hearing from?

A beautifully designed questionnaire distributed exclusively through channels that reach the people you always hear from is still an engagement problem.

Distribution is survey design.

5. Don't make your data say more than they can.

Suppose 847 people respond to an open community survey.

That's valuable information from 847 people.

It does not automatically mean: “The community thinks…”

The claims we can make depend on how participants were recruited, who responded, who didn't, how well respondents reflect the population we're interested in, and what we're trying to infer from the results.

This matters because the alternative isn't “representative or useless.”

There's an enormous amount of useful territory between those two things.

An open survey can identify experiences, surface concerns, reveal patterns among respondents, generate ideas, identify questions for further investigation, and contribute evidence alongside other engagement methods.

We don't need to diminish useful data.

We just need to be honest about what those data can support.

6. Design the analysis before you collect the data.

This is one of the easiest ways to discover a weak survey before it's too late.

Imagine the survey has closed.

Now what?

Which questions answer which information needs? What comparisons matter? Are there groups or geographies you need to examine separately? How will you analyze open-ended responses? What would constitute an actionable finding?

If you can't picture what you'll do with the answers, go back to the questionnaire.

Planning the analysis before launch has a wonderful side effect:

It exposes questions that don't need to be there.

7. Closing the survey isn't the end of the engagement.

People gave you their time. They told you about their experiences, priorities, concerns, and ideas.

Tell them what happened.

What did you hear? What did you learn? What happened next? What changed? What didn't—and why?

Closing the loop doesn't mean doing whatever received the most votes. Engagement isn't necessarily a referendum.

It means showing people there was a reason you asked.

And if we want people to participate the next time we ask, that matters.

Don't blame the method for bad methodology.

So, back to the person who told me they don't like engagement surveys.

I don't actually think they're entirely wrong.

If we're using surveys when we need dialogue, asking questions we don't need answered, writing questions that don't measure what we think they measure, missing the people we need to hear from, calling convenience samples representative, or disappearing once the responses come in:

I don't like those engagement surveys either.

But that's precisely why I'm not ready to give up on the method.

A survey can only collect the information we design it to collect, from the people we manage to reach, and support the conclusions the data allow us to make.

That's not a reason to distrust surveys.

It's a reason to use them well.

The survey didn't decide what to ask.

It didn't decide who to invite.

It didn't decide how to interpret the responses.

And it didn't decide whether anyone would ever hear what happened next.

We did.