How To Get Useful Data From Public Survey Comments
"I know that no one will possibly ever read this.”
This phrase was one of the 1,600 public comments a county received on their most recent public survey. Sound familiar?
Far too often, the public is right. As much as project managers want to read every comment they receive through public surveys (and other engagement channels), they often lack the time to do it. Sure, it’s easy enough to read through a few dozen comments, but what about hundreds, or even thousands?
This is the downside to high response rates on surveys that allow open-ended comments. When resources are limited or there’s a small window of time between the close of the survey and the publication of the results, many agencies are forced to cut corners when it comes to reading open-ended comments.
At worst, this can mean that comments are downloaded and stored but never analyzed. At best, someone may skim the comments and overgeneralize the findings without thorough analysis.
So what’s the solution? In this case, the aforementioned county invested in the qualitative data analysis services offered by NineteenEleven Consulting. That means they can assure their survey participants that every single comment was read and analyzed.
How is it possible to read and analyze hundreds or thousands of comments? What kind of insight can agencies gain from this kind of analysis? Let’s walk through how it works.
What is Qualitative Data Analysis?
Asking open-ended questions in a survey provides you with qualitative data (e.g., words). This type of data is really useful for collecting descriptive information like what, why, or how, but it’s a lot more labor-intensive to work with than the quantitative data (e.g., numbers) you get from closed-ended questions like multiple choice or ranking. Why? Because words have to be converted from a jumble of disorganized puzzle pieces into a clear, understandable framework.
Here’s a very simplistic overview of the qualitative analysis process. Step one requires that you read every single comment in full, even the ones that may not seem relevant. As you read, you’re also coding each comment, assessing sentiments, identifying overarching themes within the data, and refining the codes as necessary. Once the data are organized according to thematic patterns, the data analyst then explores the relationships among themes, the nuances within them, and their connection to any associated quantitative findings. For example, a data analyst might make connections between a quantitative question that asks about support for a new dog park and qualitative comments that include feedback about what should be considered when creating those parks. The result is a comprehensive narrative that distills the voices of the people into a deliverable that decision-makers can digest.
Benefits of Qualitative Analysis
Yep, it’s an arduous process. It takes a good amount of time and a great amount of skill to analyze qualitative data. But it is worth it.
1. It shows you’re listening. Qualitative analysis can help your agency restore faith in public engagement by assuring participants that their contributions were not made in vain. Not only did someone read their comments, but they coded and analyzed them. Perhaps the decisions you make didn’t align with their ideas, but you can prove that their perspectives were considered. That matters. Not just for the current project but for every other initiative that follows.
Properly analyzing qualitative data also makes it easier to fully close the feedback loop with participants. It’s common for agencies to release summaries of the quantitative data (the numbers), in part because staff is more likely to have quantitative analysis skills. If they don’t, technological tools like Excel or Google Sheets and engagement platforms like PublicInput will do the analysis for them (pivot tables ring a bell?). Numbers are also easier to display and explain to the public.
But respondents want to know what you did with the comments that they offered, especially because it takes a lot more effort to type out a response than it does to make a selection from a list. Analyzing qualitative AND quantitative data means you can include a full overview of participant responses as part of your survey results.
2. You get more comprehensive results. Qualitative data provide context and deeper insights for your quantitative findings. Why did respondents favor one of the design options over the others? What conditions make them feel unsafe at their bus stop? How are they making sense of the protests occurring downtown? Asking open-ended questions to get this type of information allows the respondent to use their own words (and own tone) and reduces the amount of bias that can be introduced into a question when you provide predetermined response categories.
Qualitative analysis strengthens your decision-making because you have a better understanding of your residents. Their comments may uncover perceptions and motivations that make you rethink your approach to addressing a particular problem. Their insights may reveal that your definition of the problem is a bit askew and the focus should be shifted. They may offer creative solutions that you hadn’t yet considered. Given how much there is to learn from open-ended questions, we shouldn’t completely shy away from them simply because they require more analytical attention.
The Bottom Line
What you get from qualitative data is worth the effort, but it must be done well and done thoroughly in order to be useful. If you want to do this work in-house, be sure that you schedule ample time for the process and that the work is not rushed. Further, identify staff who are trained in this type of analysis because it involves far more skills than merely tagging keywords and creating a word cloud. Use technology cautiously as ‘search and tag’ functions can miss comments containing misspellings and typos, which can amplify certain voices while drowning out others. Automation is also somewhat limited in its ability to accurately capture sentiment and tone, so analysts should always review the results.
If qualitative analysis is something that you cannot or do not want to handle internally, consider hiring a knowledgeable expert to perform the work. Look for companies like NineteenEleven Consulting that specifically offer qualitative analysis as a service and that have staff who are experienced in qualitative research methods.