Text analytics: what changes when you can read every open-ended response?

Survey scores tell you how many customers are satisfied or unhappy. Open-ended responses tell you why.

Text analytics turns those responses into data. It identifies the topics in each comment, groups similar comments together, and measures how often each topic appears. The result combines the scale of a dashboard with the detail of the customer's own words.

That changes more than the speed of analysis. It changes what a survey needs to ask and what an organization can learn.

Why teams still read comments by hand

For large surveys, teams could review only a sample of the comments. Someone had to read each response, assign it to a category, and count the results. Even so, the people responsible for improving the experience often went straight to the verbatims. Dashboards made scores, especially NPS, easy to report, but a score only shows that something is wrong. A customer's words usually offer the first clue about what to fix.

That sampling created an unavoidable trade-off:

• Scores and fixed-choice questions could be counted across the full dataset.
• Open-ended responses provided more detail, but only a fraction could be reviewed properly.

Modern text analytics removes much of that trade-off. Every response can now be categorized using the same rules, while the original comment remains available for context.

What changes when every response can be analyzed

Analyzing every comment is not simply manual coding done faster. It changes how comments are classified and how the resulting percentages should be read.

One comment can contain several topics

A customer can write about more than one thing in a single response.

The app crashed, and then nobody replied to my message.

That sentence contains at least two topics: a product failure and a support failure. Placing it in only one category would discard part of what the customer said.

For that reason, text classification should be multi-label. A comment is assigned to every relevant category. If it mentions three issues, it is counted once in each of those three categories.

The categories are defined in advance, and the model classifies every new comment against that set. This is supervised classification. The categories shown in the report are not topics the model invented for that report. They are the topics the organization decided to track, populated by the model.

The denominator matters

In Numr's reporting, each category is shown as a share of the answered base: the people who gave a comment. A category tells you how many of the people who told you something raised that topic. The platform can be set to the total respondent base instead, but that setting makes every problem look smaller than it is.

Consider a survey with:

• 400 respondents
• 120 respondents who wrote a comment
• 72 respondents who mentioned the app
• 60 respondents who mentioned slow or missing support

On the answered base of 120, the app category is reported as 60% and the support category as 50%. A person who mentioned both is counted once in each category.

Two consequences follow.

First, category percentages can add up to more than 100%. Here they sum to 110%, and that is correct. The categories overlap because one person can mention several topics. Adding the percentages together does not produce a meaningful total, so they should not be normalized to force them to 100%.

Second, the base decides whether a problem looks serious. Take the 72 app complaints:

• Answered base: 72 of 120 commenters, or 60%. Three in five people who told you anything.
• Total respondent base: 72 of 400, or 18%. That looks like a minor issue.

Same customers, same complaints, two numbers. One gets the problem fixed and the other gets it deprioritized. The support category moves the same way, from 50% to 15%. That is why the answered base is the right one.

Always state which base a figure sits on. This matters more now, not less: the platform can be set either way, and a reader cannot tell from the number alone.

Classification improves with correction

Fine-tuning teaches the classifier how a particular client's customers describe products, processes, and problems. The starting classifier is a common base model that has not yet learned that language.

The fine-tuning cycle has four steps:

1. The model classifies a batch of comments.
2. Someone who understands the business reviews the results.
3. They correct labels that are wrong or incomplete, then publish the revised model.
4. New comments are classified using what the model learned from those corrections.

In Numr's internal testing across about 2 million responses in seven languages, the base classifier achieves more than 80% accuracy. After client-specific fine-tuning, accuracy rises above 95%. These are aggregate figures for model performance, measured across categorization levels. They are not a guaranteed result for every client dataset.

Each client's categories, corrections, model, and classifications remain separate. One client's corrections do not train another client's model, and one client's category set does not appear in another client's account.

Watch the catch-all category

Review the catch-all category regularly, because it is an early-warning system. A useful category set needs a catch-all for comments that do not fit anywhere else.

If similar comments begin to accumulate there, customers may be describing an issue the current category set does not recognize. Identify the recurring theme, and add a category when the evidence supports it.

A category set should evolve as the business and its customer problems change.

What text analytics makes optional

Text analytics makes the fixed list of reasons optional. A transactional survey often asks for a score and then presents that list. It makes answers easy to count, but it also predicts the customer's problem before the customer has described it.

Fixed-choice lists still have real advantages:

• They take less effort than writing a response.
• They produce stable measures that are easy to compare over time.
• They capture a signal from people who would never type a comment.
• A selected option is usually less ambiguous than a classified sentence.

But a list can report only the reasons someone thought to include. If the real issue is missing, the survey cannot capture it. Long lists are not a good answer either. In Numr's own work, a list of drill-down options stops earning its place at about 12, and a longer one still cannot anticipate every problem.

When every open-ended response can be classified and counted, fixed-choice lists are no longer the only practical route to structured data. They can remain where speed, consistency, or response rate makes them useful. They no longer need to define the limit of what the survey can learn.

The survey design handbook explains where different question types fit.

The survey can discover, not just confirm

This changes the survey's role. Instead of only confirming a predefined list of problems, it can discover issues in the customer's own account and at a level of detail that a short list cannot match.

A fixed option might record that “the app” was the problem. An open response might explain that the app crashed at the payment step immediately after an update. The second answer points to a more specific failure, a likely owner, and a clearer next step.

Numr's conversational surveys are designed to explore that detail. Before the first question, the survey explains that its purpose is to understand the customer's experience, not to provide live support. It then asks three things:

• what happened
• what the customer tried
• what prevented them from completing their goal

Why support transcripts are not the same thing

Support conversations are valuable sources of customer language, but they are held in order to resolve the customer's problem. When a customer says, “I can't do X,” the representative should help them do X as quickly as possible.

That conversation may reveal the root cause, but it may stop once the immediate problem is solved. A research conversation has a different purpose. It can consistently probe what happened before the customer asked for help, what they had already tried, and why the problem remained unresolved.

The two sources complement each other. Support data shows what customers bring to the service team and how those contacts are handled. Survey data can examine the broader experience, including problems customers never reported to support.

The limit that remains: customers still have to respond

Text analytics cannot make every customer write a comment. It can only analyze the responses that arrive.

In the earlier example, 70% of respondents left no comment. That is the strongest reason to keep scores or carefully chosen fixed-choice questions alongside open text. They provide a signal from people who will not write.

The improvement is not that open text replaces every other question. It is that the responses customers do provide can now be analyzed systematically instead of merely sampled or skimmed.

Last word

Scores tell you where to look. Customer language tells you what may need to change.

Text analytics makes that language measurable across the full set of responses. The better survey is therefore not “all open text” or “all fixed choice.” It uses each where it is strongest, without allowing a predefined list to cap what customers can tell you.

Frequently asked questions

Why do category percentages add up to more than 100%?

Because one respondent can mention several topics. That person is counted once in every relevant category, and each category is divided by the same base: the people who gave a comment. Each percentage is meaningful on its own; their sum is not.

What is fine-tuning?

Fine-tuning teaches the classifier how a particular client's customers describe their experiences. A knowledgeable reviewer corrects the model's labels, and those corrections are used to improve future classifications. The training remains specific to that client.

Why ask an open question when a list is easier to report?

Because a list captures only the reasons included in advance. An open question can reveal an unexpected issue and describe it in enough detail to act on. Text analytics supplies the structure afterward by classifying and counting the response.

Should fixed-choice questions be removed?

Not automatically. They remain useful when any of these matters:

• response effort
• trend consistency
• coverage

The point is not to replace every list. It is to stop treating the list as the boundary of what the survey can discover.