How to Analyze Product Feedback

If you have read our article on how to ask for feedback, or already have a pile of feedback stacked somewhere you might find yourself at a point, where you are not sure what to do with all the gathered insights exactly. Feedback needs to add value and help you improve your product, but how can you categorize and analyze it?

After all, the feedback you get is only as good as the actionable insights you gain from them. But when it comes to categorizing and analyzing product feedback there are a few challenges to face. 

It can take a long time to categorize feedback. The creation of categories itself can be time-consuming, not to mention the time it takes to manage and maintain the list of created categories. 

Furthermore, visualizing the data and insights you collected can be tricky. 

So, how can you successfully face those challenges? Here is a step-by-step guide on how you can manage to categorize and analyze feedback efficiently. 

Step 1: Get it All in One Place

Even though it seems obvious, many times the problem with feedback management starts right here. And who is really to blame? There are so many different channels from which feedback can reach you. If you don’t have a process set up mistakes are bound to happen when it comes to transferring feedback. 

Setting up a manual process may also be an option for you if you are not looking to invest resources into building or buying a solution. In this case, make sure to outline the manual process and inform your team about it. This way you make sure that everyone understands the process and can follow it in order to not misplace any information.

Where to Transfer Feedback to?

The next big question that arises when it comes to analyzing feedback is: Where do you transfer your collected feedback to? The answer seems so obvious - a spreadsheet! 

Whether it is via Google Sheets or Excel, they still exist. Often with details about the customers, numbers, and mail addresses, as well as complicated processes that were built around them, just to make this tool work for collecting feedback. Maintaining this solution takes time and, therefore, money - and let’s not even get started on the privacy and security issues that go along with this method. 

Of course, spreadsheets can be used in a very helpful way. It can help collect all the anonymized feedback you have accumulated via multiple channels. 

For analyzing data, a spreadsheet, however, will have its challenges. While it is of course possible to use this document type it will need quite a long set-up process and maintenance can be a problem. Most likely, you will need the help of other tools to make sense of all this data. If you are looking for complex dashboards you can feed the collected data into analytics platforms like Tableau.

Step 2: Turn Qualitative Insights into Quantitative Data

Once you have located all the data in one place, it is time to organize the insights. To do this it is important to know what your customers are saying, not how they are saying it. 

If you are going through your feedback manually this will take some time. If doing this manually seems like a source of error to you, using a feedback management tool can help you categorize your insights more efficiently. Whichever way you chose to do it, we have some helpful tips for both.

Analyzing Feedback Manually

If your company can spear the time and resources, here are a few helpful tips to go through feedback manually that make categorizing your feedback easier:

Create categories first: 

It is very helpful to create a few categories before you go through the feedback, especially if you chose a manual process. Think on a general level – what will your customers be interested in? 

UX and Design, Bug report, or Integrations might be some of those categories. It is easier to start with more generic categories at first. If your categories are too specific, they can get confusing fast. This of course will change over time. Once you have a working feedback process set up, your categories will change, refine, and improve.

Create unambiguous categories: 

Make sure your categories are clear and unambiguous. If you have one category that is called “Billing” while another category is “Pricing”, you might know exactly what the difference is at the moment of creation, but keep in mind that this might change or someone else might be confused by this.

You could of course insert a comment to every category and differentiate them clearly. But some of your colleagues might not see the comment and just assign the first category that seems right to them. This is why unambiguity is important. It helps you to not make a mess of the feedback process you worked so hard on setting up.

Duplicate Category Check: 

Make sure that you are not using different variations for the same category. “Improve UX” can otherwise easily turn into more than one category, like “UX improvement” or “UX bug”. This can quickly happen when you are working on the data with a team. Our tip here is: clearly define guidelines for naming categories beforehand and have someone check for duplicate categories.

Word Spotting:

While word spotting is far from a scientific approach, it can be helpful if you don’t have an overwhelming amount of feedback. The idea behind it is, that if a word appears in a comment, you assume that this text is about this specific topic. For example, for the word “billing” you will assume that the feedback is about the billing feature. 

Using a formula or the search feature in a spreadsheet can sometimes simplify this process. However, we do not recommend relying on this method alone, as it is faulty. A comment that for example says “invoice feature bug” would slip through as the word “billing” is not included. It can be useful as a first run through your data. But count in the time you will need to go over the results again.

Biases:

If you are analyzing your feedback manually, you will face the challenge of personal tendencies. Be aware that you are on some level, whether it is conscious or not, biased. If you have a roadmap or prioritization of tasks in your mind, it is easy to look for feedback that matches your vision. To get rid of personal biases, make sure that more than one team member is involved in the feedback process.

Analyzing Feedback Automatically

If you are using a tool to help you categorize your feedback, this is how you will most likely proceed. Keep in mind that every tool is different. So, this can’t be seen as a general approach but rather a small push in the right direction.

Product Feedback Tools

If you are opting to automate your feedback analysis process, it is advisable to look into tools specifically designed for the purpose of collecting, analyzing, and prioritizing feedback. Here is what you should look for in such a tool and how they can support your workflow. 

Analytics Tools

If analyzing data by categorizing and prioritizing isn’t enough for you, you can go one step further and integrate an analytics tool. These take your insights and give you various visual presentations of it.

Metric Analytics:

If you are collecting feedback with surveys, like the NPS or CSAT or any other rating-based survey, an analytics tool can be of great help in calculating and interpreting the results.

Text Analytics:

Text analysis is time-consuming, especially if you have a significant amount of data coming in at once. Some analytics tools, therefore, work with AI. The AI is trained to group similar phrases into themes and, therefore, saves time in managing big text datasets.

Uncovering Unknowns:

Sometimes you are not quite sure what you are looking for, or correlations are unnoticeable to the untrained eye. However, analytic tools can help find insights and trends you might have missed otherwise.

Visualizing Data:

Getting all this new information can be overwhelming. To always be on top of things, visualizing the data is a big help. A good analytics tool will offer a variety of charts, graphs, and visual patterns. This makes it easy to share the data with other stakeholders and make data-driven decisions.

Step 3: Link Product Feedback to Customer Data

The usage of your product might differ from customer to customer. Of course, this also becomes evident when you are analyzing the feedback. There are two main things to consider when it comes to the feedbacker: their job and if they are a paying customer.

If your target audience, for example, is Marketing Managers, the feedback of an Accountant may not reflect your typical user base. 

It is also of note to differentiate between free and paying users. The reason behind this is evident – building something for non-paying customers will not earn any revenue.

Additionally, paying users are usually more focused on improving the product itself, while free customers prefer a wide range of features.

Separate the Wheat from the Chaff

After looking at the person behind the feedback, it is time to separate the wheat from the chaff.

As a product manager, you will know firsthand what path your product wants to follow. However, it might happen that the feedback you get does not fit in your product vision.

If you are building software for SEO Marketing, for example, you will probably not have any intentions to expand it to the field of e-commerce. We advise you to stay on the yellow brick road. However, looking around and seeing what is out there doesn’t hurt from time to time.

If you get feedback from a single individual about a new feature, don’t base your entire roadmap on just that. Instead, if you think their observation is a good idea, ask others specifically if they would like to see this feature in the future too. Maybe some of them have just found a workaround to solve the problem, or the problem still exists for them too. 

This step will clearly show you that not all feedback is equal. If you want your product feedback process to be effective, it is indispensable to link your insights to as much information about your customers as possible.

Here is a quick guide to linking customer data to feedback, so you can later derive accurate decisions:

• Long-time or short-time user?
• Paying customer?
• Existing or churned?
• Who is the account manager?
• Have they given any ratings (i.e., NPS, Google reviews, etc.)?
• If so, what are they?
• What is their MRR/ARR?
• How is their company structured? (i.e., one-person operation with one account or an international corporation with multiple subsidiary companies and accounts, etc.)

Step 4: Visualize Data Effectively

If we break it down, feedback comes in two forms: text or scores. While scores are typically fairly easy to visualize, text can be trickier.
The advantage of scored feedback is that you can calculate averages, frequencies, impact, and more efficiently. Here are a few ways to visualize scored feedback quickly and usefully.

Bar and cake charts

There is probably no need to mention these, as they are the very standard form of visualization. However, for the sake of completeness, we will quickly go through them here too. 

If you are working with averages alone, it doesn’t tell you much about the distribution of the feedback. If you take the CSAT score, for example, it will tell you the average satisfaction of your customers. This could either mean that the majority of answers were scattered around the average or that some of your customers tended towards the extremes. 

This could make a difference in your decision process. Bar charts are a great way to visualize the distribution of the answers. 

If you use a survey with closed questions and provide the possible answers you can treat this text feedback like a numeric one by assigning each possible answer a number. 

Example:

Question: “What did you think of the new feature?”

Possible Answers: “helpful, confusing, useless, intuitive, other…”

Numeric value: (1) helpufl, (2) confusing, (3) useless, (4) intuitive, (5) other

Keep in mind that you cannot calculate an average with these numbers. However, they are useful for determining the frequency. Bar and cake charts are great for this purpose. 

If your customers rated multiple features on the same scale, and you want to compare them directly, a stacked bar chart will help you do that. 

Profile Plot 

Those visualizations, of course, have their limitations, especially when it comes to text feedback. Here is where the profile plot comes in.
If you have categorized your text feedback, these are a great way to show how frequently each category came up while taking another variable into account.
This way you could, for example, directly compare what features or improvements have been suggested by your highest paying versus your free customers.

Word Cloud

If you are not (yet) working with categories, you might want to think about creating a word cloud. Word clouds are one of the most common ways of bringing word frequencies into a visual form. It is popular because it is easy to use, and it gives a good overview of frequently used phrases. However, it needs a little bit of groundwork to be an effective tool. The feedback has to be prepared so that only content words and stings of two or more words are present in the cloud.

If a word cloud doesn’t provide enough information for you, you might want to look into content correlations. You can do so by measuring how often words co-occur within the feedback. Even though it provides a lot of helpful information, correlations are not easy to do yourself, so we recommend using a tool for this. 

Impact & Effort Matrix

If you don’t want to go down the rabbit hole with your visualizations, we recommend a simple impact/effort matrix. It is a great method for prioritizing feedback and communicating which feedback will pay off if realized. And it is super easy to do! If you are doing this with your team, all you need is a big piece of paper or a whiteboard. Alternatively, you could use an online template, create your matrix, or maybe you already have this feature in your feedback tool. Either way, it can be advantageous to do this with your team. Have a group discussion about how much effort the realization of feedback will take as well as how big of an impact this change will have. Then place each feedback topic into one of four quadrants:

• Quick wins: top priority tasks, cost little work but offer great value
• Major Projects: great value, but also a lot of work
• Fill-ins: low impact but very easy to complete
• Thankless tasks: high effort tasks that have a low impact

Step 5: Be Thorough and Consistent Over Time

Many companies don’t see feedback management as an ongoing process but dependent on the point in time. Often it is driven by deadlines or an urgent executive decision.

This, of course, leads to incompletely and incorrectly analyzed feedback.
Oftentimes, only the newest feedback or feedback from only one channel gets analyzed, leaving countless possibly valuable insights uncovered.
Accordingly, this leaves you with distorted results that you base your decisions on, or worse, forget about them as soon as the deadline or task is over.
Not only does this way of feedback “management” waste valuable resources, but the data also can’t be used in the future, as it is just a momentary snapshot.

Aggregating feedback over time is crucial. Even if the feedback is time-sensitive, keeping it longer will help you generate trends and improve your processes over time. If you keep receiving comments on improving your usability, the bigger issue will not be clear immediately but develop over time.
The real payoff for a consistent product feedback process is that you can derive deeper and more farsighted insights. Important insights won’t get lost, and you can use your time more efficiently.

Conclusion

Not every feedback will require immediate action or action at all. However, how will you know which feedback adds value to your product? Gathering your feedback in one central place and sorting through it is the first important step to collecting valuable insights that allow you to derive actionable tasks.
After all, when your customers give you feedback, they expect you to listen to them and do something with it. So, whether you will implement suggested feedback or not, keep your customers in the loop. If you want to know more about how to act on product feedback, follow our guide and find out the most important things to look out for.