We’ve talked about why brainstorming product ideas is essentially a gamble in which you place all your money on fickle Lady Luck (and usually lose). We’ve also explained the flaws in traditional means of filtering out these ideas on the off chance you’ll find the winning needle in the haystack. So you’re probably thinking you have the perfect solution: Feedback! Why not just ask the actual customers what they want? Surely they must know their own needs better than anyone else?

The reasons that you’d jump to this conclusion are understandable, but the truth is, customer feedback is no magic bullet for success. It can be a useful tool, under certain conditions, if -- and only if -- you understand how to ask the right questions and how to evaluate the answers more critically. But while customers may indeed understand what job they’re trying to accomplish, asking them to come up with solutions to their own problems generally goes horribly wrong, as we’ll see.

The Case of The Misguided Solution

In the early 1980s, Kawasaki, the makers of Jet Ski, dominated the recreational watercraft market. They wanted to continue to improve their product, so they asked their users for suggestions. At that time, standing models were the only type of Jet Ski available, and the customer feedback was clear: They wanted extra padding on the side of the Jet Ski to make the standing position more comfortable. And so, Kawasaki dutifully focused on giving their customers exactly what they asked for. Meanwhile, other manufacturers developed models that allowed the user to sit down while operating the watercraft, and Kawasaki was eventually bumped from its position as market leader.

Kawasaki’s big mistake was asking customers what specific improvements they wanted, rather than digging deeper and trying to understand not only what they were trying to accomplish, but the obstacles that stood in their way. To put it another way, if Kawasaki had focused on ascertaining what job(s) users were hiring the jet ski to do -- quickly and comfortably navigate a body of water, for example -- they may have removed what was standing in their way, innovated a more comfortable seated jet ski, and remained in the lead market position.

In fact, asking the customer to come up with innovative ideas can actually stifle innovation. This can not only lead to the wrong kinds of products, but result in making incremental rather than bold innovations. In Zero to One, Peter Thiel and Blake Masters warn against accepting such incremental types of innovation as the status quo. The bursting of the dot com bubble in the late 90s, they explain, has made people afraid to make big leaps, and incorrectly believe that: “Anyone who claims to be able to do something great is suspect, and anyone who wants to change the world should be more humble. Small, incremental steps are the only safe path forward.”

But true innovation isn't about the kinds of tiny changes you'll be spurred to make if you only ask your customers how to improve your product — just like with the Jet Ski.

Here’s another example. Take the job of “listening to music.” Throughout history, people have always looked to a succession of products and scenarios accomplish this goal: they’ve gone to concerts and listened to phonographs, record players, tape players, CD players, MP3 players, and finally, streaming services. Each successive technology does the job better and better, removing more obstacles to the desired result. The jump from CD players to MP3 players, for example, meant that the previously unmet need of being able to quickly and easily determine and control the order of the songs -- a playlist -- was now met.

But ask yourself: If you could go back in time and ask the users of a portable CD player how to improve the product, do you think they’d describe anything like an MP3 player? Of course not -- they wouldn’t be aware that technology could make such a thing possible, even on the highly improbable chance that they would come up with the idea in the first place. They’d probably suggest something like: Make a CD player that doesn’t skip when I exercise. Or: Make a CD player that can play more than one disc and shuffle songs. 

But what if you had dug deeper and tried to understand everything they wanted to accomplish when listening to music, as well as the obstacles that stood in their way? Ideally you would have combined listening with observation. You may have observed the behavior of people burning their own custom CD mixes. Perhaps you would have discerned their need to have more portability of their device and control over the order of the songs played. It would then be your job to innovate the right solution -- something like an MP3 player.

Feedback Traps To Avoid 

As a general rule, observation is superior to feedback. Watching customers interact with your product (or lifelike prototype) means you get to observe and measure the successes and obstacles they actually encounter when trying to make progress in a task. Observation allows you to collect quantitative, objective data, while even the best feedback is qualitative, subjective and open to interpretation.

That said, you can only observe people if there is a product, feature, or realistic, high fidelity prototype for them to interact with — and occasionally even observations need to be clarified with a bit of digging. In other words, sometimes you have to gather feedback. But if you want to turn feedback into valuable action, you must follow an organized methodology. We call it the SLOW interview, and we'll explain it in detail here. Without a method, you'll find yourself falling into some inevitable feedback traps and pitfalls:

You'll end up asking your customers for solutions.

As we explained above, it isn’t your customer's job to come up with solutions to their own problems. They simply don’t have the tools to do so. Take this hypothetical scenario. Imagine this pre-iPhone conversation between Hillary Clinton and Steve Jobs:

“So Hillary, what would you like to see in my new smartphone?”

“Well, Steve, I’d really like a keyboard and curved screen.”

If this had actually happened, and if he had taken Hillary's advice, Jobs would have (maybe) improved on the Blackberry. But of course Jobs wasn’t thinking about how to improve an existing product, and he wasn’t soliciting this kind of incremental feedback from users. Instead, he was preoccupied with creating a radically better smartphone, one that would allow users to accomplish more goals than they ever imagined possible.

You'll unwittingly ask leading questions

In a blog post on "Digital Marketing Headhunter," data analyst Jim Durbin says that “Feedback is easily gamed by how you solicit or design the gathering of data.” In other words, the way you ask questions can have an unintended impact on the answers. For instance, most people will answer “yes” if you ask them a question like, “Would you like it if we added a shiny new red button to the homepage?” For one thing, it’s human nature to try to give the answer we think someone wants to hear. And for another, it’s difficult to imagine exactly what a theoretical feature would be like, so answers to such hypotheticals are unreliable.

You'll confuse the customer with the job they’re trying to do. 

Remember, you’re trying to gather feedback to help you understand what the customer is trying to accomplish and the obstacles that stand in their way so that you can make a better solution. That’s not the same as a misguided attempt to understand your consumers’ psyches or create personas (more on that later). In the Harvard Business Review, Harvard Business School professor Clayton M. Christensen, Intuit’s Scott Cook, and Advertising Research Foundation’s Taddy Hall ruminate on this mistake:

Why do so many marketers try to understand the consumer rather than the job? One reason may be purely historical: In some of the markets in which the tools of modern market research were formulated and tested, such as feminine hygiene or baby care, the job was so closely aligned with the customer demographic that if you understood the customer, you would also understand the job. This coincidence is rare, however. All too frequently, marketers' focus on the customer causes them to target phantom needs.

Rather than these “phantom needs,” when gathering feedback stay focused on the real issue at hand: The job that your customers are trying to accomplish, and the obstacles that stand in their way.

You'll take feedback at face value

In his book Competing Against Luck: The Story of Innovation and Customer Choice, Clayton Christensen recounts a story that perfectly illustrates successes you can achieve when you stop taking feedback at face value and instead dig a little deeper.

He tells the story of Bob Moesta, a Detroit-area builder tasked with the job of boosting sales for newly constructed condominiums. The company had targeted downsizers, and had provided a list of over 30 customizations to choose from, but sales continued to lag. They dutifully acted on feedback gathered in focus groups -- adding bay windows, for example -- but nothing was working.

So Moesta decided to do some detective work. He approached people who had already bought a condo to figure out the job that those people were hiring the condo to do. In other words: Why did people buy these condos? What would stop them from doing so? What obstacles stood in the way of purchase? He found out something surprising: People expressed intense concern over what to do with their dining room tables in the new smaller space. This, more than any other factor, seemed to be holding people back.

If Moesta had taken this feedback at face value, he might have continued to make larger and larger dining rooms to accommodate the larger tables. But that wouldn’t have resulted in more sales because it wasn’t the real underlying issue. Moesta realized that people were experiencing anxiety about giving up their belongings, and so the issue wasn’t about a table per se -- it was was about the memories associated with the table, as well as the emotions attached to the personal objects they had to sort through in order to downsize.

Once Moesta understood the real struggles people were facing he was able to innovate solutions to their problems. As well as making sure the units’ dining rooms could accommodate a large table, he found solutions for their concerns and anxieties: supplying moving services, two years of on-premise storage, and a sorting space. The result? In 2007, when the real estate market was plummeting around them, the Detroit condo sales grew by 25 percent.

You won't know when or how to supplement your observations with feedback

Sometimes, even if you’ve been observing and recording all of your users’ behaviors, you need to supplement your observations with feedback to fully understand what you’ve seen. As always, the correct approach involves thinking critically and not putting the solutions in the hands of your customers.

Take the case of a maps feature Intercom included in its product. This map feature was intended to allow Intercom's users to see where they were acquiring their own customers and track geographic trends. Intercom was surprised when they discovered that users were sharing the map on social media and in presentations with clients. So they asked users why and how they were doing this, and discovered something surprising: The map had become a sort of bragging tool to show off a global customer base.

Instead of asking users how to improve this feature, they took this opportunity to create ways to improve the customer experience and remove existing struggles standing in the way of what the customer wanted to accomplish. They made the map more visually appealing, implemented a way to hide commercially sensitive information, and made sharing easier. Simply put, they improved the feature to make it easier for people to accomplish their goals.

The lesson in all this? The true purpose of gathering feedback is often misunderstood. If you ask the wrong questions and fail to understand your customers’ goals and obstacles, you’ll end up making small changes that don’t really impact the customer experience. And, like Kawasaki, you may even find yourself left in the dust, lapped by companies who weren’t afraid to dig deeper.