Why User Testing Sucks

Ever wonder why we have such an astonishing amount of articles telling us: “User testing is important”, “You can’t release products without user testing”, and “Do user tests or die”?

I mean, if it’s such an important and obvious concept, why do we need so much persuasion?

You could think it’s about money. As if startups can’t afford to do user testing the right way:

Quite easy to-follow:

1. Find five potential customers from a target audience
2. Produce a script with realistic tasks
3. Be quiet

Doesn’t look that expensive.

So it’s not about money. Maybe it’s about laziness? I’ve always suspected entrepreneurs are the laziest people. Especially in the first few years when they work 63% longer than the average 40-hour worker.

So why not do user testing? Why do most people still prefer to listen to their gut?

Here’s my observation: user testing does not always work the right way. Why? Ironically, because it involves users. There’s an inherited flaw in user testing, the inner contradiction: we’re trying so hard to understand human nature, yet at the same time we’re trying to eliminate it as much as possible.

Drama intensifies…

We want to capture a sincere reaction of users to our products, meanwhile putting so much work into creating artificial scenarios and environments that have nothing to do with the real world.

More drama…

We’re trying to deeply understand what drives our customers. Instead, we end up creating generalized personas that have less in common with the real people using our products than “Married… With Children” does with our neighbors.

Cue in Hans Zimmer’s “Time” …

We live by faith that every user’s experience is precious, yet we gather as much data and analytics as possible to minimize a single person’s impact.

Ok, enough drama. Without making this post a philosophical rant, I’d like to address some grey areas of user testing. I’ve been doing user testing at Icons8 during the last few years. [1,2,3] This article tells what I don’t like about it.

Bias

Anytime people are involved, there are biases. In user testing, people are on both sides of the equation: users and user testers.

First bias comes from user testers themselves. Observing people, we may feel like we understand other people. It gives us the confidence to interpret the results of user testing any way we want. In order to prevent that, we try to be as impartial as possible. We try to suppress our own point of view. We’re practicing empathy. So now we’re totally dependent on the point of view of our users. But here comes the problem…

Observer-expectancy effect

No matter how realistic testing scenarios and scripts are, users behave differently when they know they are being watched. This never lets us get the real picture of how people interact with our products. Now we’re dependent on the altered behavior of our users. We’re dependent on someone else’s bias.

What do we do when there are so many biases involved on both sides? We trust our gut to make the right decision.

Of course, there’s another way. And it’s getting more popular by the year.

Ambiguous data interpretation

We try to observe people in their natural environment. There’s only one way to do that: putting all kinds of trackers into our software and then building behavior maps. Now we see exactly how our users interact with our product.

Problem is, every user is a collection of digits now, a dot on the graph. There’s no user there anymore, we’re watching numbers.

Now numbers make us confident that we know something. They give us the confidence to interpret the results of user testing any way we want.

I’ll use this example from my old article:

Suppose I have an egg farm. If I got a report that 10% of our eggs are cracked, what should I do?

a) raise a number of chickens to cover the shortage
b) focus on the safety of existing chickens to reduce losses
c) fire my cousin

Big data makes us very confident but doesn’t save us from ambiguous interpretations.

To interpret numbers objectively, we need to suppress our own point of view, again. Evoke empathy. Only there’s a problem now. There’s no one to be empathized with anymore. There are just numbers. And generalized personas.

No matter how much objective data we’ve gathered, there’s always going to be some kind of subjective interpretation by someone. A business owner, a user tester, an analyst – and trust me, the gut feeling will be involved in every case.

Summary: user testing will allow you to see the problems, but it won’t guarantee you’ll make the right choices. Kill the hype.

Explained… After.

Gee, don’t we like to explain things…

Imagine you’re watching a movie with a friend. Halfway through the movie, a major, unexpected plot twist occurs. “She killed him”. Your eyes are rolling in awe, you’re breathing hard, your mental condition is similar to that bemusement of an old lady in a supermarket who’s being told that her coupon is out of date.

Excited, you slowly turn your head to the left, where your friend sits, anticipating him or her to be in a similar shocked state to the one that you’re vividly experiencing. However, that is not what you are seeing. Your friend is laid back, channeling the detachment of a surfer on a pension. Their lips are slowly moving and after a short delay you hear: “I knew it was coming from a mile”.

Bewildered, you go on Youtube only too see ten more explanatory videos about why the twist wasn’t a twist at all and a hundred more comments conveying the same notion.

Suddenly, you feel like you’re the only person in the world who didn’t see anything coming.

You’re probably thinking now: what in the … does this have to do with user testing?

Well, in the mentioned scenario, you are you. The movie is your web service or an app. And your friend, Youtube analysts and countless commenters are all user-testers.

You see after something happens in a movie [read: with your service], somehow there are always people around saying: “We saw that coming from a mile”. However, they always do that after the movie ends.

In user testing, everyone likes to explain everything. After the testing phase, someone needs to explain and make sense of all that’s happened. People, after all, are paid to do that. At the same time, explaining things is something our brain likes so much. It gives us comfort.

This is why user testing sucks.

Everything is explained only after everything has already happened. Why we lost users? Because we redesigned the website. Because we added this one button. Because our competitor made a new product. Because our website was down for 12 minutes.

Bring five more people in, show them the numbers. They all will have explanations. Just different ones.

Anyone who has the number speaks the truth now, but their own truth. This is why there are so many stories of successful entrepreneurs who seem to know everything about their success. Explaining success is different from attaining it. And this is the exact same reason why those same stories haven’t help everyone to achieve the same outcome. Explanations are just that… explanations. A chilling pill for the brain.

Why did Twitter become successful? Pretty sure all the articles are dated post-2006.

Summary: user testing spawns a plethora of explanations. Go figure out the right one.

The Goals of User Testing Are Conflicting

The unified goal of user testing is artificial. Something along the lines: “make our products easy to use”, “improve user experience” and so on. User testers have their goals. Business owners have theirs. Worse yet, those goals are constantly changing thanks to trends, technical breakthroughs or weather. What can you expect from a field that was popularized by just one company?

After the iPhone release, everyone became obsessed with simplicity. After Kindle, with ubiquity. Each popular product spawns new rules, new explanations of what UX is. UX has to be an organic experience, so let’s make our buttons imitating real objects. Wait, no. Let’s make them simple and get rid of all the extra details.

So the new rules are the rules now. Simple, flat, mobile-first. Trends replaced goals. Yet we have to ignore that services like LinkedIn, Facebook, and Skype disrupt most of the rules we’ve established. And yet, they’re still popular. Simple and UX’ed Slack didn’t kill Skype. It killed simple IRC though.

What we actually do with all the rules and explanations is trying to replicate the success of products released before us.

Then comes another big product that totally undermines everything that has been stated before. Just like the iPhone did. And it becomes the next big thing. And we’re going back to new explanations and theories again.

What’s the real goal behind UX? To make users happy? To make them productive? To make businesses more money? Go figure.

Summary: user testing involves a bunch of people with different agendas, making it almost impossible to find a solution that works for everyone. At the same time, UX trends can be more volatile than Bitcoin in 2017.

Why Do People Choose Trusting Their Gut?

After everything I’ve mentioned, this is what someone conducting user testing is supposed to do:

• Eliminate all kinds of inherited human bias between themselves, users, stakeholders, analysts, and business owners. The bias that we’ve had for thousands of years. At the same time make sure you’re still a living, breathing human.
• Disregard any kind of explanation as most of them seem to be a subjective interpretation of something that has already happened. In other words, an opinion. Listen to this song.
• Find a compromise in a bowl of contradicting goals: following current superficial best UX practices, making users happy, making businesses happy, changing the world, whatever. Don’t lose your job.

The whole structured process of simply finding five potential customers and giving them tasks gives a false sense of things working properly. The structure is there, and it gives false comfort. Yes, you will unearth problems, but solutions. There’s no guarantee that your solutions will be right. Too many things go out of hand. Too many explanations will be wrong.

Despite so many resources on the importance of user testing…
There is, after all, a reason, beyond money or laziness, why people sometimes prefer to trust their gut.

“In many ways, the introduction of Amazon Prime was an act of faith. The company had little concrete idea of how the program would affect orders or customers’ likelihood to shop in other categories beyond media…
…But Bezos was going on gut and experience.”

-Brad Stone, The Everything Store: Jeff Bezos and the Age of Amazon

In April 2018, Amazon finally revealed that there are over 100 million members using the service.

About the author: Andrew started at Icons8 as a usability specialist, conducting interviews and usability surveys. He desperately wanted to share his findings with our professional community and started writing insightful and funny (sometimes both) stories for our blog.

Title image created with Icons8 Photo Creator