UX

Don’t Listen to Users and 4 Other Myths About Usability Testing

I’ve conducted around 50 usability studies at Icons8. While I was learning my craft I read many articles on the topic. What I found is that some practices are blindly borrowed from laboratory research while others are heavily misinterpreted. And some are pure dehumanizing.

I’d like to address some of the modern usability myths in this article in comparison to my own experiences.

Myth #1 Do Not Listen to Users

Whoever said that was… Jakob Nielsen, one of the pioneers of usability research. Thus the first rule of usability was born. Of course, getting feedback based on real interaction with a product is paramount, but that doesn’t mean usability specialists shouldn’t listen to people at all. The question is, when? Sure enough, this question is answered in the very same article where this first rule was minted.

“When should you collect preference data from users? Only after they have used a design and have a real feeling for how well it supports them.”

What can you get by listening to people that is hidden while they are doing tasks? When I was studying our request icon service:

  • people perfectly coped with all the tasks
  • people liked the new, modern design

Everything was fine, but one thing. After finishing the tasks, they asked “what’s this whole interface about?” Now, after I heard more or less similar questions from several attendees I got goosebumps. Thus, the infamous 47% story was born.

Summary: listen to people after they are done with tasks. There are at least 47 reasons to do that.

Myth #2 A Hundred Opinions are Better Than Five

Whoever said that 100 people are better than 5 has never been stuck on the subway. To rephrase, the myth is that “automated large-scale tests are better than live interviews.” Automated tests are a good tool, but they are not superior.

First, it takes real skill and practice to interpret big chunks of data in the right way.

Suppose I have an egg farm. If I got a report that 10% of our eggs are cracked, what should I do?

a) raise a number of chickens to cover the shortage
b) focus on the safety of existing chickens to reduce losses
c) fire my cousin

Big data makes us very confident but doesn’t save us from ambiguous interpretations. In personal interviews just one participant is enough to shatter my inner world, leaving me in constant doubt about everything I thought I knew, and, eventually, make me more objective.

I’m not encouraging you to interview every chicken to solve the farm problem (3-4 would be enough). I was making a point – automated test results can be interpreted in a more ambiguous way than results from direct interviews.

Second, one-on-one interviews give so much unique information if you really listen to people.

Summary: Both automated tests and live interviews serve a different purpose and are not superior to each other, but complementary.

Myth #3 Do Not Change the Script

If you tear your notes up 2 minutes before a presentation it’s called courage. If you edit  the script while shooting a movie it is called a unique vision. But if you change the script during the usability interview it’s called “WTF is he doing?”

Why are UX people so afraid of changing the script along the way? Because the results will not be objective, they say. You need 5 people to do exactly the same thing to draw some reasonable conclusions about the interface being tested.

While I agree on getting the most objective feedback possible, I disagree that words in your script should be carved in stone. In one of my studies, I watched how people devotedly ignored one button they all saw from the very start. If I hadn’t come up with an additional question, “Did you notice this button before the task?”, I wouldn’t have ever understood the real reason behind such strange behavior.

Another argument for having a rigid script is that it adds comfort to everyone – a participant feels like an interviewer knows what they are doing, and the interviewer feels like they know what they are doing. What can I say? Genuine information is more important than comfort for UX professionals.

Summary: Having a script is very handy, but don’t be afraid of adding to it on the way, just make sure it doesn’t disrupt the overall flow of the research.

Myth #4 Don’t Talk to Participants

A typical laboratory setup is a chair, a table, and a participant sitting at the computer, doing tasks. Cameras and sensors tracking everything – eye movements, facial expressions, body language. And no one around. The goal is to exclude the experimenter’s influence on the participant, leaving them one on one with a product being tested.

For some reason, several authors suggest creating this very lab atmosphere every time, even in more informal Skype interviews. “Don’t talk with users, listen.” While I agree that researchers should listen more than talk, they don’t have to treat their participants like Wolverine in his early lab days.

Even if you are silent, in another room or whatnot, people know they are being watched. You don’t even have to do anything, the experimenter role is your default one. And when you play the role of an experimenter, these are some possible effects:

  • participants see you as a figure of authority and they try to please you with their answers and actions, thus bending the truth
  • they fear admitting mistakes in UI because they are afraid to look silly in your eyes
  • they adopt a behavior model that is different from when they really use your product (at the office, at home)

So take a more friendly role. I always liked making smalltalk with attendees. A few jokes (better funny), a few side questions, a relaxed tone, and people are more willing to share everything that’s on their mind. So you get valuable information about the product, things that might otherwise be outside of a participant’s comfort zone.

Summary: Don’t be too talkative, but strive to create an atmosphere that is as authentic as possible.

Myth #5 Common Sense Isn’t Enough

Usability is a topic everybody has an opinion about. Everyone from your boss to your unemployed cousin. That doesn’t mean there are no real usability professionals. There are quite a few things usability professionals may be familiar with:

  • social psychology
  • behavioral psychology
  • design basics
  • communication skills
  • data management
  • a lot of practical experience

Such a specialist is quite costly, thus many companies take another approach. They simply assign this role to someone. And if that someone has common sense, it pays off.

If you can’t have an experienced usability specialist in your pocket, but desperately want to test your product, why wouldn’t you take the same approach and become that someone?

A foundational  design principle is that 80% of people will use 20% of the product, it’s “core.” Define the core and come up with a few tasks affecting it. Then watch how people do them. Never give hints, just wait patiently until they ask for your help. Thus you’ll see the real problems and benefit from the research much more than by simply adhering to someone’s subjective opinion.

Summary: Experienced professionals will benefit everyone, but having someone with common sense is a good start.

Conclusion

Usability myths I addressed in this article are more or less directed towards eliminating the “human factor,” to make all usability studies and services as impersonal as possible. From one point of view, that’s correct – in real life, there is no experiment between a product and a person using it. Imagine every time you used a new shower someone came up with a notepad and asked how the water was.

But with trying to “hide” the experimenter from the experience, the opposite happens – people behave in a way they would never behave in real life. The best way to eliminate the experimenter is to observe the “user” in their natural environment, a full-scale surveillance program, and I doubt the usability industry would be the first ones to get their hands on it.

So, what’s left? Creating a friendly, trusting atmosphere during research. Not eliminating the human element, but adding to it. Don’t pretend as if you’re not there.  At least, if we’re talking about a non-laboratory environment, like Skype sessions. But who knows how the same approach would work in a laboratory as well?

By the way, here’s in one of my very first usability studies: Drag and Drop vs. Click – Are They Rivals?

I’m not the only one who uses uncommon approaches in his work – meet Ivan, founder of Icons8, and his Blind Method approach in hiring new people.


About the author:
Andrew started at Icons8 as a usability specialist, conducting interviews and usability surveys. He desperately wanted to share his findings with our professional community and started writing insightful and funny (sometimes both) stories for our blog.

Recent Posts

Icons that are perfectly shaped for any design

Each design has its own scale. It applies to elements size and like thickness. Now,…

2 hours ago

What is a glyph icon?

There are many ways to define a glyph. Let's take a look at what glyph…

24 hours ago

Hues unraveled: exploring lavender in design

Learn how to use lavender color in your designs. (more…)

2 days ago

6 websites that demonstrate the power of surprise

Keeping visitors on your page with surprising, creative tactics can help you convert them into…

1 week ago

Search for an icon or generate one? SVG icon AI-generators test

As designers, we don't need just any icon. We need one that is a perfect…

1 week ago

The ultimate graphics plugin for your Miro boards

This Miro plugin has it all: icons, illustrations, and photos from the Icons8 library to…

3 weeks ago

This website uses cookies.