What kinds of questions should a test administrator ask of a usability test participant gets stuck?

In the field of User Experience and Product Development, research and planning rule the show. Neglecting either of them will backfire on you and your organization with high user abandonment, increased need (and cost) for customer service, increased cost and time for development, and all in all—a complete waste of available resources.

But what can you do when deadlines are near and the budget is low? You can do Guerrilla Usability Testing, a lean and agile approach to testing which doesn’t break the bank.

In 7 simple steps, we will show you how to conduct Guerrilla Usability Testing to get the maximum out of it:

• #1: Come up with a list of tasks
• #2: Prioritize tasks
• #3: Turn your tasks into scenarios
• #4: Start guerilla testing
• #5: Capture testing insights
• #6: Fix your usability problems
• #7: Test again, validate, make it a habit

But first let’s answer the 3 most common questions about guerrilla usability testing:

How does guerrilla testing work?

The main goal of guerrilla testing is to find errors and fix them as quickly as possible. Guerrilla testing is easy to set up because you don’t need to worry about:

• You don’t need expensive recording equipment or a research lab
• You don’t neet to get participants to come to you (which includes travel costs, arrangements, schedules and often cancellations)
• You don’t need to deal with paperwork and administration
• You don’t need to worry much about recruiting the right participants with their demographics matching your target audience.

The right people for guerrilla testing are the people available right now. This makes Guerrilla Testing extremely lean and agile. You can get insight in as little as a couple of hours. The sessions are very short (about 15 minutes) and compressed.

Here’s what’s happening in a guerilla usability test:

1. You approach a person,
2. Ask them if they would like to answer a few questions about your product,
3. Give them a couple of tasks to do,
4. Observe their interaction,
5. Ask about their experience,
6. And you’re done.

Because you’re collecting qualitative data during the tests, you just need 3 to 5 people to spot the biggest usability issues. Discovered usability issues can be improved right away and the improved design is validated during the next round of testing.

Guerrilla research, therefore, is the perfect starter drug for stakeholders who struggle to acknowledge the value of usability testing.

It’s a lot easier for them to accept to spend a few hours and almost no money on a guerrilla testing than to spend a fortune on a full bloated usability study which could take months in planning and execution. Especially when they know the team is on a short deadline and a limited budget. Some insight is better than no insight. So even if the research you conduct is quick and simple, it’s very well worth it and much better than no research.

Where do I find the right participants for guerrilla user testing?

Compared to a traditional usability study, participants for guerrilla tests are not recruited, you just approach them. The easiest way to find participants is to use your family or friends as test participants—after all, they can’t escape that easy because you know where they live. 😉

The thing is, your close friends and family often know too much about you and your product, it’s difficult to get a true first-time user experience from them. Also, your mum or your elderly aunt Shelly may be extra nice and careful with your emotions… For that reason, it’s often better to test with new participants without any knowledge about you and your product. Perfect places to find such people are locations where people are naturally in a good mood like coffee shops or shopping centers.

Pro Tip: Make sure to pick a location with a stable Wi-Fi connection if your prototype or website is online. You shouldn’t forget your charging cables or additional power packs as well. You don’t want to run out of battery during a test.

When should I start with guerrilla testing?

We did hundreds of usability tests for both our businesses (Userbrain & Simplease) and have not yet come across a single one where there wasn’t at least one minor problem found. That’s why you should really test as early and often as possible because fixing problems will get more and more expensive the more advanced your product is as this chart by Raygun shows:

Remember: You don’t need a finished product to experience the value of user feedback. If you wait to test just before launch it will be too late to fix anything.

Step 1: Come up with a list of tasks

The first thing is to write down a list of all the important tasks people need to be able to do on your site. If people come to your site to buy something, you should have a task that asks testers to purchase a product. If you want people to create an account, you just ask them to sign up to your site. And so on …

For example, Facebook’s actions would be:

• Scroll through new posts
• Update your status
• Send a private message
• Upload a photo
• Add a friend
• Change your password

Now try to do it yourself. Think about all the important things people do on your website and write down a shortlist of tasks. And don’t waste your time on details, just get down your list. We’ll deal with the rest in the next steps. If you’re stuck, here’s some help:

• How to write better tasks to improve your usability testing
• 6 Usability tasks you haven’t tried so far, but really should!
• 3 things you shouldn’t ask in a usability test

Step 2: Prioritize tasks and decide which to test

When you have your list, it’s time to prioritize all items by giving points from 1 to 3 based on how frequently the tasks are performed.

Add:

• +3 points to tasks that most users will do most of the time
• +2 points if they do it occasionally
• +1 point if they only perform this task every once in a while
  So, here’s our list for Facebook, prioritized:
• Scroll through new posts: 3
• Update your status: 2
• Send a private message: 1
• Upload a photo: 2
• Add a friend: 1
• Change your password: 1

Don’t spend too much time on this ranking; just make sure you find the tasks most people do most of the time. Why? Because you can’t test everything at once (not in guerrilla usability tests). And if you focus your first tests on finding and fixing problems of the most important parts, you will, of course, get the highest return on investment.

Alright, here are our 3 most important tasks again:

• Task 1: Scroll through new posts
• Task 2: Update your status
• Task 3: Upload a photo

Again, guerrilla usability testing is about quick results and acting on them immediately. Just select the 3 most important tasks and get going. You can test the rest later.

Step 3: Turn your tasks into scenarios

Now it’s time to turn your tasks into a scenario that your users can read, understand, and follow.

Here’s a short list of rules to write useful task scenarios:

• Don’t give clues in the scenario. You have to phrase your scenario so that it’s clear and easy to understand, but you have to do it without using uncommon or unique words that appear on the screen. Otherwise, you turn your test into a simple word-finding game.
• Make your scenario easy to follow. Write the way you talk and avoid scientific or academic language whenever you can. Pre-test your scenarios with friends and colleagues and make sure they are easy to understand and people can follow them without any confusion.
• Trim any detail that doesn’t contribute. Your scenario provides your users with context (“You are …”, “You want to …”) and gives them all necessary details (e.g. username and password for the test). Everything else is unnecessary and you should keep your scenarios as short as possible.

So let’s try this with our 3 example tasks:

Task 1: Scroll through new posts

We can’t just tell users to “scroll through new posts” without any motivation or goal. Instead of describing what to do, we can provide people with a reason to do it.

Bad: Scroll through the page to look at new posts.

This will only get your users to perform the task like a robot. Without any motivation whatsoever, they will just scroll through your page and you won’t learn anything about how to improve your usability.

Okay: Look at this page and find out what it’s all about.

This is at least open and lets people explore the page more naturally. Just by telling them to look at the page, your chances are high that they will be scrolling through the posts to find out what it’s all about.

Good: Imagine this is the first time you’re checking Facebook today. Now go and find the first post that was published today.

Finally, this works because it gives your users a real goal. Instead of just telling them what to do (bad) or hoping they will do it (okay), you can give them something specific to do that will naturally motivate them to use your site (good).

Task 2: Update your status

For the next task, you need to motivate people to enter actual data (not just scroll through). Let’s look at different ways of doing it:

Bad: Write a status update and post it to your profile page.

This gives too many clues on how to use the site. People will simply look for the words “status update” and maybe a button labeled “post to profile page”. And again, the scenario doesn’t offer any motivation or reason why you would want to update your status.

Okay: Let your friends know what you’re doing by updating your status.

This explains why someone would want to update his or her status update. But it’s still not as good as it could be. And the fact that “updating your status” is still there makes it a simple word-finding game again.

Good: Find a way to let your friends know what you’re doing and tell them you’re currently testing a website.

This works because it avoids any clues about how to use the interface. Again, it gives users a specific goal (find a way to let your friends know …) and even supplies them with the information they can use. This helps reduce the mental effort often caused by thinking about what kind of data you’re supposed to fill in during a test.

Task 3: Upload a photo

This one’s a bit tricky because we definitely want to avoid the words “upload” and “photo” (because they are on the screen) and yet we have to somehow tell our testers to upload photos …

Bad: Find a way to upload a photo.

This is too obvious to even talk about it …

Okay: Find a way to share some pictures with your friends.

This one is actually pretty good. The motivation part is still somehow missing, but the idea of sharing pictures with friends is a nice way of framing the action of uploading pictures to make sense to most people.

Good: Last night you were at a party and took some funny pictures and now you’re looking for a way to share them so your friends can see them as well.

This provides all the necessary context to understand the situation and the goal you’re trying to achieve as a user. Notice how every detail makes sense and contributes to the overall understanding. Like the fact that the pictures are funny, which will cause some users to take extra care of privacy settings (the word “funny” in the context of partying with friends is not necessarily something most of us want to be public).

Pro Tip: As long as you get your scenario right, your testing will always work. And making good scenarios will help you test as if you had your target audience.

Step 4: Combine all 3 scenarios

OK! We just turned our most important tasks into scenarios and now it’s time to combine them:

• Imagine this is the first time you’re checking Facebook today. Now go and find the first post that was published today.

• Find a way to let your friends know what you’re doing and tell them you’re currently testing a website.

1. Last night you were at a party and took some funny pictures and now you’re looking for a way to share them so your friends can see them as well.

Well, that was easy. Now all you need is to start testing. But first read on and learn how to do it:

Step 5: Start guerrilla testing

As mentioned in the beginning, the secret of guerrilla usability testing is to approach people and ask them to test your site, app or prototype. There’s no need to care about who you’re testing with, no need to worry about how to record the sessions, and thus you don’t need written permissions or anything else to get started. Just walk over to someone and ask for a few minutes of this person’s time.

For better results, you can even offer a little compensation and thank you for their engagement, like free coffee or muffins. And once you’re comfortable enough to approach people like this, you will figure out the rest as you go and get better and better after each session.

How to get people to participate

Here’s what you could say when you approach people:

“Hi! We’re working on a website (app, prototype, whatever it is) and want to see if it works as intended. That’s why we’re asking strangers to test this site for a few minutes and tell us what they think about it.

Would you like to do one of these tests? Before you answer, there’s absolutely nothing you can do wrong. Plus, you can enjoy a cup of coffee, muffin, whatever, at my expense as a thank you.”

Explain how the testing works

Since most people don’t know what usability testing is, you usually need some explaining before you can give the scenario:

“The first thing you have to know is that we’re testing the site, not you. So you don’t have to worry about making any mistakes. You’re supposed to point out OUR mistakes.

While you’re using the site, we would like you to think out loud. Just say what you’re looking at, how it affects you, what you expect to happen after every interaction, what you’re trying to accomplish, and so on.

And please don’t worry about our feelings. We want to improve the site and need to hear your honest reactions.”

And now comes the fun part. Just give them your scenario, lean back and observe how they use your website, app or prototype. And…

Remember to shut up

As in the scenario, you don’t want to give any clues about how the site is used. You don’t want to lead your users. You want to see how they figure out things for themselves; or how they don’t figure them out, which helps you learn even more! The best way to avoid leading your users is to remain silent during testing sessions. In real life, you wouldn’t sit next to them while they use your product. And you want the testing session to be as real as possible. Your participants will, of course, have questions and since you sit next to them they will naturally direct them towards you. To avoid any bad feelings whenever they ask you something, just tell them something like this:

“That’s a good question. I don’t want to answer it right now, because we’re interested in how people use the site without anyone sitting next to them, but I’ll give you the answer it after the test. But now that you have this question, what would you do if I wasn’t here and you were using this site on your own?”

You will find your own ways of handling questions and everything else that comes up during the tests. The only time you should talk is to remind the user to think aloud. They often go silent, especially while trying to figure something out, and that’s when you have to give them a little nudge. Just keep in mind the goal is to have your user do most of the talking. Ideally 100% of the time.

Discuss questions

Once your tester is done with the scenario and tried all tasks, it’s time to address the questions that arose during the test and maybe ask some follow-up questions yourself. If you happen to test with people from your target audience, you may also want to turn it into a customer interview and ask people if they would actually use your product. But since this is no longer part of usability testing, we just move on to the next step.

Step 6: Capture guerrilla testing insights

Don’t try to capture your insights during the test. Yes, that’s right. Because if you sit next to your participant and you’re writing all the time, you screw up the experience. People will feel stressed and wonder what you’re writing and if it’s good or bad or if they’re doing anything wrong … And during the tests you should only focus on observing the user and trying to decipher the subtle nuances of their behavior, their words and the hidden meaning behind them. Yes, the user is thinking aloud, but people are very bad at getting their subconscious thoughts into the conscious and verbalizing them. That’s why you need to always ask yourself: “Is there something more behind the thoughts they are expressing?” Your mind has to be set on observing (not writing), and you will see that you can remember everything that was important. And even if you forget something, you will do another test (as you’ll see in step 7) and if it’s a real problem it will come up again.

So how do you capture your testing insights?

Method #1: List top 3 usability problems

The most obvious way of capturing your findings is to simply list the top 3 usability problems identified in the tests:

There’s no need to write down everything you find. Guerrilla testing is about finding and fixing the most severe problems, not agonizing over every possible obstacle a user might encounter. And if you want to learn how to discover usability problems, watch this example video of Steve Krug explaining his approach (which is kind of like the blueprint for this post).

Method #2: Capture task completion

A more sophisticated way of documenting your findings, is to capture the task completion for each of your participants. This is particularly useful if you need something to present to clients and other stakeholders. You can use the following designations:

• If a user can perform the task quickly and with no trouble, mark it a 3.
• If a user can perform the task but has some problems, mark it a 2.
• If a user couldn’t perform a task, mark it a 1.

This is what your result may look like:

FREE TASK COMPLETION TEMPLATE:Download the free task completion template we use for our own guerilla user tests.

Step 7: Fix your usability problems

As mentioned at the beginning of this article, fixing problems can be expensive. Especially if the issues affect running applications and live websites. That’s why it’s wise to build prototypes and test as early and often as possible to fix major issues before you even write a line of code.

But what if you’re already past this prototyping phase and the problems you’ve discovered through guerrilla testing cost your company hundreds of development hours and thousands of dollars to be fixed?

Fix the biggest problems first

“If it’s your job to eat a frog, it’s best to do it first thing in the morning. And If it’s your job to eat two frogs, it’s best to eat the biggest one first.” — Mark Twain

The attentive reader will notice a pattern:

1. First, we test only the most important tasks.
2. Then we write down only the most important problems.
3. Now we fix the biggest problems first.

And we do it quickly …

Implement quick fixes

“When fixing problems, try to do the least you can do.” ― Steve Krug, Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems

Always ask yourself what’s the smallest, simplest change you can make that’s likely to keep people from having the problems you’ve observed. That’s especially true for live websites, and yet many people resist this idea.

In his book, Rocket Surgery Made Easy, Steve Krug makes the point that you don’t have to find a permanent solution for a problem. Just find an easy to implement quick fix that’s likely to solve the problem and do some more testing to see if it already works.

Tweak, don’t redesign

After you’ve done some usability testing and start thinking about how to solve a problem, it’s always tempting to make changes that go beyond the problems you actually observed. Sooner or later you’ll find yourself at a point where you want to redesign whole parts of your site.

This bias towards redesigning has many reasons, and it’s easy to understand why many people like to start over to get it right this time instead of just implementing a quick fix. We’re guilty of this ourselves, but hopefully, this list (again from Steve Krug) helps us remember why it’s better to tweak than to redesign:

1. Tweaks cost less.
2. Tweaks require less work.
3. Small changes can be made sooner.
4. Small changes are more likely to actually happen.
5. Most people don’t like change, so a redesign annoys them.
6. Tweaks don’t ruin lives, break up families, and wreck careers.
7. A redesign means involving a lot of people in a lot of meetings. Enough said.
8. A redesign means making a lot of changes at once, with the attendant complexities and risks.
9. If you make larger changes, you’re more likely to break other things that are working fine in the process.

If you’re still not convinced about quick fixes and how they are better than whole redesigns, continue with the next (and final) step of guerrilla usability testing. Everything will make sense to you afterward…

Test again and make testing a habit

Usability testing works best if performed on an ongoing basis. Instead of spending your time and money on elaborate usability studies to inform a big redesign, you can just implement quick fixes (step 6). The trick is to keep testing to make sure your solutions work.

How often should you test

There’s some interesting discussion about how many users you need in a usability test. A popular rule of thumb is to test with 5 users to discover the most severe problems of any application. But as always, it depends… If one user stumbles upon a major usability bug after 30 seconds, you don’t need 4 more tests to satisfy the scientific method. In the end, this is NOT rocket science. Sometimes it’s just plain obvious. Even after watching only one user. The only thing that matters is to retest on a regular basis (weekly if possible). This way you can see if your tweaks really solve the problem, and you make sure to spot any new problems that may be caused by your changes.

Make testing a part of your workflow

The reason why guerrilla testing is oftentimes better than traditional usability testing is that you can afford to do it on a regular basis, and I’m not only talking about money. Guerrilla usability testing is fast and easy and cheap, it’s effective as hell, and there’s no excuse not to do it. At the end of the day, guerrilla usability testing comes in many forms. Consider making up your own approach as you go: learn by doing.

FREE CHEAT SHEET:Download a step-by-step check list to perform guerilla usability testing on your website.

How others do it

Here’s how others are Guerrilla testing:

• Airbnb Guerrilla Usability Testing
• A Guerilla Usability Test on Dropbox Photos
• Guerrilla Usability Test: Yelp
• A Guerrilla Usability Test on Apple’s Shared Photo Stream Feature
• Kayak Guerrilla Usability Study
• Examining SoundCloud’s Mobile Redesign
• Changing the Guardian through guerrilla usability testing

But if you don’t have time for testing, because an hour of your time costs 50+$, you can resort to remote usability testing. Remote testing removes many of the challenges related to scheduling, travel, paperwork, and setup. You can literally shave off days, or in some cases weeks, with remote testing.Userbrain is one of the cheapest remote usability tools on the market and works as a subscription. This means you get 1 or 3 weekly tests, or 1 test monthly.

Which of the following is the most important difference between observing users in a usability evaluation and observing them in a usability test?

What type of sentence states summarizes or forecasts?

Which of these statements does not identify an advantage of turning a paragraph into a list?

What is meant by the claim that technical communication is a threshold skill?