A step-by-step guide to User Research

Matthias from Just Ask Users is so excited about The UX Blog that he created a special one-time free offer for you, our valued readers: If you haven't used Just Ask Users before, you can sign up for free and get your first 6 participants for your UX research there – totally free of recruiting feesClick here to enter your first recruiting order and look forward to meet real users for your product.


Your user research team knows: Today, product users have tremendous power. The user decides whether a product is fun, useful, and easy to use – in other words whether or not it will be successful.

Do you know how a typical user would react to your new product or the awesome new feature that you are building? Find it out doing user research! This post shows you step by step how to approach user research from preparation to complete user feedback.

What is User Research?

In user research, you interview prospective and actual users of your products about how what they see, hear, think and feel when they do their daily jobs or when they use your product.

You might use it to figure out why people aren’t adding recommended products to a cart, why they’re not clicking through your emails on mobile phone, or why adoption of your application has fallen significantly.

You may also be developing a new product or redesigning a website and know you want to do it differently this time around. All of these are great reasons to let your users tell you how they feel about your product.

Users today have tremendous power. Due to the omnipresence of the Internet and the various social networks, users are well-informed before they decide to purchase your product. And while they evaluate new products or services, they tell their friends and networks about great or subpar experience.

You can use this immense power through proactive user research early in the project (and also repeatedly during product development and maintenance). This will avoid potential disasters later.

Clarify your research purpose

Before you go ahead and dive deeply right into the process, you should clarify your goals and objectives so that your stakeholders, sponsors and team members understand the direction in which the project is going.

Purpose

What is it that you are trying to achieve with your user research?

  • Are you in early evaluation of a problem space or market?
  • Are you already generating concepts?
  • Or are you testing and evaluating user reactions?

Depending on where you stand, you will want to choose from a variety of research methods to select those that will maximise the learnings that you get from your research effort.

Quality or quantity

test with 5 users

Are you interested in gathering insights (e.g. “How can we improve the onboarding process?”), use a qualitative approach. Watch few participants and see with your own eyes how they use your product. You will see patterns emerge with about four to five participants (see Nielsen’s classic article Why You Only Need to Test with 5 Users).

When you need a more detailed answer (e.g. “How successful are customers buying a certain dress using a suggested, new product page?”), use a quantitative approach. You’ll have hundreds or more participants and rather than seeing what they do with your own eyes, you’ll get the numbers that will tell you what happened (or not).

To sum it all up: A quantitative approach usually explains what happened while a qualitative approach tells you why it happened. To get the full picture, use both approaches and combine their results.

Define target audience and methods

Collect what you need before you embark on your journey into user research to make sure you’ve got everything with you.

Target audience

From your research, you want to get usable data to make decisions for your product. This data comes from the people you need to know about. It could be people who are currently using your product, it could be people who are considering using it. If you have a website, it could be the visitors of your website who purchased something (or abandoned the shopping cart as well). It could just as likely be people who use a competitor’s product.

Figure out what it is that you are trying to learn and then who are the best people you will learn this from.

Come up with a profile of those people:

  • Basic demographics: age, education, income level, gender
  • Everyday life, e.g. lifestyle, has teenage kids, commutes to work, uses a tablet on the bus, likes to cook, reads books on a Kindle
  • Domain knowledge: career, school or work experience, e.g. college student, travels for work, career in healthcare, graphic designer, stay-at-home parent
  • Tech-savvyness: knowledge of web, device, app, software, hardware, social media use, etc.

Other people on your team, your stakeholders, everyone who is going to use the results from this study – make sure they agree that you have built the right profile! Imagine you run a study and you present the results and suddenly somebody stands up and says “Well, but that’s not the people we are trying to sell to” or “I don’t think these people are important”. In that case, you will have a very hard time making a case that the results are still worthwhile and something that everybody on the team be paying attention to.

So, don’t build the profile by yourself but make sure it gets vetted by other people on your team and especially your stakeholders.

Type and format

Think about what you want to learn from the study. This will help you to choose the right type of study:

Methods

Depending on where you currently stand in your design process, you will want to choose from a variety of research methods to select those that will maximise the learnings that you get from your research effort.

Some examples:

Methods for early evaluation: Card sortingUser InterviewsFocus groupPhoto study

Methods for concept generation: Desirability TestingParticipatory DesignWorkshopPrototyping

And some for testing and evaluating: Usability TestingRemote Moderated ResearchWizard of Oz

Degree of presence

Do you need people from your target audience to be present in the room with you? Or is it sufficient to meet them over the Internet using a popular screen-sharing and audio-sharing tool like Skype? Do you want to see people work in their everyday context (in that case you would rather visit them and do a contextual inquiry).

Be clear about the degree of personal presence that you need from people.

Attitude or behaviour?

Do you want to learn something about people’s attitude (i.e. what they say) or about people’s behaviour (i.e. what they really do)? If it’s attitude only, you can use an online survey. If it’s more about behaviour, you choose to watch people while they complete an actual task.

Money and recruiting

Some things that are common about every project will also have an influence on what kind of study you will run:

  • How much money do you have to spend on it?
  • How much time do you have for the project itself?
  • Who is going to do the recruiting of participants? Are you going to do it yourself, is an admin doing it or can you actually hand it off to someone else, e.g. hire a recruiter or an automated system to take care of it?

Make a plan

Assemble your Team

When you’re putting together your UX team, focus on individuals who share your organization’s passion for creating a great UX. People who can work well in cross-functional teams and are open to feedback make great candidates.

Having dedicated people for each aspect of UX will help your team focus on what they’re great at while still understanding their relationship with the rest of the team and process.

Your finished team should look something like this:

  • UX team lead – Responsible for the entire team and delivering great user experiences. This person should have extensive UX experience, and be well versed in best practices, and comfortable championing the UX mission throughout the entire organization.
  • UX designer – Responsible for researching and designing the overall user experience, including conducting user testing, prototyping, field research and combing through analytics.
  • UI designer – Responsible for making sure the user can interact with and understand the product as intuitively as possible through the design and layout of visual elements.
  • Content strategist – Content is just as important as the visual design. This person makes sure the words a user reads are clear, easy to understand, and in line with your brand.
  • Developer – Responsible for bringing all your team’s great ideas to life with code.

Have everyone on the same page early on. Do not assume that a developer should “come in later” to save his/her capacity for current development projects. This is a classic mistake. Have a developer on the team right from the start – he or she will greatly benefit from seeing real users work on a task. If you don’t do this you will have to explain everything to the developers, later, which will cause a lot of misunderstandings.

Allocate Time slots

Plan time slots to meet the people in your target audience. Decide whether you need one person or a whole group in each time slot, depending on the format and methods being used in the study.

5 or 500 people?

Do you need 5 or 500 people? 5 people will be the right number for a qualitative, behavioural study. 500 people will be the right number for a quantitative study

For qualitative studies, how many do you have time to watch: consider the analysis effort attached to launching a study that generates too much video and more than you and your team can handle.

For quantitative studies, avoid tie situations: if you recruit too few participants, your confidence interval might be too wide, and results might not be statistically significant. In other words, you will have no way of knowing if your numbers are pointing in one direction or not.

Compensation

There is a lot of pro and con discussion going on around the subject of incentives.

On the pro side:

  • It’s nice
  • It’s a way to say “thank you”
  • It’s a way to attract people who otherwise might not actually sign up to be part of your study, so it might widen the pool of people who are willing to participate

Now, the cons:

Some researchers really value intrinsic motivation and are interested in people who are going to participate because they want to.

You need to be very careful in the U.S. that you are not paying them for their time with this incentive. Payment for time in the U.S. equates legally to employment! You are not employing them, so you don’t want them to ever think you have employed them.

So, be careful to say “As a thank you in return for talking with us, we can give you…”.

Figure out language that makes it very clear that you are just saying thank you and that you’re being nice, not that you’re paying them.

A good way to get out of this dilemma is to give participants a gift card (e.g. from Amazon or another retailer if they prefer) instead of plain cash. Buy them a gift card online. Send it to them via email or print a voucher and hand it to them if you see them personally.

The test plan

Create a short and concise plan for your research effort. The gathering of this information should be collaborative and involve interviews with stakeholders and subject matter experts.

Collaboration isn’t absent from this process. All parties should have ample time to review the document to ensure they’re comfortable with the approach.

  • Objectives: What do you want out of the research effort?
  • Methodology: How are you going to gather insight?
  • Participants: Who are you gathering insight from?
  • Outline: What will the flow of the session be?
  • Test environment: Where will the research will take place?
  • Project schedule: When can you expect the research to take place, when will initial results be available, and when will the full report be delivered?

Get started and recruit users

Write a screener

To find the right participants for your study that both qualify to participate and are willing and available to do so, craft a screening questionnaire that will screen people into or out of your study:

  1. List your assumptions about participant criteria (e.g., business traveler).
  2. Transform participant criteria into measurable benchmarks (e.g., travels for business at least three times a year).
  3. Transform the benchmark into a screening question or questions (e.g., How often do you go on an airplane?). If a person chooses the “right” answer, she’s in. If not, she’s out.
  4. Craft a screening questionnaire (also called a screener) that you can send people. Pilot-test the screener with a couple of people and make improvements.

Invest time into writing really good screener questions. Avoid questions that give away what you’re looking for. Example:

You want to target people who are likely to buy a new laptop. Directly asking “Do you plan to buy a new laptop?” will cause people to say “yes” because they want to participate and get the incentive.

You can indirectly ask “What kind of laptop do you currently use?”, followed by “How long have you had it for?”. You will then choose those respondents who have had their laptop for quite a long time.

Be specific about what you want the participants to know or know how to do.

Example:

“Check off all aspects of Excel that you can use quickly and well:”

excel tasks

This was a screening question to avoid advanced Excel users. When a respondent checks too many of these answers, you will know that (s)he is an advanced Excel user and you can exclude him/her from a study that targets Excel beginners.

Pilot your screening questions with a number of people before you send them out. Look whether they let the right people through and tweak the questions if not.

Write an invite to get participants

The trick to writing a good invite is to show the person that you know them a little bit, that you value their time and you value what they are good at. You want to make sure that the invitation shows that you like and respect them.

Your invite ideally tells them right up front:

  • “Hi, here’s the kind of person we are looking for”
  • and “I think you’re a great example of one of them”
  • and “I know you’re really busy so this is going to be short”
  • or “I really want to get to know you so this will not be short”

A typical structure for an invitation is: a header with a title or subject, then in the body you will have a description and possibly some screening questions. At the end, you will want to make it utterly simple to sign up and make it clear what will happen next.

Example for a simple call to action (sign-up):

“Click this button and we’ll send you a link to the survey”.

In general, your invite should be…

  • well-written, easy to scan
  • explanatory
  • trustworthy
  • persuasive

Here are two examples for an invite to a gaming survey:

Good example (clear, succinct, easy to understand):

Subject: A 5-minute survey about gaming that gets you a $20 gift card!

Do you eat, live and breathe online games? We have a 5-minute survey that asks about your gaming experience. Answer the questions by 7/22 at 12:00 PM and you’ll get a $20 Amazon gift card.

Click here to go to the survey! Know any other gamers? Send them this link!

Thanks! [Real person’s name] Customer Team at [Company]

Bad example (too wordy, generic and fuzzy subject, not trustworthy):

Subject: We need you to do a survey

We need 40 people to do a short survey that will ask you a set of questions about your experience participating in and playing online computer games on your computer.

If you are interested in participating there is a link below to click on to go to the survey. Everyone who completes the survey will receive a gift of a $20 Amazon gift card.

To take the survey, click the link for the survey here: www.linktothesurvey.com

Survey must be completed by 12:00 pm on July 22, 2016 for you to be eligible to receive the gift card. If you know any other people who you believe might be qualified for this survey, please forward this email to them so they can also participate.

Thank you for your assistance,
[Company]

Position yourself as an individual who wants to talk to an individual. Make it as personal as you can.

Publish the invite

You can write the invite as an email to your mailing list, provided that people really have opted in to receive such messages. Check your sales or CRM system, your customer feedback, or your product registrations for valid addresses.

You can also write the invite as a web page and run ads on social networks like Facebook, Twitter and LinkedIn, depending on where your audience hangs out.

You can send out the invite via a professional recruiting firm, especially when you are targeting a very detailed group of professionals, let’s say investment bankers with a household income of over 400.000 dollars per year.

For a more consumer-like audience, you can use automated systems like justaskusers.com. You configure a recruiting brief inside that system. Just Ask Users will send out invites, track responses, checks whether the answers to your screening questions match your desired participant profile, checks availability for study time slots and notify you accordingly.

Track the invites

It is a good idea to keep track of all those people whom you invited to your study. Keep a worksheet with some information:

  • Contact info
  • Did they respond?
  • Could you use them in this study?
  • Did you like them or not?
  • Did they work really well?
  • How much was the compensation?
  • Any other notes about them.

One reason for this is: You don’t want people to become professional testers over time. If you invite them too often, they will cease to be “everyday users” and will start to behave differently. An effect that you don’t want.

On the other hand, it can be interesting to have a small group of people and run a user test with them, then make changes to the UI and run a test with that same group of people, again.

Save time: If you use an automatic recruiting system like justaskusers.com, it will do this tracking for you automatically.

Send people a “thank you” message after every study. This will build rapport with them and will allow you to ask for a favor later: “Would you mind forwarding this invite to your friends?”.

Review incoming applications

If you do this manually, you (or your admin) have to check a number of points:

  • Does the respondent match the profile?
  • Does she fit the demographics, the lifestyle, the knowledge, the tech-savviness required?
  • Did she answer the screening questions correctly?
  • How good is the match?
  • Can she make it at the required time(s)?
  • Does she agree to sign a non-disclosure agreement?
  • Does she live in the desired timezone?
  • Can she show up personally (if required)?

Just Ask Users (as a system) checks all this automatically for you and tells you the results. This can save a huge amount of time for you.

Select participants

Now you can decide whom you really want to invite as a participant. People with a high match rate for your demographics, other characteristics and answers to the screener questions will be your preferred candidates.

Assign participants to time slots

When you know whom you really want to invite, assign the persons to the time slots that you specified in the beginning. Make sure that the number of persons is right for the research method being used (e.g. one for a user test, or seven for a focus group, or 500 for an online survey).

Conduct the study and gather insights

Meet the people

Now you’ve got the people who show up at the desired times. Meet them, give them their assignments and watch how they do. Make sure that your equipment for video or screen-capture works flawlessly so that you really get the data you need.

Collect results and extract data

Identify characteristic user behaviour or answers and collect them in a way that you are able to extract the data that you need in order to generate insights, later.

Gather insights from the data

Generate the insights that you need:

  • what people need
  • who the users really are
  • how they currently solve “the” problem
  • what the user’s current workflow is
  • whether people really want the product/feature you have in mind
  • whether they can use the product
  • which design generates better results
  • how people find your solution at all

After the research effort is complete, a research findings report should be delivered to synthesize and report common insights. Types of documents delivered will depend on the methodology used, but you can expect a document that contains data, quotes, photos, videos, and enhancement recommendations.

As a follow-up to the test plan, all items listed in the plan should be included to ensure everyone on the team is aware of how this feedback was gathered.

Any influence on your design?

Now decide: Which ones of the insights that you generated from the data will influence your design? You will possibly be surprised by what you’ve got as user feedback and this will motivate you to create better solutions for your target audience.

What you’ve got now

The benefits of seeing real users

Once you’ve gone through the user research process, you will have had (sometimes jaw-dropping) insights about how users see your product. With this new knowledge, you can now improve your product significantly.

You can also save a lot of money for unnecessary design iterations. Imagine saving a month of work for your entire team just because you need not walk down a road towards a dead end that user research has revealed to you – how cool is that?


matthias bohlen

Matthias Bohlen, a German expert and coach for software engineering, has spent the last 20 years with many product development teams, helping them get beautiful and rock-solid software out the door without losing their minds in the process.

Matthias founded “Just Ask Users”, an online service for recruiting participants for UX research activities, e.g. user tests, focus groups, interviews and more. He got this idea when he saw various teams struggle when they built software that no customer really wanted. What they needed was input from real people: future users of their product.

Matthias is so excited about The UX Blog that he created a special one-time free offer for you, our valued readers: If you haven't used justaskusers.com before, you can sign up and get your first 6 participants for your UX research there – totally free of recruiting fees! Click here to enter your first recruiting order and look forward to meet real users for your product.