If you had the opportunity to see exactly how prospects engage and behave on your site, would you take it?

We spend hours, weeks and months planning our websites and funnels, but we don’t really know how people end up using them do we? Sure, you think we designed the perfect experience, but are you sure?

Google Analytics tells us where the problem is in the funnel, heatmaps and session recordings show us what people are doing on the page, but not WHY.

“Why are they clicking those buttons?”

“Why aren’t they scrolling all the way to the top?”

“What’s stopping them from taking the next step?”

If you’ve found yourself asking these questions and many more, it’s time to start considering user testing.

User testing has a bad rep for some reason, but if you pay close attention to today’s training you’ll see the HUGE benefits it can deliver and how it can help you figure out exactly what changes to make on a page to get it to convert.

In this session the queen of all things user testing Els Aertz, walks us through her exact process for running them successfully.

Here’s what you don’t want to miss:

  1. The biggest mistakes people make with user testing (and how to avoid them)
  2. How to recruit the best users (and make sure they show up)
  3. The costs and everything that goes in to planning user tests
  4. How to act and what to do during these tests to make them successful
  5. How to get your team on board with user testing

And much more!

Check out the reply here:


PS- If you’d like to take a crash course on user testing, checkout Els’s course here (you can also use the embarrassing discount code she kindly provided: TALIAROCKS)

Full Transcript:

Els: All righty, there we go. Thank you for that lovely intro there, Talia, and making me sound funny before I’ve even started. So you guys, I hope I will do Talia’s intro justice. We have a lot to go through. It’s a user testing webinar, baby, so let’s go. Now, what you really need to know about me is that I’ve been doing user testing since 2001. 2001 is the year I founded AGConsult together with my business partner Karl Gilis. And since that time I have moderated, yeah, over 3,500 user tests. It looks like a ridiculous number when you see it like that, it just mean I’m old, okay. There we go.

Now, when we talk about user testing we can talk about two different things. We can either talk about moderated user testing or unmoderated user testing. And with unmoderated user testing, you have a lot of nifty tools and cool stuff that I won’t talk about at all today because the focus today is on moderated testing because there’s a certain value that you get out of moderated testing that none of the other tools can give you, none of the other research methods can give you so. In detail, what we’ll be talking about today is pretty much the basics of how to set up a test, what you can test, how you can test it. Also recruiting because that’s something that I usually get a lot of questions on, also in scenario writing, and of course, how to be a good moderator. Looking forward to questions on any and all of that, anything else you have, shoot. Okay, the basics. So maybe for those of you who aren’t familiar with moderated user testing, just a quick recap. Moderated user testing is when you arrange for a small number of people to perform a series of typical tasks on your site. It’s one-on-one so focus groups are not user testing.

Talia: No, they’re not.

Els: Of course they’re not. User testing is one-on-one, okay. You have a moderator who typically sits next to your test participant and who really sets tasks and leads the test. Now, observation is key in moderated user testing. You observe the user while they’re using the website. User testing is particularly interesting to find out, to discover why something is a problem. Because you probably have a lot of quantitative analysis going on, you probably have log files going on, you’re probably doing surveys, you’re running those heat maps, you’re watching user session recordings. Those are also very interesting and can sometimes already lead you on a path to a why, but not always, and that is when moderated user testing comes in. When you’re creating a completely new website, user testing can also help you with the what, what is exactly a problem in your site? Because your site isn’t live yet, you can’t A/B test, you don’t have log files, user testing is the only thing you’ve got. What it can also do is help you generate A/B test ideas because it’s definitely an outcome user testing very often.

Now, what exactly can you test and how can you test it? Obviously, you can test your own website, your intranet, or your app. But what a lot of people don’t think of is that you can also do your test your competitor’s websites or apps and that can actually be super, super interesting. Because very often it happens that, you know, one of your competitors launches this shiny new feature and you think, “Oh, I gotta have that,” maybe and maybe not. Why don’t we let users decide, do a user test, have them test that shiny new feature that you want so much and then see whether they really like it before you start investing money in something that maybe your users aren’t all that fond of at all. I remember quite vividly a comparative user test that we did for a car brand of a number of car configurators, now this was a while ago. And the car configurator that our client, Peugeot, really wanted to emulate was the one that did worst in the test because it looked the best at first sight but it was very, very hard to use. So we actually saved them a lot of money by making sure they chose the right way to go and not necessarily the shiny one.

When can you test? Basically, as soon as you have an early, I would say, clickable prototype. Now, why clickable? Because these are not web people you wanna test with, these are regular users, and so you wanna make your early prototype as true to life as possible. You wanna create a scenario that they can actually click through and where you also have more or less the copy that is supposed to be on that page. So never ever test with Lorem Ipsum wireframes or prototypes because that just does not give you a real experience and you will not get great feedback from users. Obviously, you can test on a desktop, you can test on a tablet, you can test on mobile, all of those screens, you can task.

Now, there are three types of moderated user testing you can do. First I’ll go into the setup of these three. And then we’ll talk about stuff that is important regardless of what type of testing that you’re using, which is a recruitment, a scenario, and a moderating. Now, your typical set up for in-person moderated user testing is the two-room set up. So in one room, you have your testing room, this is typically quite a small room where you have your moderator and your test participant. And everything that the two of you say and everything that the test participant does, be it on desktop, be it on a tablet, mobile, it’s recorded. You have your second room, which is your observation room where you can have your team, or if you’re an agency you and the client, watching because everything that happens in the testing room is shared on the screen in the observation room.

Now, a lot of people say that you need a usability lab a two-way mirror to do this kind of testing. I say no, I say hell no, God no. I mean, I have clients who have usability labs and I beg them to please not let us not test there. Because it just makes test participants feel silly. They know there’s somebody else watching on the other end of the mirror. It is not a good situation. Another myth is that you should never test in the client’s offices. So if you are a company yourself you should never test in your own office, eh, because it’s supposed to intimidate the test participant. And again, I would say, “Well, it depends.” You know, if your logo is plastered over every surface of your office, if you’re wearing a T-shirt logo of your company, that’s a big no-no, but that’s obvious, right? If your name is on the bell but otherwise your offices look pretty neutral, relax. Seriously, we do user testing both in our offices and the client’s offices and it is definitely not the case that we get more positive results or people saying nicer things in the client’s offices, not at all. If something sucks, you will know, trust me.

One reason I really, really like in-person moderated testing is because in this observation room, and this can be as you can see here, sometimes this needs to be quite a big room because this is actually at the short end of the table that’s Eleen [SP] who is our user researcher, and at the long end of the table that the client’s team. Now, this is what I love to see when we do in-person moderated testing. This is a client, this is the Bayco [SP], one of our Dutch clients. They sent one, two, three, four, five six, seven there’s another one not in the picture, seven people to watch the test. And this is very important because this way they share this experience of a user test live, because as you can probably tell from this picture, things in the testing room are not going well, right? You see this, “Oh, my God, what’s happening?” Experiencing this first-hand and as a team is very important. It really lets you empathize with that test user. It really lets you feel their pain, and if you can do that as a team it also bonds you as a team. When I have clients like this who send seven people, I love them, okay, because this means you take your customer experience seriously.

Another benefit of in-person user testing is eye tracking. Now, eye tracking, if you know what this is, the red dots is where the attention is centered for a while, the red lines indicate the eye movement pattern. And this is really, for me, very interesting because it helps the people in the observation room follow the test better. Because as a moderator when you’re seated next to your test participant, you know, you’re close enough to really tell where they’re looking and when they’re saying something what they’re actually talking about. But when you don’t have eye tracking and people say something and you hear it in the observation room, sometimes it’s not clear whether they’re talking about the filters or maybe the sorting options or something else and the eye tracking helps you get extra information about that.

Now, if you’re just starting out and you’re thinking, “Oh, I really want to get started and do a lot more user testing,” should you worry about investing in an eye tracker? I would say no. We need to have it because it’s basically our core business. We didn’t have it when I started out in 2001. An eye tracker, my God, I couldn’t bloody well afford it. Did we learn less? Not really because eye tracking is the cherry on the cake. What you really get your insights from is the observation of your test user. You can use the eye tracking to back up your observations but that’s it. And please, please pretty please with a cherry on top do not be tempted to create heat maps when you only have five test users, okay. Those results are totally bogus so use it as a tool to back up your findings and especially to help people in the observation room.

So what do you really need to do in-person user testing? Not a lot. You need two rooms, a small room for the testing, a larger room for the observation room. Oh, yes please, none of those glass bowl rooms because that makes people feel completely watched, right. Hardware, of course, you need something to test on whether it be a laptop, tablet, or phone. You need another laptop to monitor on, and in the observation room, a beamer, microphone, speakers, that’s it. You can do a recording with Morae or Camtasia or if you do eye tracking I would say go for Tobii. We used to have another alternative, which was a very cheap and lovely eye tracker called myGaze, unfortunately, they’ve gone out of business.

user testing

And as far as I know, Tobii is the only other one that is actually small enough to really do eye testing properly on a laptop screen. We’ve looked at other eye trackers but they were too big and chunky and so if you wanna do users testing with eye tracking, well, with a mobile lab, the Tobii is a very good option. Mobile, what do we use there? Well, when we test on iOS we use Reflector and Airplay, which is basically on your phone ready. For Android we use SideSync and we also use an additional webcam so we can film the user’s face. There are other tools out there. I have not yet found a tool that I am completely and utterly and super in love with, I have to tell you, so.

Now, that is in-person user testing and that is great. It does, however, mean that, you know, you have to have that, you know, a bit of technical set up and you have to have those two rooms, you have to get people to come to your office. And sometimes that, you know, is not an option, which is where Guerilla user testing comes in. And in Guerilla user testing there is no technical set up. It’s just you and you’re testing device and you go out there into the wild and you test there. Now, Guerilla user testing is not really an exact substitute for in-person user testing because you can’t have people test on your website for an hour or so. If you just go out to a coffee shop or you go out to maybe a shopping mall, if you get 15 minutes of somebody’s time you should be very grateful because usually people have, you know, a life outside of helping you during your user research.

Guerilla testing is great if you only wanna get feedback on a very specific part of your website. If you have a very specific task that you wanna set and that you want a lot of input on. Because when you do go to that mall, when you do go to that coffee shop, and this is important, where your test audience is, more about this later, there’s a higher chance of getting a lot more people to do the test with and the task with than when you would do it in person. Nobody’s going to come to your office to do a test of 15 minutes, that’s not worth anybody’s time.

When is it also interesting to leave your office and to go where the user is, is when you’re testing for something that is when it’s an application or when it’s a site that people use in a specific environment. So, for example, we did testing for the extranet of a large bank and insurance company, and so we wanted to see, okay, how are these people using it? Do they sometimes have to use it while there’s somebody at the counter? Because that adds pressure, that means your internet has to be extra super fast, thank you very much. Do they sometimes have to turn the screen and show their client something? Okay, that means certain sensitive information maybe should be hidden behind another click. So all of those things are very important and they give you a good view of what the website or app is normally used in in a natural environment.

Now, this still involves you getting out there. If you don’t wanna do that, you don’t wanna leave your office, then remote moderated user test is also an option. And again, I quite like remote for smaller, shorter tests. I’m not a huge fan of remote if it’s a very complicated website with a lot of steps and features to go through. Because while, you know, you have the screen and you have your webcam…this is a little inside joke for anybody who knows my business partner, Karl, there you can recognize him from a few years ago. You have your webcam and you have your screen but you’re not next to the test participant, there is a certain distance in the conversation and you just feel that. Now, that’s okay, if that’s the best option you have it’s okay. But definitely make sure they turn their webcam on because sometimes people can say something and you can’t always tell by the tone of voice what they mean but if you look at their face you usually can. So watching somebody’s facial expressions is a big part of user testing, observing that, so if you can get that remote as well that’s awesome.

As you can see, this is the old version of GoToMeeting that we use here and we still use either GoToMeeting or Zoom, like today’s webinar, to do our remote moderated testing. Why? Well, because it’s easy, you’ve got screen sharing and you got the recording in one, you’re all good to go. Now, regardless of which type of test you choose, and that might depend on your budget whether if you want the client to watch it live, yes or no, whether you can get the test participants to come to your office, yes or no. Recruitment is very important. Recruiting the right users, making sure they show up, especially if you do in-person testing, key.

Now, never test with anybody who loves you, so not your partner, not your parents, not your colleague. They may not love you but they probably they know too much about your product to be a real test user. You need to test, and this is the most important thing about your test user profile, you need to test with people who are actual or potential clients of your product, people who give a shit about what you’re trying to sell them or about what you’re trying to tell them. This is far more important than social demographic data. I get sent screeners that are ridiculously, ridiculously precise. Like we’re looking for women between 30 and 36 with 3 children ages this and that other child has to be this and that. Like, for God sakes people, we’re testing a toy shop, relax already, right? What are we looking for? People who buy toys, it doesn’t matter whether they’re 29 or 32. Do you really think there’s a magic difference there? Seriously, there isn’t. Potential actual user of your product, that’s the most important thing.

If you’re testing on mobile, make sure you always ask people’s device preference as well because…and make sure you have testing devices of both kinds. Now, we sometimes test on prototypes that are only available on, for example, Android. There is absolutely no point inviting iPhone users because when they struggle you will have no idea whether they’re struggling with the Android device or struggling with the app. In my case, it would be the Android device but you just don’t know, that’s the point, so be very mindful of this. Even versions can be tricky so when you call your test participants be sure to ask all that. To recap, test with strangers, recruit for engagement and not demographics, and be very mindful of device familiarity.

Now, should you do recruiting yourself or should you outsource it? Well, I would outsource it in two cases, actually. One is if you’re looking for test participants who’ve never heard of you, and sometimes people want that, people want a completely fresh set of eyes on the websites. For some companies that’s very hard. If you’re Amazon, good luck with that. But it also means that you can’t recruit people through your own channels and so that’s when it’s best to outsource recruitment to a special recruitment agency. Now, if you just Google “recruitment agency, marketing research, recruitment agency, user research,” I’m sure you’ll find one where you live. Typically, the cost per person via a recruitment agency, in my experience, is between €50 and €150 per test participant depending on the country that you’re in or the test participant’s profile.

If you’re testing with your own customers or if you have no budget to outsource…if you have the budget to outsource, outsource is very important. If you have no budget to outsource you can do it yourself. And you can just use your own channels, you know, have a little pop-up or a slide-in on your site where you say, “Hey, we are always after feedback from our customers. Would you be willing to participate in a session in our office? It will be then and there, da, da, da.” Same message to your newsletter subscribers, to your client database, your social media, get people from your call center involved. And if they get people on the line with a specific problem that you wanna research further, let them get it in. If you have physical locations put it up on a notice board. And if you’re looking for a non-customer, but you know, you want them to maybe be familiar with your product or your competitors, scour the web. Put messages up on Fora [SP] and see whether you can actually recruit.

This is always very interesting. People who are actual clients of your competitors, always great to have these people in because they can tell you what they really like or not like about your competitor versus what you’re offering them. You recruit. Never, ever say that you’re doing user testing because it sounds a bit creepy, okay. It sounds like you’re testing users this is long been a debate in usability circles. Of course, I know we’re not testing users, for Pete’s sake. We’re testing the usability, the user-friendliness of a certain product, of a certain site with people. “User testing,” just make sure you never say it in recruitment or during the actual user test itself. Once you’ve found your participants don’t just think, “Oh, yeah, these people fit the profile. I’ll invite them.”

No, you really need to screen your candidates, and even though in your recruitment screener you should, of course, always ask things like age and occupation etc., sometimes people can be a bit vague about this, and so yeah, you should get a bit Sherlock Holmes-y on them. You have to sort of Google the candidates and check out their LinkedIn profile because what you really wanna avoid, unless you’re testing a site or an app for IT web marketing for communication professionals, is to get any of these guys in. Because instead of using your site or app, they will start giving you very well meant but completely unasked for advice.

They will start talking about, “This site’s speed is very slow. I wonder if it’s maybe because the image of…” oh, for Pete’s sakes, no normal user would ever say that, okay. They might go, “Well, this is slow, it’s taking forever,” that’s normal, but not speculating on image size, etc. As soon as that happens you might have an amateur web professional on your hands and that’s also not a great thing. So do try and weed those people out, that’s not who we’re after. Sometimes you also get people who actually work for the competition. Well, this has happened to us. So it’s very important not to invite these people, especially not when you’re testing a prototype of a new feature or of a new version of the website. You don’t want the competition to get a heads up there.

Now, that’s not enough. I always advise to always speak to your test participants beforehand. As soon as you make your selection of what do you think a representative sample of your target audience is, get them on the phone. Call them to check whether they’re still available, make some small talk with them, and then you can sort of get a feel for the person. Because, ideally, test participants in a user test, they can be shy but they’re preferably not too shy. They’re preferably not so shy that you can really almost not hear them talk, because that’s very annoying. If you have to do that with, you know, five users, six users all day long, that’s not really great. People in the observation room probably won’t hear. There’s nothing wrong being soft-spoken, there’s nothing wrong with having a stutter, there’s nothing wrong with any of that, it just makes it a little bit harder if you’re doing user testing.

Another thing that you can often really tell on the phone in the conversation is whether the test participants are in it for the money. Good test participants don’t ask you anything about the fee because you have communicated this fee beforehand in your recruitment screener. People who are in it for the money will ask you things like, “So this reward, is it like cash or is it a voucher and when do I get this? Will I get it as soon as I arrive? Will you send it later? Do I have to do well?” No, if the talk goes to money, then sort of like brush them off and try to select another test participant because people who are in it for the money, again, won’t have the natural user behavior that we are really looking for.

Don’t confirm participants once, at least twice, my preference three times. So call them, say, “Okay, great yeah, lovely that you can come all right, fine I’ll send you an e-mail with all the details and the location, etc.” Ask them explicitly in the mail to send you a confirmation e-mail. Send another reminder a week before the test. And you’ll thank me for this last one, call them up again the day before the test to ask, “Oh, hi, yeah, we have you know, we have a test scheduled…we have an appointment scheduled tomorrow at 2:00.” Most people will go, “Yeah, and I confirmed it twice,” and you will go, “Yes, thank you, that’s great.” And some people go, “Oh, that’s tomorrow? Oh, shit, I have a doctor’s appointment planned.” And you will go, “Oh, well, I’m sad to hear that but I guess you can’t reschedule that.” So you have to go and find a new test participant, and if you call them the day beforehand at least you’ve got a day to do that.

Or, ha-ha, tricky, tricky trump card here, what a very good thing to do is, is have a sort of spare test participant handy. So if you’re planning to do tests with five people, an hour-and-a-half per person, actually recruit six. Have one person on standby for the whole day. Obviously, this should be somebody who lives really close to the test location and somebody with a very flexible job or somebody who works part-time who can get there quite quickly. You will need to pay this person to be on standby even though they don’t show up. But let me tell you, especially if you’re on the agency side, it beats paying an extra person rather than sitting there for sometimes an hour-and-a-half in the middle of the day not having any test participants, that can get really annoying. So having a spare handy is just a good practice. Another good practice is to also if you’ve done a big recruiting round send an e-mail to everybody who didn’t get selected to participate in the test and thank them for their interest. Because that way, next time you launch a campaign to recruit user testers then they’re more likely to volunteer again and, you know, it’s also just a decent thing to do, really.

Okay, money, you’re gonna have to give your test participants something. Now, this also again depends on the country that you’re in. In Belgium, we can give cash to people. We used to give gift vouchers but then people emailed us, called us up a year later and said, “Oh, I got this gift voucher from but it’s no longer valid, can I get a new one?” Oh, my God, no. So now we just give cash, cold hard cash works really well, you can do anything with it. I know that in Switzerland, for example, you can’t do that, it’s against the law, so do make sure you’re not breaking the law. Gift vouchers are a very good alternative, Amazon gift vouchers [inaudible 00:36:02] gift vouchers if you’re anywhere in the Netherlands or Belgium. And if you’re a SaaS company, I would actually advise against these types of incentives. If you’re a SaaS company, what you should give away as an incentive is X months of subscription both to your existing clients who participate in the test and to potential clients who participate in the test.

Because that way, one, your existing clients, you keep them on for longer, you give them a very real benefit because they’re already using your service. And for the potential clients, you know for sure that they’re participating because they want your product, they’re interested in the kind of solution that you have to offer. Also, B2B, giving away cash is a bit awkward so, you know, give X months of subscription to your service. If you’re an e-commerce, a great thing to do is basically give your test participant money, cash, a voucher to participate. But also give them a voucher to spend during the user test. Now, this is gold because, let me tell you, it makes a huge difference whether somebody is looking for a pair of shoes, and it’s just really a hypothetical pair of shoes because they’re not actually gonna get them in the mail, versus, “Here’s €50, you can spend it.” Oh, boy, all of a sudden there’s a lot more attention on, “Hmm, is this…okay, this is a size 37 but I have never tried this brand. I wonder if it sizes big or small?” This is not something people stop and ask themselves if it’s not their real money their spending.

So sometimes people get completely caught up in choosing a pair of shoes, or anything else really e-commerce, that you have to sort of say, “Let’s check out their return policy,” because otherwise, they can spend basically an hour-and-a-half browsing for a product. So that’s an important one to keep into account as well. The question that always pops up, how many people do you need to test with? And usually people come up with this graph, which is a bit, if not to say a lot, misleading because this graph would lead you to believe that if you do user testing with 15 people, boom, you will have the perfect product, you will have a perfect website. Obviously, I don’t need to tell you this is not true.

How many people you need to test with? Completely depends. It depends on the complexity of your website. It depends on how many problems there are to begin with. It depends on whether you have one target audience versus several target audiences. Now, my advice would still be to start small, start testing with five or six people per target audience per iteration. Now, when I say target audience I don’t mean test with 5 men, test with 5 women, test with 5 people in their 20s, test with five people in their 30s. No, if you have a website that has a section geared towards professionals and there’s a section geared towards consumers, test with five professionals and test with five consumers. Or if you have a website in different languages, those are also different target groups. Because in Belgium, we have a lot of French-speaking people, a lot of Dutch-speaking people, sometimes behavior preferences can differ there.

What’s important is that you iterate because…and you’ll notice when you do testing, of course, test user number one, everything is new, you find out so much stuff. Test user number two is very often repetition of test user number one and a few new things. Test user number three, more repetition, you get where I’m going. By test user number five, very often you see a lot of patterns emerging. You really already know the biggest things to fix. And the smarter thing to do instead of testing with 20, 30 people at the same time, which I sometimes get asked to do and I just flat out refuse because it’s boring, I say, “No, let’s test with 5, 10 people maximum, fix the issues that we found or test the solutions that we thought up, and then test again on the improved version.” Iteration is very important in improving any website and it’s also a very important part of the user testing process.

Which brings us neatly to another very important part of the user testing process and that is writing the scenario. Now, in your scenario you need to set the scope of the test first. You need to think about, okay, what is my research question? What are perhaps things that have been flagged by quantitative research? What did you see in your survey as an answer that you really wanna get to the bottom of? Where do you see in analytics that there’s a big drop off that you just can’t figure out why? What are people in the call center getting questions about? What do you really want to see and get qualitative input from? Where do you wanna go deep?

If you don’t have data yet because your site’s not live because it’s completely new project, well, you know, common sense to the rescue there. You have to be your own test user, you just think that, okay, why would people come to this website? What would they look for? What would their questions be? Try and to do that and make that into the tasks. Always focus on the important things. Focus on the top tasks on a website. Don’t go asking people to look for absolute exceptions, because quite frankly, there are always going to be exceptions and things that are hard to find, especially on very large websites. That’s life. It’s not that important, 80/20. It should be easy to find and easy use those things that are important for 80% of our visitors.

When you have your test scope set make a list of possible tasks. Well, how many tasks should I set? Again, that depends. I have done user test on a ridiculously complicated flow once where basically that was the task, complete that flow. Of course, there were many sub-questions that I asked but they were not part of my scenario because they are part of what happens during the user test. What you as a moderator ask or say or don’t say completely depends on what the test participant is doing. Now, if you’re testing the overall usability of something, and I usually have between 10 to 15 tasks, it never hurts to have more tasks than you actually asked because, you know, especially if you do in-person moderated testing you have these people, in you have them for an hour-and-a-half, use them to the maximum time possible. Ask any and all questions that you have. If they finish it on your site, even if you weren’t planning to do a comparative test, take them to your competitor’s site.

A very important thing here is that you shouldn’t set the tasks always in the same order. You should vary the order in which you ask the questions. Because if you don’t you that you will find that at the end of the day task eight had a much higher success rate than task one, which isn’t very surprising at all because people have been spending time on this site, this is probably after 40, 45 minutes. They have already seen stuff while they were doing other tasks and it’s natural for them to find the answer to that question a lot sooner. So mix it up so you change the order, and also if you do comparative testing like, for example, your site and two of your competitor’s websites, make it an ABC, first your site for the first test user then the next one you show your competitor’s site first to the second test user, etc. So there as well that you counter possible bias.

Also a big no-no when writing your test scenario when setting your tasks is using words that are actually on your website, you know, it’s setting a task, it’s not leading people through the thing. For example, if you’re a travel website and you have city trip in your main menu, don’t ask people to book a city trip to Berlin, always try and find a way around it. Now, I don’t know a synonym for city trip so I say stuff like, “It’s been a long time since you spent a long weekend away with your partner and you’ve been to London and Berlin a couple times, so you’re looking for something new.” And then you’ll see, do they go to city trip, yes or no?

Also tasks there are too specific or aren’t relevant for the participant, don’t you that. Tailor your question to the participant. The question I asked earlier, you’re looking for a city trip to go to London. No, really wrong. Why would you tell them where to go? Maybe they’ve just been to London, maybe they hate London. In the pre-test interview that you always do with your test participants where you ask them questions like, “Did you get you find your way okay? Oh, my God, it’s raining again here in Belgium, isn’t that terrible?”

You find out a little bit about them as well and about their preferences when it comes to the product, the site you’re testing. So on a travel website, you should ask them questions like, “Who is your favorite travel companion? What style of travel do you like? Are you a camper? Are you more of a luxury type of person? Any favorite destinations?” So you can actually say I like, “Oh, so do you wanna go away for a long weekend? I heard your best friend is your favorite travel partner, that’s great, you know, is there anything on your shared wish list of destinations? Dublin, that sounds awesome, the home of McGuiness, a good place to go.” That’s what you should do.

Now, sometimes you can’t always tailor the question to the test participants’ needs and that it can help to sort of start the beginning of a story. For example, for a web shop you could say something like, if you want people to get to use the filters, “Okay, so your friends have recently moved house and their throwing a housewarming party. It’s a themed party and everybody has to wear red. You have a lot of red in your wardrobe?” Most people don’t. “No? Okay, get shopping.” And that’s it, don’t tell them to buy a red dress or don’t tell them to buy red this or that, let them finish the story, let them think, “Oh, red, party, okay, what’ll I wear?” Now, some girls might go for a red party dress. Some girls might go for a red onesie. I am all for that. Let the test participant finish their own story, you start it, they finish it.

This is that horrible question you should never ask, “You wanna go to New York for Valentine’s Day?” No, no, no, no, keep it vague, keep it tailored to your participant and realize that your user test scenario is a guy and not a straightjacket. Very often the test participant will do something or say something that is just gold for you as a moderator to pick up on. Don’t feel fixated that you need to ask these questions. What you should know is, what do I want to find out? What do I really want to learn about this test? Tailor that to your participant. So those are the takeaway for that one.

Kick-ass moderator, extremely important thing to be and the hardest, because yeah, the moderator is both the biggest strength and weakness in moderated user testing, and one of the reasons, that Talia said in the beginning, why a lot of people think user testing is useless. Not if you’ve got a good moderator. A good moderator has to be like Jared Spool [SP] says, “Three personalities at the same time.” You have to be a flight attendant, you have to set your test participant at ease, people have to feel comfortable sitting next to you, people have to indeed feel like they’re free to surf that website, use that app as if they would at home. If their phone rings and they stop and pick it up, actually…well, it’s not good because they should’ve switched off. But that’s not a bad sign because it means they’re comfortable. This is an important job.

You also have to be a sportscaster because you as a moderator right next to the test participant, and in the observation room they’re not that close, so you have to sort of relay the actions of the test participant to that observation room. You also have to be a scientist. You’re not there…this is not a conversation that you’re having with the test participant, it’s not an interview, okay. An interview can be super valuable as a research method but a user test is not a conversation, it is about observation. Those are Jared’s three.

This is my fourth. You also have to be Switzerland, by which I mean that you have to be neutral. If you have a really comparative user test and you have a preference for one of the sites, and as much as you try not to be biased very often this just happens, you cannot, not, not let that bias show, you have to be very aware of the way you sit, where you watch on a screen. We do user testing workshops in-company and sometimes we have beginning moderators, they ask a question and then proceed to look at the position on the screen where the test participant is expected to click. This is not good. You know those horses that can count? This is exactly that. You start clapping as soon as they reach the number, yeah, that’s good. Don’t do that. Seriously, this happens a lot. So no, this is very hard.

Also never ask leading questions. Don’t ask, “So do you like the way this…do you like this product photography?” It takes a very competent test participant to say no to that, you know, it’s obvious that you like it. Never ask questions like that, always ask open-ended questions. I think Talia says it really well when she says, “Ask why until there are no more whys left,” that is how you do research. Also, don’t do too much of the talking, 99% of the talking during a user test should be done by the test user. And it’s not about expressing their opinion but it’s about saying what they see, saying what drives them, and sometimes they will say things like, “Weird.” The only thing you have to say to get more response out of that is, “Weird?” They will happily start talking again. Seriously, not a problem, you don’t have to say, “What exactly is it that you find weird?” No, maybe they find something in general weird, just echo the weird, that’s it.

Finally, because I think we’re running short on time, you won’t be a good moderator the first day you test. You probably won’t be a good one the second. This is something that one you need an innate sort of…you have to be a people person but you also have to practice, especially the part of the non-biased. If you practice, you will get good. Thank you very much. You’re thinking I’m forgetting sounding, aren’t you? You’re thinking I’m forgetting the present I have for your webinar visitors, Talia. I haven’t forgotten, no, especially for you. If you like user testing and if you’re interested in learning more about user research I have a whopping €200 discount on our online video user research course. It’s not taught by me. If you’re sick of seeing me, don’t worry, it’s taught by my colleague, Karl. And that’s the URL and, of course, the coupon code. What else could it be but TALIAROCKS, okay? There is a limited…it’s only valid for a limited time so it will only be valid until next week but Talia will send you a reminder in your inbox.

Talia: Duh, I wanna take that and close now. I mean, I’ve been looking at this, like listening to you so avidly. You probably couldn’t see my expressions, like I had my cell phone on you mute. But anyone who’s in here who’s just looking at my expressions was like. “Wow.” There are some really, really awesome things in there that I took away for when I do user testing, I don’t do much, unfortunately. When I do it for clients I usually outsource it. That’s a really, really cool insight and we actually have we don’t have much time for questions but I just wanna throw in a few. So, Jay…you mentioned that you should iterate the same thing…and he was asking if you should iterate with the same participants? Again, like after you’ve done the fix should you ideally iterate with the same person or bring in new people?

Els: A mix. I’m a big fan of the mix. I think it’s a great question. It’s interesting sometimes indeed to have the same people because we had a project recently were basically tested in three iterations. And there were two test users of every target audience that we took along with us on every testing round and then there were new ones. Because sometimes, again, especially when it’s a new, new product you want people, new users, to be able to use it. But it can be very valuable to get indeed that input from people who have seen the previous version and who can then say, or who you can then tell oh, they can’t use it before and they can use it now. Sometimes it’s also they couldn’t use it before, they still can’t use it now, but there you go. But I would go for a mix, but it’s an excellent question.

Talia: That’s actually an interesting thing because you were saying like we do the user testing, we gain our ideas and our hypotheses about like what needs to be fixed, and then you go in and you make a change. So usually I guess there’s a long time between each user test and the next because you’re waiting for the results to come in.

Els: Well, again, that also depends because if you’re testing on wireframes what we used to do was basically…and this also happens on software where you do Wizard of Oz testing. Basically, if it’s wireframes, and I’ve had to do this thankfully not very often, but test user number one and it just bombs the structure, bombs the interface, bombs, you know, like, “Oh, God, this is terrible.” Second…but this might be a fluke, it’s the first user. Second user comes in, it’s a disaster. I’m not gonna wait until five users are done to make any changes. These are wireframes, but you don’t eat lunch and you fix those wireframes with everything you’ve learned in the first two sessions and then use those for the remaining three because this is what user testing is about. It is not about getting pretty statistics. You can never say X percent of testing…

Talia: Right.

Els: …got this. There’s five other people, but it’s about getting insights. But you’re right in the other user testing, the three steps that we did, there were often a couple of months between test iterations because the changes that needed to happen were quite big and they were not in wireframe stage anymore, so yeah.

Talia: Right. Okay, I guess that also kind of answers Jed’s [SP] question about the frequency of user testing. So the question was, what frequency of user testing do you recommend, every couple of months, every month? And what ratio of face-to-face moderated message remote do you suggest? So you kind of actually mentioned that in the beginning, we don’t have much time to get into it, but I guess frequency is about what you need, right? You locate a problem try and fix it, when you think you’ve got it, go and do the testing again.

Els: Yeah, and obviously, you don’t have to do user testing as often as you have a look at your analytics. But if you only do user testing once a year, I think you’re selling [inaudible 00:58:16] your short. So once you do it, I think you’ll find that you get a lot of insight out of it and it really pays off to do it on a more regular basis. Again, also depending on how often, for example, you launch new features etc. that can create special circumstances to do it even more often.

Talia: Okay, two more questions, really important ones. One is how do you recommend selling user testing in a company? So if your client isn’t really into user testing or your manager isn’t and your boss isn’t, how would you do that? Such a huge, important question in like a minute, but how would you do that?

Els: Yeah. Well, I always try to emphasize the fact that there’s simply insights you get from user testing that you get from nothing else. And I do realize that it can be a hard sell. Fortunately, I’m blessed with clients who usually are very open to it because they know that we are basically complete advocates of it. And I have never had a company say, “Oh, well, that didn’t deliver what I wanted.” I think you should just say that basically, “The quantitative research gets you a lot of data but it gets you thin data. It tells you how many people did something and it tells you what is wrong but it doesn’t tell you why something is wrong.” And very often I also see the more complicated a product is, the more complicated a survey is, the more there is a need for qualitative user research like interviews, like user testing because you really need to get into the minds of people, you really need to get go deep with a few people and get rich thick data to solve the problems that you are having.

Talia: Right, okay, then I guess we’ll have to end with the last question, which is, how do you suggest to build a report of the user testing sessions for those people who couldn’t be there like in the observation room?

Els: Yeah, this was a section I had to cut out because I’ve…but I’m glad you asked because, well, I’ll tell you the mistake that we used to make. We used to do reports that were like 150 slides long and…

Talia: Wow.

Els: …I know, and have every problem listed in there. You know what this does? This completely confuses the [inaudible 01:00:59] out of your clients. Most clients are very happy when you give them the five most important things to fix. So that is actually what I think most reports should be focused on. Even if you’re an agency or if you need to do a big report, take the five most important things to fix, focus on that at the beginning of your report, focused on that at the end of your report.

Talia: Yeah.

Els: And also, if I can add one more thing about reporting, if you need to report to management or C-suite, what is really, really nice to have are clips of the recordings. We make what I call “worst of videos” where basically the pain point that keeps coming back again and again and again, we just have little clips, user one number, bam, user number two, bam. This, I tell you, creates awkward silences but it really hits home and this is also what people tell me afterwards, “We still watch those clips that you sent just two years ago,” and I’m like, “Excellent, that’s good.” Yeah, when somebody goes, “Maybe we should try that,” no, no, no, that’s how it used to be. Let me tell you, that is not the way to go.

Talia: People do not like that.

Els: People do not like that, no.

Talia: Els, I wanna thank you so much for taking the time to teach us and school us on user testing. I know that I’ve learned a ton. People who saw me knew that I was writing down all sorts of notes for myself. As I said at the beginning, the recording will be available in a few days with a transcript and a few screenshots of important stuff that you need here. Els, thank you so much.

Els: Thank you.

Talia: And thank you everyone, yeah, for taking the time today. Have a great evening or morning wherever you are in the world. Bye.

Els: Bye.

Facebook Comments


Powered by Facebook Comments

Pin It on Pinterest

Share This