In user testing, you typically only have one chance to get all the feedback and data you need out of your participants before you never see them again. Missing out on user data (especially for tests with limited numbers of participants) could potentially prove costly to the overall findings of a study.
Getting all the information you can in a session does not consist solely of asking all the relevant questions. It's also heavily dependent on the way those questions are asked.
Instead of using a complex product covered by all sorts of non-disclosure agreements as an example, here is a random photo I took a few weeks ago. The goal is simple: Get information about what the photo displays, using three different versions of the same question that could be asked during a user interview.
1) "I will now show you a photo I took. Does this look like a bathroom floor to you?"
Suggesting an answer within the question, whether it be the right one or not, is one path to getting a flawed response. Do not be surprised if the first answer you get is, "That does look like a bathroom floor, now that you mention it" and you find yourself struggling to get more out of the user's response. The closed-ended nature also suggests that you would be happy with a yes/no answer, regardless of any other details.
2) "I will now show you an interesting photo. Where do you believe I took that?"
This question is what I call "the common mistake." It comes across as a question that is not leading when it really is.
Sure, it does not provide an answer, but an adjective (like "interesting" here) will almost always change the mindset of the user you are interviewing (especially when that question is combined with a smile — and UX researchers specifically are notorious for our flashy smiles). Your users will be asking themselves: "Why is it interesting? What is the back-story? Where would he take it? What should I say? It can't just be a boring wall...or can it?"
Once you get the user to over-think an answer, it is very easy to get a lengthy, seemingly solid response that really has nothing to do with the user's initial thoughts and perceptions. Think of it as one of those terrible relationships where answering a question is more about "what should I say that sounds right?" versus "here is what I really think." The result? Inaccurate data that could make its way into your findings — and a not so happy ending.
3) "I will now show you a photo. What do you see?"
This is the preferred, neutral way to ask the question. It lets users express their opinions without any psychological interference, and the open-ended nature of the question subtly suggests that you are looking for a more detailed answer.
To get more of the valuable answers, I have found that the best information comes with three magic words: "Tell me more."
The first question would most likely get you a good answer, but that simple three-word follow-up is where the users really start digging deeper into their thought processes and articulating their perceptions. After answering that the upper image is of a bathroom floor, they could use the follow-up to share what else it could be, what it initially seemed like, and why they chose to provide you with the answer they gave. You get an additional sense of how sure or unsure they are of the answer they gave — and could even see them have a change of heart and give a new answer altogether. All of this is valuable information that would have been too easy to miss by simply moving on to the next question.
Of course, the three word follow-up does not have to happen after each and every question. The key is to get a sense of what additional information you may be able to get — and weigh whether this information is worth the limited, valuable minutes you have with the user.
If you are curious, the photo is of an old kitchen wall (at a bachelor's house).
1) "I will now show you a photo I took. Does this look like a bathroom floor to you?"
Suggesting an answer within the question, whether it be the right one or not, is one path to getting a flawed response. Do not be surprised if the first answer you get is, "That does look like a bathroom floor, now that you mention it" and you find yourself struggling to get more out of the user's response. The closed-ended nature also suggests that you would be happy with a yes/no answer, regardless of any other details.
2) "I will now show you an interesting photo. Where do you believe I took that?"
This question is what I call "the common mistake." It comes across as a question that is not leading when it really is.
Sure, it does not provide an answer, but an adjective (like "interesting" here) will almost always change the mindset of the user you are interviewing (especially when that question is combined with a smile — and UX researchers specifically are notorious for our flashy smiles). Your users will be asking themselves: "Why is it interesting? What is the back-story? Where would he take it? What should I say? It can't just be a boring wall...or can it?"
Once you get the user to over-think an answer, it is very easy to get a lengthy, seemingly solid response that really has nothing to do with the user's initial thoughts and perceptions. Think of it as one of those terrible relationships where answering a question is more about "what should I say that sounds right?" versus "here is what I really think." The result? Inaccurate data that could make its way into your findings — and a not so happy ending.
3) "I will now show you a photo. What do you see?"
This is the preferred, neutral way to ask the question. It lets users express their opinions without any psychological interference, and the open-ended nature of the question subtly suggests that you are looking for a more detailed answer.
To get more of the valuable answers, I have found that the best information comes with three magic words: "Tell me more."
The first question would most likely get you a good answer, but that simple three-word follow-up is where the users really start digging deeper into their thought processes and articulating their perceptions. After answering that the upper image is of a bathroom floor, they could use the follow-up to share what else it could be, what it initially seemed like, and why they chose to provide you with the answer they gave. You get an additional sense of how sure or unsure they are of the answer they gave — and could even see them have a change of heart and give a new answer altogether. All of this is valuable information that would have been too easy to miss by simply moving on to the next question.
Of course, the three word follow-up does not have to happen after each and every question. The key is to get a sense of what additional information you may be able to get — and weigh whether this information is worth the limited, valuable minutes you have with the user.
If you are curious, the photo is of an old kitchen wall (at a bachelor's house).