A while ago, I got the chance to beta-test a new version of a website I use several times a week. Initially, I was thrilled: Giving feedback on products is kind of what I do, so applying those skills to the relaunch of a site I loved would be even better, right?
Then, the actual beta test started, and almost immediately, I turned into a nightmare user.
I didn't log into the beta site as many times as I was supposed to. I didn't submit as much feedback as I thought I would. And worst, when I noticed errors or bugs, I didn't always report them, figuring someone else would do it. Basically, I turned into the user I always hope I never recruit!
Being on the other side gave me a new perspective on longer-term user experience projects such as beta tests or diary studies. No matter how motivated users are at the start, it can be hard to maintain that level of enthusiasm once testing has to fit the rhythms of everyday life.
While I largely blame myself for my bleak levels of engagement, I also kept thinking of ways the beta could have been set up to better encourage my participation. Yes, all of my complaints could be seen as excuses for being lazy. But who among us has never had a user (or
been a user) who makes excuses for being lazy?
Here are three ways I think the beta setup could have gotten more out of me:
- Make it easy. I needed a special URL and password to use the beta site. It doesn't seem like a big deal, but the extra step deterred me. My default behavior was still to go to the main site — which continued to be functional — and not dig out the special login details. Had it been possible to access the beta through a link on the regular site, I have no doubt I would have used it more.
- Send reminders. Otherwise known as, lay on the guilt. There's a balance between overwhelming people with requests to use the beta and just going silent and hoping they'll do what you want them to do. In this case, I could have used one or two more reminders — let's say, one nudge per week of the test — to urge me to participate better. Bonus: Include the login details in each of those reminder notes so participants don't have to go searching.
- Provide multiple routes for participation. I'm most comfortable reporting bugs and other issues by taking screenshots. However, the primary way to submit feedback for this test was through a survey form with no upload field for images. I'd sometimes become so frustrated by having to describe my issue in words that I worried the point wasn't coming through — or, worse, I'd choose not to submit the issue at all. If I could have uploaded or e-mailed a screenshot in addition to submitting the form, I might have been a better tester.
As someone who's designed studies, I see all the reasons these things might
not have been doable or even desirable. Putting a beta link on the main site might not have been technologically feasible or might have caused hassles when non-beta users tried to log in. There's much to be said for streamlining feedback and capturing all of the issues in a standardized form and place. And it's certainly imperative not to annoy your most motivated users with constant reminders to do something for you.
Still, it's interesting to realize what my own ideal beta testing experience would have been and think through the tradeoffs. It's something I'll certainly keep in mind the next time I'm designing a longer-term study of my own.
Have you ever been "on the other side" as a user or tester? What did you learn from the experience?
