Testing new software with real users during an iterative development process is a good idea. In fact, stated like that it seems pretty obvious… but being an obviously good idea does not make a thing easy to do.
In 2016, we embarked on the journey to create better usability testing capability in Polymorph. So in late January, we started with a two day user testing course run by User Experience Consultant Cara Winterbottom. The course, attended by Designers, Developers and Agile Managers, included the expected host of information on user testing in general – including introducing a variety of ways to engage with real users depending on the objective (e.g. card sorting vs. formal user testing, etc). It also included a number of practical tips on how to run a usability test successfully in the real world.
In true Agile fashion, we immediately defaulted to proceeding iteratively and building in as much feedback as early as possible. So since the timing was great, we decided to use our own website as the subject of our first attempt at running more formal usability testing. In other words, testing our own new website would be the “skateboard version” of the new capability we’re creating.
So we downloaded EvoCam, spent half a day playing around with its settings and thinking about how we’re going to do the actual test – should we have a traditional “test room” for the facilitator and participant, with space for observers and a scribe behind a one-way mirror? Some of our proposed participants were executives. Are we really going to ask these high-powered people to spend half a day going out of their way to do us a favour that it must seem we could ask anyone else for? It just didn’t feel right – we needed a better option.
Should we rather stream the video feed live from EvoCam to the observers and scribe? Then at least we could be mobile, and go to the participant. This would also allow observers to watch remotely, i.e. no need for everyone to commute halfway across the peninsula to be present at a 40-minute user test. This felt like it fit better with the way Polymorph works, but we ran into networking difficulties. EvoCam can run an internal web server that gives observers a URL to connect to in order to see the video & hear the audio: however, on some browsers it’s very slow and laggy – on others the audio stream just doesn’t work. In addition to which, it requires you to open a port on your router to work at all… which isn’t very practical to do at a client’s site.
At Polymorph, we are no strangers to working remotely, nor to the value of enabling asynchronous communication. Then we realized there isn’t any need for anyone to see the tests live. At Polymorph, we are no strangers to working remotely, nor to the value of enabling asynchronous communication. So why should this be any different? In the end, we used EvoCam to record the feed from the webcam and desktop to the laptop’s hard drive – which means only the participant and facilitator have to be present at the same time. As soon as the video is uploaded (we used Google Drive), any number of observers can download it and “observe” the test. They can then use any of a number of tools (we picked Trello for our first attempt) to record their observations for later discussion and analysis. Here is what it looked like:
So now we were happy with the tools we had to perform the tests. We shortlisted 6 participants as follows:
- 2 Employees (in other words, people with a technical background who know Polymorph from the inside)
- 2 Existing clients (people with a technical background who know Polymorph well, but from the outside)
- 2 Acquaintances (in other words, people with a technical background who don’t know Polymorph well at all)
Since we were going to run the test using Evocam to record video to disk, there were almost no infrastructure requirements that would stop us testing anywhere. We had the great advantage that any of the 5 project team members could facilitate the tests. Since we are as scattered around the peninsula as our test participants, it was reasonably easy to link each test participant with a facilitator in such a way that neither had to give up too much time to unproductive commuting. Even so – one of the big things we learned is the reason why there exists such a thing as professional recruiters. Setting up and confirming appointments for 6 busy professionals to meet with one of us (who are also running other projects as part of our daily lives) took up a lot of time and mind space to accomplish.
...one of the big things we learned is the reason why there exists such a thing as professional recruiters.
Once the tests started happening, the next step was to review the footage and analyse the results. Initially, the idea was that all 5 of us would review each of the 6 videos, and make comments on all of them, but we soon realised that we had too much feedback (which was thrilling, because it was early proof of the value of the process). We dialled it right back to having each one in the team watching a different video and making comments (on Trello cards), and the facilitator also made comments from memory. We spent some time before the team analysis meeting to categorize the feedback cards and throw out duplicates which left us with 80 Trello cards. Still an oversized body of input for a 90-minute meeting, but we dove in lean-coffee style, and by the end we had a bird’s-eye view of all of the comments, AND discussed half a dozen in depth.
We were able to identify a number of small changes that could be implemented immediately, as well as 2 internal workshops that needed to happen around two specific pages.
Some key lessons we learned about the user testing process itself:
- Use a professional recruiter to find test participants. Doing this is a massive time-sink, and if the person doing the recruiting is also involved in other projects, it will become nearly impossible for them to do everything well.
- Someone too deeply involved in the software itself will struggle to run an objective test.
- How do we decide where to draw the line between cost and the value generated by doing more testing? There is no single answer for this, but it’s a valuable question to ask for each project.
- Your recording software can and will crash at some point during 6 user tests. This will probably mean that you lose the recording of that test. Make enough notes!
- Asynchronous testing is the way to go for us (i.e. we record video and the observer(s) watch at their convenience). It makes everything simpler for all involved.
- There may be great value in having a collaborative observation & discussion session, with several team members physically present. We don’t know yet whether this will add enough value to be worth the cost, and how to decide on when it is needed and when it isn’t.
- Using Trello to collect & organize feedback works well right up until you try to run a lean coffee off of the Trello board. Trello doesn’t do well with lean coffee discussions – so we will use a different tool next time.
- User Testing fits very well in iterative development – you lose a key advantage if you only test once on a project. Instead of doing that, even small projects should have at least 2 rounds of user testing, even if they’re simplified greatly to fit into a small budget.
Although we have not found the answers to all the questions that came out of our learning, we are well on our way to offering a more complete product development process.
