Skip to content

Recording mobile device usability testing sessions – guerrilla style

April 9, 2013

One of the challenges of running usability testing on mobile devices is how to record a video of the session.  On desktops recording sessions is easy. Applications like Morae or Silverback have been allowing us to do screen captures with a picture-in-picture window of a participant’s face for years now. They simply use the mouse pointer and webcams to capture of full picture of what the user is doing.

On mobile devices it is not so easy. On mobile devices it’s just as important to see what the users hand/fingers are doing as the targets they are tap on. When you combine this with the movement of handheld devices recording a usability testing session becomes somewhat more complicated.

On a recent project we ran a round of formal usability testing, where we ran ten participants through a competitor comparison of apps. To record these sessions we put together a somewhat ad-hoc, yet highly effective recording setup that used this option of mounting a camera of the device. Here’s an overview of how we made it work, including:

  • Choosing a video recording approach
  • The pop-up usability lab
  • The camera mount – Mr Tappy
  • The cameras
  • Recording software
  • Remote broadcasting of the sessions
  • Lighting
  • What would we change next time?

Choosing a video recording approach

There are 3 current options that I’ve seen used to capture a session on a mobile device:

  1. Attach a camera to the device itself.
  2. Mount a camera above the space where the device will be used.
  3. Record a screen capture of the device.

Each of these has their relative pros and cons:

Option Pros Cons
Attach a camera to the device itself
  • Stable view of the screen.
  • You can see the participant’s hands and finger movements.
  • Device agnostic. Device can be quickly and easily swapped as they don’t require anything to be locally installed.
  • The user has to overcome the weird feeling of an unusual camera mount attached the device.
  • The physical mount can get in the way of form factors or the way participants would naturally hold a device
  • Screen usually has slight perspective tilt due to the camera angle.
  • Camera recording resolution has to be balanced against physical weight of the camera mounted on the device.
Mount a camera above the space where the device will be used
  • You can see the participant’s hands and finger movements.
  • Device can be quickly and easily swapped.
  • The user can freely pick up and hold the device in a natural way.
  • Device agnostic. Device can be quickly and easily swapped as they don’t require anything to be locally installed.
  • Unstable view of the screen depending on how much the user moves the device around in their hand.
  • User has to keep the device within a certain area bounded by the camera view.
Record a screen capture of the device
  • The user can freely pick up and hold the device in a natural way.
  • High quality screen capture.
  • Little limitation on how the user needs to hold the device.
  • No visibility of the participant’s hands and finger movements.
  • Requires screen capture software to be run on the device, which is currently inconsistent between platforms.
  • Screen capture software can overload some devices.

Personally I prefer the first option of attaching a camera to the device itself. In most situations having a clear view of the screen and the hands outweighs the participant’s feeling a little bit weird about the camera hanging over the device.

The pop-up usability lab

Having chosen to go with the option of mounting a camera on the device, we set about turning two meeting rooms in to a full-featured mobile device usability testing lab.

Here are some pictures of what we put together.

The testing lab room setup and ready to go.

The testing lab room setup and ready to go.

A session in progress, with the participant using the phone and moderator guiding the session.

A session in progress, with the participant using the phone and moderator guiding the session.

The participant’s view of the device and recording equipment.

The participant’s view of the device and recording equipment.

The testing lab and observation room side-by-side.

The testing lab and observation room side-by-side.

A note taker observing the video feed from the next room, taking notes on post-its.

A note taker observing the video feed from the next room, taking notes on post-its.

The video output, the device view with a ‘picture-in-picture’ view of the participant holding the device.

The video output, the device view with a ‘picture-in-picture’ view of the participant holding the device.

What’s happening in the room?

The testing lab was made up of the following equipment:

  • Two cameras; one mounted on the device, the other in front of the user capturing their face and them holding the device.
  • Both cameras feeding in to a laptop, which was recording the session. We used a MacBook pro for this setup. This machine comfortably dealt the processing load for this setup.  It also had the 2 x USB inputs and 1 x HDMI output that we needed to connect to the cameras and TV.
  • Laptop sitting to the side where the facilitator could see it. This meant that the facilitator could sit back and watch what the user was doing through the screen rather than having to always be looking over the participant’s shoulder.
  • HDMI output cable running to a large TV in the room next door broadcasting the screen share and audio to observers.

The camera mount – Mr Tappy

The centerpiece of the setup was our camera mount – Mr Tappy. Mr Tappy is a simple mount, which you can attach a mobile device to, with an adjustable arm that extends over the top of the device to attached a camera to.

Meet Mr Tappy:

MrTappy-005 MrTappy-003 MrTappy-004 MrTappy MrTappy-002 MrTappy-001

The best part about Mr Tappy is that he can be used for all types of devices; phone, tablet, iOS, Android, Windows. The mount is flexible enough to accommodate any sized device. The only restriction is needing to attach some velcro to the back of the device for Mr Tappy to secure to.

Mr Tappy is the brainchild of Nick Bowmast, who I can happily say I used work with a few jobs ago. He took the idea that came from an old improvised mobile testing sled made out of perspex and turned it in to a high-quality product designed specifically for this purpose.  I highly recommend it.

One thing we learned was that it’s important to pick up and hand the device to the participant so that they start with it in their hand. The attached camera mount does make participants reluctant to handle the device. Handing it to them when you first ask them to start doing something on it helps get it in their hand and using it.

Still, a few participants just put it back down on the table and used it there. In our context it wasn’t too much of an issue as using a phone whilst it sits on a tabletop is not an uncommon behavior anyway.  Otherwise, it’s important to be aware of it that impacts what you are trying to test.

The cameras

For the camera mounted on the device we used the iCubie camera that Mr Tappy recommends. It’s the smallest and lightest camera that could be found at the time.

It’s downside is that it’s not super high-resolution. Therein lies the trade-off. We considered using a higher quality camera but they all just became too big and bulky to use on the mount. Instead we decided to concede the quality for being less obtrusive.

For the other camera we had on the table camera focused on the participant we used a Logitech HD Pro Webcam C910.  It was less important which specific camera we used for this. We used the Logitech as we had one on hand and knew it did good quality HD video and audio.

Recording software

The recording software was where we had to improvise the most. All the applications out there at the moment were either too expensive to bother with, didn’t allow us to capture the output from two different webcams at one time, or required too much post-production work to sync video feeds.

In the end we took an ad-hoc approach of simply having the two camera feeds open on the desktop using separate applications, then using a third application to do a screen capture of the whole desktop. Whilst not the prettiest video outputs you’ll ever see, it captured everything we needed it to.

Where it got tricky was finding the right combination of apps that didn’t have conflicts and worked with the various cameras. We had to do a bit of trial and error with different media players, plus trouble-shooting to make it work but we eventually found the right combination that worked. Here’s what we had running:

  • Device camera feed - Quicktime 7 Pro
  • Participant camera feed – Quicktime 10
  • Screen capture – Quicktime 10
  • Audio output for HDMI feed to TVLineIn

We had to ensure we left enough hard drive on the laptop to record 50-100gb of video. Full resolution videos of an hour session typically ended up at 30gb before we optimised them to a lower level. We kept an external hard drive handy to off-load videos.

We also had two laptops that setup so that we could quickly swap them if necessary. Occasionally this was necessary if a video was taking a while to be processed and saved.

Remote broadcasting of the sessions

The setup of the office, with two rooms next to each other and a big TV in one, worked out really nicely for our setup. An alternative option that we were also considering was to broadcast the session to a separate location using Skype (or similar) to screen share the video and audio to another location.

Lighting

Reflection/glare from the ceiling lights can be an issue in seeing the screen of the device through the camera mounted on Mr Tappy. To avoid this we turned off the lights directly above user (we unscrewed the fluorescent lights).

This creates a problem where the camera can’t deal with the contrast between the screen brightness and the dull background. To compensate for the lack of light we setup a lamp to the side of the user, which lit up their hands. We also turned down the device screen brightness to balance out the difference. The balance between all this needs to be tweaked depending on the exact room.

The unintended side effect of all this was to create a somewhat ‘intimate’ setting in the room for participants to walk in to. We made sure we called it out and explained why so that participants weren’t too put off by it.

Controlling the laptop and video feed

Given that the two rooms were directly next-door we connected a Bluetooth keyboard and mouse to laptop, which we put in the observation room. This allowed the observers to run the laptop and control the video (i.e. starting/stopping and trouble shooting), which helped reduce the load on the moderator in the room.

What would we change next time?

Overall we were very happy with the whole setup. I’d happily take the same approach and recommend it to others – hence why I just put this summary together.

Mr Tappy worked well as the camera mount and solved the problem of a camera mount. I’d still keep looking for a higher resolution camera than the iCubie though.

I’d also keep my eye out for a better media player solution. In fact, as I was putting this post together someone recommended SecuritySpy to me. Despite being made for a different purpose, looks like it would do the trick for a small license fee.

Otherwise, here are some of the approaches I’ve found that other people have used.

About these ads
4 Comments leave one →
  1. April 10, 2013 11:34 am

    Hi Ben, Nick, aka Mr. Tappy here,
    Wow, a comprehensive post, and you’ve obviously covered a lot of the same ground.
    If you sit people at a table (which is the standard setup for usability labs) … they will use the table.
    I suggest you lose the table, and replace the chair with a comfy armchair.
    I do this and find it simulates a more realistic behaviour.
    You need to be somewhat tolerant to users resting the device on a surface, or their leg, but at some stages I find myself asking the user to stand up, which forces them to hold the device and interact with it simultaneously, which is realistic, and presents a more challenging design constraint.

    And yes, it was great working with you way back then!

    Nick.

    • April 10, 2013 12:03 pm

      Of course, remove the table. Such a simple and obvious solution to the problem.

      The thing I love about the whole mobile testing lab setup is how much of an exercise it is in finding practical solutions to problems.

      Great tip, thanks Nick!

  2. June 13, 2013 3:56 am

    Hi Ben,

    I’d like to use one ofyour pictures in a book I’m finishing right now. You can read more about the book project at http://appdesignbook.com

    In your blog I found a picture from you and it’s just what I need to communicate about mobile user testing.

    The picture is this one http://benmelbourne.files.wordpress.com/2013/04/pop-up-mobile-usability-testing-lab-2.jpg

    The book will be only digitally published and also will be available a free version in our website.

    If you agree to share your rights on this picture, of course your name will be next to it in a caption. As it’s an independent project is the best I can offer :)

    I’d appreciate your answer even if it’s a no.

    Thank you in advance.

    • June 13, 2013 4:25 pm

      Hi Jose

      You’re welcome to use the image. I appreciate the credit and you asking permission.

      Best of luck with the book
      Ben

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 297 other followers

%d bloggers like this: