One of the challenges of running usability testing on mobile devices is how to record a video of the session. On desktops recording sessions is easy. Applications like Morae or Silverback have been allowing us to do screen captures with a picture-in-picture window of a participant’s face for years now. They simply use the mouse pointer and webcams to capture of full picture of what the user is doing.
On mobile devices it is not so easy. On mobile devices it’s just as important to see what the users hand/fingers are doing as the targets they are tap on. When you combine this with the movement of handheld devices recording a usability testing session becomes somewhat more complicated.
On a recent project we ran a round of formal usability testing, where we ran ten participants through a competitor comparison of apps. To record these sessions we put together a somewhat ad-hoc, yet highly effective recording setup that used this option of mounting a camera of the device. Here’s an overview of how we made it work, including:
- Choosing a video recording approach
- The pop-up usability lab
- The camera mount – Mr Tappy
- The cameras
- Recording software
- Remote broadcasting of the sessions
- Lighting
- What would we change next time?
Choosing a video recording approach
There are 3 current options that I’ve seen used to capture a session on a mobile device:
- Attach a camera to the device itself.
- Mount a camera above the space where the device will be used.
- Record a screen capture of the device.
Each of these has their relative pros and cons:
| Option | Pros | Cons |
| Attach a camera to the device itself |
|
|
| Mount a camera above the space where the device will be used |
|
|
| Record a screen capture of the device |
|
|
Personally I prefer the first option of attaching a camera to the device itself. In most situations having a clear view of the screen and the hands outweighs the participant’s feeling a little bit weird about the camera hanging over the device.
The pop-up usability lab
Having chosen to go with the option of mounting a camera on the device, we set about turning two meeting rooms in to a full-featured mobile device usability testing lab.
Here are some pictures of what we put together.
 The testing lab room setup and ready to go. |
 A session in progress, with the participant using the phone and moderator guiding the session. |
 The participant’s view of the device and recording equipment. |
 The testing lab and observation room side-by-side. |
 A note taker observing the video feed from the next room, taking notes on post-its. |
 The video output, the device view with a ‘picture-in-picture’ view of the participant holding the device. |
What’s happening in the room?
The testing lab was made up of the following equipment:
- Two cameras; one mounted on the device, the other in front of the user capturing their face and them holding the device.
- Both cameras feeding in to a laptop, which was recording the session. We used a MacBook pro for this setup. This machine comfortably dealt the processing load for this setup. It also had the 2 x USB inputs and 1 x HDMI output that we needed to connect to the cameras and TV.
- Laptop sitting to the side where the facilitator could see it. This meant that the facilitator could sit back and watch what the user was doing through the screen rather than having to always be looking over the participant’s shoulder.
- HDMI output cable running to a large TV in the room next door broadcasting the screen share and audio to observers.
The camera mount – Mr Tappy
The centerpiece of the setup was our camera mount – Mr Tappy. Mr Tappy is a simple mount, which you can attach a mobile device to, with an adjustable arm that extends over the top of the device to attached a camera to.
Meet Mr Tappy:
The best part about Mr Tappy is that he can be used for all types of devices; phone, tablet, iOS, Android, Windows. The mount is flexible enough to accommodate any sized device. The only restriction is needing to attach some velcro to the back of the device for Mr Tappy to secure to.
Mr Tappy is the brainchild of Nick Bowmast, who I can happily say I used work with a few jobs ago. He took the idea that came from an old improvised mobile testing sled made out of perspex and turned it in to a high-quality product designed specifically for this purpose. I highly recommend it.
One thing we learned was that it’s important to pick up and hand the device to the participant so that they start with it in their hand. The attached camera mount does make participants reluctant to handle the device. Handing it to them when you first ask them to start doing something on it helps get it in their hand and using it.
Still, a few participants just put it back down on the table and used it there. In our context it wasn’t too much of an issue as using a phone whilst it sits on a tabletop is not an uncommon behavior anyway. Otherwise, it’s important to be aware of it that impacts what you are trying to test.
The cameras
For the camera mounted on the device we used the iCubie camera that Mr Tappy recommends. It’s the smallest and lightest camera that could be found at the time.
It’s downside is that it’s not super high-resolution. Therein lies the trade-off. We considered using a higher quality camera but they all just became too big and bulky to use on the mount. Instead we decided to concede the quality for being less obtrusive.
For the other camera we had on the table camera focused on the participant we used a Logitech HD Pro Webcam C910. It was less important which specific camera we used for this. We used the Logitech as we had one on hand and knew it did good quality HD video and audio.
Recording software
The recording software was where we had to improvise the most. All the applications out there at the moment were either too expensive to bother with, didn’t allow us to capture the output from two different webcams at one time, or required too much post-production work to sync video feeds.
In the end we took an ad-hoc approach of simply having the two camera feeds open on the desktop using separate applications, then using a third application to do a screen capture of the whole desktop. Whilst not the prettiest video outputs you’ll ever see, it captured everything we needed it to.
Where it got tricky was finding the right combination of apps that didn’t have conflicts and worked with the various cameras. We had to do a bit of trial and error with different media players, plus trouble-shooting to make it work but we eventually found the right combination that worked. Here’s what we had running:
- Device camera feed - Quicktime 7 Pro
- Participant camera feed – Quicktime 10
- Screen capture – Quicktime 10
- Audio output for HDMI feed to TV – LineIn
We had to ensure we left enough hard drive on the laptop to record 50-100gb of video. Full resolution videos of an hour session typically ended up at 30gb before we optimised them to a lower level. We kept an external hard drive handy to off-load videos.
We also had two laptops that setup so that we could quickly swap them if necessary. Occasionally this was necessary if a video was taking a while to be processed and saved.
Remote broadcasting of the sessions
The setup of the office, with two rooms next to each other and a big TV in one, worked out really nicely for our setup. An alternative option that we were also considering was to broadcast the session to a separate location using Skype (or similar) to screen share the video and audio to another location.
Lighting
Reflection/glare from the ceiling lights can be an issue in seeing the screen of the device through the camera mounted on Mr Tappy. To avoid this we turned off the lights directly above user (we unscrewed the fluorescent lights).
This creates a problem where the camera can’t deal with the contrast between the screen brightness and the dull background. To compensate for the lack of light we setup a lamp to the side of the user, which lit up their hands. We also turned down the device screen brightness to balance out the difference. The balance between all this needs to be tweaked depending on the exact room.
The unintended side effect of all this was to create a somewhat ‘intimate’ setting in the room for participants to walk in to. We made sure we called it out and explained why so that participants weren’t too put off by it.
Controlling the laptop and video feed
Given that the two rooms were directly next-door we connected a Bluetooth keyboard and mouse to laptop, which we put in the observation room. This allowed the observers to run the laptop and control the video (i.e. starting/stopping and trouble shooting), which helped reduce the load on the moderator in the room.
What would we change next time?
Overall we were very happy with the whole setup. I’d happily take the same approach and recommend it to others – hence why I just put this summary together.
Mr Tappy worked well as the camera mount and solved the problem of a camera mount. I’d still keep looking for a higher resolution camera than the iCubie though.
I’d also keep my eye out for a better media player solution. In fact, as I was putting this post together someone recommended SecuritySpy to me. Despite being made for a different purpose, looks like it would do the trick for a small license fee.
Otherwise, here are some of the approaches I’ve found that other people have used.
A surprisingly enjoyable errand
I just had the surprising pleasure of having to stop in to the Post Office over lunch to send a parcel. This is not an errand I would normally describe a pleasure but this is the first time I’ve seen the Brisbane GPO since their store layout redesign.
The product shelving layout, self-serve kiosks (using the same hardware as supermarkets), easy to follow interactive instructions and well labelled drop-boxes saw me happily serve myself without the need for assistance from any of the floating staff. And all without having to wait in a queue for 20mins.
I’m now looking forward to the next time I have to go old school and use the snail mail.
Kudos to Post Office design team for a job well done.
The Art of Presenting a Concept
I recently had the unpleasant experience of watching a badly presented design concept.
It was unpleasant because of my negative knee-jerk reaction to it. Before I control my mouth I started nit-picking minor details, like the choice of font and styles, when all they were after was feedback on the name and the concept.
At the time I was rationally aware that I wasn’t being helpful, yet I couldn’t stop.
The concept itself wasn’t bad; in fact I really liked it. It was just the pitch itself that was badly done.
It was really challenging experience for me because I’m usually on the other side, presenting the concept. I know how frustrating it is to put a lot of work in to a design; considering the proposition and the target audience, how it is represented in the content and structure of a page, and then how the colours and fonts provide just the icing and cherry on top of a well thought out package.
I know what it is like to do all this work and then have someone fixate on just the superficial skin of the concept – it’s really annoying. And yet here I was doing it myself.
The experience got me thinking about what happened. I didn’t want that irrational reaction to be the only feedback they got from me. Thankfully it was an internal piece of work, with colleagues I know well. This gave me the opportunity to mull it over for an hour before getting back in touch and telling them that I actually really liked their concept and that they just had to work on their pitch.
The happy ending to the story is that they did a presentation version 2 the next day and they blew everyone away. Rather than walking out feeling negative, the audience were all really excited by the concept and went away feeling pumped.
What did they do wrong? They made 3 common mistakes:
- They didn’t explain what the design that they were showing us was for and the context of how it would be used. They assumed we knew what they were doing and why.
- They jumped straight in to the solution without explaining the journey of how they came to that particular solution. In the interests of time they just cut to the chase and presented the final idea.
- They presented a draft concept in a high-fidelity polished format, even though it was still a work in progress.
Here’s what they did different the next day which made it far more successful:
- They booked a longer time and took us through a more detailed presentation. They provided all the background, context, and alternatives ideas they considered before getting to the solution. Even though the audience already knew bits and pieces of this story it made sure all the gaps were filled in.
- They hand drew their concept and presented that instead. If you want someone to focus on the underlying concept (not the fonts and colours) present ideas in the same level of fidelity as the idea – rough drafts. You can’t comment on fonts and colours if they aren’t any there. We do a lot of lo-fi prototyping for exactly this reason.
The best advice I’ve come across on how to present a design concept was in Matthew Frederick’s book 101 Things I Learned in Architecture School, on page 57 he gave this simple set of instructions:
An effective oral presentation of a studio project begins with the general and proceeds toward the specific.
- State the design problem presented.
- Discuss the values, attitude, and approach you brought to the design problem.
- Describe your design process and the major discoveries and ideas you encountered along the way.
- State the parti, or the unifying concept, that emerged from your process. Illustrated this with a simple diagram.
- Present your drawings (plans, sections, elevations, and vignettes) and models, always describing them in relationship to the parti.
- Perform a modest and confident self-critique.
Never begin a presentation by saying, “Well, you go in the front door here” unless your goal is to put your audience to sleep.
Put your money where your mouth is – hire a User Experience Designer
A frustration point for me recently has been companies that list the user experience as their most important priority, yet don’t ever do anything about it.
It’s great that people are starting to recognise the importance of the Experience Economy and that a top notch user experience is a must have for company who sees itself as a serious contender in a market.
As a consultant I tend to get around a few different companies and in the last year or two I’ve ended up doing quite a few project kick-offs in different places. Pretty much without fail, every company has listed creating a great user experience as top priority.
When we run through our project Trade-Off sliders User Experience is usually listed as the least negotiable factor in a project.
Yet, more often than not these companies aren’t willing to put in the effort or resources required to build a great user experience. They don’t do regular user research and have limited direct contact with their customers/users, don’t have any internal design capabilities, and don’t ever actually do anything other than saying “yes, it’s our top priority!” or “It has to have a great user experience“.
Despite all their verbiage, they are really no more than a Level 2 on the UX Maturity scale: They recognise UX is important but it receives little funding.
Courtesy of Planning your Strategy, Renato Feijó - Johnny Holland Magazine
I regularly get people walking past our sketchboards and design walls commenting on how exciting the UX driven approach is. They get excited by our UX tools and the vision they create. Then they walk off in wonderment thinking about how great it would be for their team to work like that – and do nothing about it.
Less talk, more walk.
Taking a user centered approach to building a product doesn’t just happen by accident. It takes effort and expertise to do these things - but any team can if they have the right mix of skills.
Stop outsourcing the user experience design of your product – your most important asset – to external design agencies.
Bring design capabilities in to your organisation. Hire a UX Designer, someone who specialises in creating awesome experiences.
They will help you get outside of the building and talk to someone who doesn’t have an employee number. They will help you drive projects with a focus on what the user experience will be, not just what 100 page requirements document says. They will help you build a more mature UX practice.
If you make excuses such as “We don’t have the resources” or “We can’t increase our head count” then you’re simply not prioritising your user experience over other things so stop claiming that you do.
Day of the yak shave
Some times you just have one of those days where you need to accept that there is a hairy yak standing in front of you and get on with removing it’s hair…
One of the most useful terms I’ve learnt since working side by side with Devs is Yak Shaving.
Yak Shaving: Any seemingly pointless activity which is actually necessary to solve a problem which solves a problem which, several levels of recursion later, solves the real problem you’re working on.
Today has been a day focussed on becoming fully integrated with the Borg that I’m currently working at so that I can access all the internal systems/tools. At the end of the day I have far more hairless yaks to show than tasks completed.
Here’s the long version of the yak shaving story, courtesy of Seth Godin:
Yak Shaving is the last step of a series of steps that occurs when you find something you need to do. “I want to wax the car today.”
“Oops, the hose is still broken from the winter. I’ll need to buy a new one at Home Depot.”
“But Home Depot is on the other side of the Tappan Zee bridge and getting there without my EZPass is miserable because of the tolls.”
“But, wait! I could borrow my neighbor’s EZPass…”
“Bob won’t lend me his EZPass until I return the mooshi pillow my son borrowed, though.”
“And we haven’t returned it because some of the stuffing fell out and we need to get some yak hair to restuff it.”
And the next thing you know, you’re at the zoo, shaving a yak, all so you can wax your car.
Dear Vodafone, I’ve just fallen out of like with you
Dear Vodafone
I’m disappointed that I had to write this letter, but due to my recent experience I would like to break up with you. You have failed me in my time of need and have thereby lost my faith and affection.
My iPhone 4 screen shattered last week. I dropped it when putting it in my pocket and it landed at the wrong angle on concrete. Everything else still works, it’s just the screen that shattered.
No I don’t use a case. I’m morally against covering up a beautiful physical product with an ugly case. A well-designed product shouldn’t need one. I often wonder what Jonathan Ive thinks about people covering up his beautiful industrial design work with a crappy piece of rubber…but that’s another conversation.
Vodafone, why I blaming you for the iPhone’s flaws is because I’ve been paying $10/mo on phone insurance for the last couple of years which is supposed to take care of me in these situations. This is the first time I’ve needed to use the insurance and I’ve found that it is a completely disappointing product.
Let me tell you about my disappointing experience…
I broke the phone on a weekend. That Monday I went in the Vodafone store in the Queen st Mall to figure out what I needed to do to get it fixed. My assumption was that you’d be able to help me fix it there on the spot somehow. It would probably involve sending it off somewhere to be fixed, but it would only take a day or two to get done. In the mean time you’d supply me with a loaner so that I wouldn’t be left without my crucial communication tool (read ‘extension of my personality’).
It was a busy time for me, I had a crazy busy work week and was going away for the upcoming weekend. I didn’t have the time to follow up and certainly couldn’t afford to be without a phone for long.
When I showed my poor phone to the friendly girl in store she told me that to get it fixed I had to go on the Vodafone website and submit an insurance claim. Once the claim had been processed I’d be given a claim number that I could then bring back in to the store and then they would help me. I asked if I could sit down and do it now on the computer she was showing me on. Unfortunately, there was no point as the claim would take approximately 5 working days to be processed….5 working days.
What was I supposed to do in the mean time?
Now my phone was still just functional. It ran fine and I could just make out what it said through cracks in the shattered glass. But what would have happened if it didn’t work at all? I’d just have to wait patiently for 5 days while someone processed the paperwork?
She told me the alternative was to go to an Apple store and get them to do it. It would be more expensive (about $225 compared than $125 for the insurance excess) but it would be doe while I wait.
But then why have I being paying for insurance on my phone all this time?
Flabbergasted, I went back to my office to submit the claim online as a matter of principle.
5 days later, I received an email from the insurance company checking my phone number. Due to the poor usability of their web form I accidentally left out a digit of my phone number. That was on a Thursday, I then had to wait another 4 days to Monday before they replied with my claim number.
Today I went back in the store with my claim number, the lovely girl told me that I’d have to then send off my phone to the company to be fixed. She said it would probably take about 5 days, but wasn’t sure because they don’t usually see the phones again in store as they get sent straight to your address.
There was no automatic offer of a temporary replacement phone to keep me going in the mean time.
I felt sorry for this poor girl, she seemed genuinely pained when I told her that I would prefer to go away as a disgruntled and unhappy Vodafone customer to sort it out myself at an Apple store. She was perfectly friendly and as helpful as she could be. I almost wish that Vodafone would apologise to their staff for putting them in to this position.
I spent a day in a Vodafone store once as part of some customer research. Usually the staff are always hugely helpful and friendly, they are highly affective (and creative) at fixing the problems that they have systems access to sort out. But they are often handcuffed in key areas and end up copping abuse from customers for things they have no control over. They are often left with no choice but to tell customers to call the call-centre to get some things done. From a customers perspective that makes no sense. They are Vodafone staff, working in a Vodafone store, wearing Vodafone branded clothing, why can’t they help me with everything?
This all stems from a bad product decision. Whoever is the product manager of the phone insurance product deserves the blame for this. They chose and use a poor 3rd party vendor who offers an extremely disappointing customer experience.
Either way, as a customer I don’t care. I pay Vodafone for my insurance and am extremely disappointed with how you are (not) helping me in my time of need.
The insurance is provided by a 3rd party company, but it is still your responsibility. I am a Vodafone customer, not a Marsh customer. I pay you $93 per month for your service, part of which is for insurance in case something happens my phone.
I’ve been a content Vodafone customer for a few years now. Your network issues that everyone always calls you ‘Vodafail’ for don’t bother me too much. I’m a city dweller where I get good enough coverage. Whenever I leave the city I’m generally quite happy to escape phone coverage. I’m happy paying less for the inferior network.
I’ve also always had a soft spot for Vodafone after having worked there for a few months in the Online User Experience Team. They used to be a clear step above their competitors on the online customer experience front, which is more important to me (as a side note this has changed somewhat nowadays as Telstra has managed to catch up quite a lot).
I used to be a net promoter of yours. I would happily give you an 8 on the NPS. Despite all the other complaints I had heard, I was always quietly happy with my service.
Now I have become a detractor. Now I’m left bitching about my experience and how bad Vodafone is to anyone who will listen. I would be lucky to give you a NPS rating of 4 at the moment. In my time of need you were not there for me.
Now I can’t wait for my contract to finish so that I can play the market a little and see what other providers out there are like.
Vodafone, I was hoping that this day wouldn’t come but unfortunately now it has. Unfortunately, it’s not me, it’s you.
I would like to break my ties with. I hope you can understand and do something to change your insurance product in the future, even if it’s just for the benefit other customers other than me.
Yours sincerely
Ben Melbourne
Defining Design Thinking
Listening to an interview with Sir George Cox, chairman of the UK’s Design Council, on ABC Radio’s By Design this morning, he gave a great definition of Design Thinking – something that I often struggle to articulate in a simple way.
This is my paraphrased version of what he described. Yes, it’s multipart, but to me that is what makes it more meaningful as it explains the different elements that make up Design Thinking:
- Creativity is coming up with new ideas.
- Innovation is introducing new ways of doing things.
- Design is channelling creativity for a purpose.
- Designers are people who have been trained to reframe problems and approach them from different angles.
- Design Thinking is a way of using design methodologies to find new and innovative solutions to problems.
.
User Experience & Interaction Designer. Researcher. Design Thinker. Ideator. Collaborator. Visual Thinker. Sketcher. Agile Believer. 
