Skip to content

Practical Product Innovation

April 26, 2015
broken lightbulb

Today, constant innovation defines our marketplace. Businesses must respond to customer expectations for better digital experiences.  How do leading organisations launch successful new products and respond rapidly to external change? How do they move beyond the desire to innovate to actively be innovative every day?

“Innovation” is such a big word that people often overcomplicate. They put too much emphasis on producing the “next big innovation” and become attached to their ideas and outputs. Instead, the opposite is required. To be innovative, you need to be responsive and adaptable. You need to be willing to try an idea, assess, adjust, and continue moving forward.

These concepts of learning by doing, failing fast, and adapting to change are all core Agile practices. They are the reason why iterative development exists, and the reason it can be so effective.

These proven Agile principles and techniques can enable product innovation. Using lean thinking, fast feedback cycles, and iterating, you can consistently – and rapidly – bring new ideas into the market.

What is innovation?

The cliché view of innovation is a genius inventor having a light bulb moment. It rarely happens that way. Instead, most innovations are a culmination of small ideas incrementally built upon other ideas, often in unpredictable ways. Steven Johnson describes this as the Hummingbird effect. He says:

Change almost always comes as a surprise because things don’t happen in straight lines. Connections are made by accident. Second-guessing the result of an occurrence is difficult, because when people or things or ideas come together in new ways, the rules of arithmetic are changed so that one plus one suddenly makes three. This is the fundamental mechanism of innovation, and when it happens the result is always more than the sum of the parts.

Where the rubber hits the road in a commercial context is when a company is able to translate a new idea into a product or service that customers are willing to pay for.

Thomas Edison wasn’t the first person to invent a light bulb. He just improved it. He was the first astute businessperson to create a longer lasting light bulb and put in place a power grid. When combined together this enabled people to use a light bulb in their homes, whilst paying him money for the service.

How do we do innovative things?

All the well-known historical examples of innovation are the result of many hours and years of hard work. They resulted from smart people rolling up their sleeves and taking practical steps forward. Here are some small practical techniques that you can do today that will produce innovative results.

Understand and reframe the problem

A lot of people just jump straight to designing and building a product, which stifles innovative thinking. You need to ensure you understand the problem you are trying to solve. Start by understanding your customers’ needs. Go out and talk to them. Observe them in their own context. See what they say or do. Observe the problems they face. This will challenge your assumptions about their needs.

Identify and define what your challenges are. Take the focus away from the solution for a while. “How might we…” statements are a great tool for this. Using the light bulb as an example: “How might we…allow people to see at night”.

These are deliberately open statements. They explain the goal, not a prescriptive solution. They will often reframe your understanding of the problem and open up new possibilities.

Generate lots of ideas

Once you understand the problem or the challenges that your business is facing, you can start looking for ways to solve them. A big blocker at this stage is getting stuck on one cherished idea. Whether it’s one person’s pet idea or just the way a company has always done things, it’s the same result – it stops you trying new things.

They key to doing this without becoming fixated on one idea is to generate as many as you can. Surround that one idea with lots of other good ideas.

Collaborative Design is a great tool for generating ideas. This involves getting everyone in a room together sketching, sharing, and creating a vision. This process embraces divergent and convergent thinking, which is a core part of the creative process. This involves ‘going wide’ and exploring lots of ideas, then converging down to one or two refined ideas.

Design is a team sport. Get everyone involved. Designers, developers, product owners, business stakeholders, and executives. Everyone has good ideas to contribute.

Turn your ideas into insights

It’s important to recognise that ideas are just thoughts. Smart business people don’t spend their money on ideas. They test and validate them, proving which ones work and which ones don’t. Turn your ideas into knowledge. You need to prove that customers are willing to pay money for your ideas, as quickly as possible.

There is a plethora of rapid prototyping tools out there. You can create interactive prototypes in a matter of hours. This means you can generate an idea in the morning and have it in the hands of a customer that afternoon.

Be deliberate about what you test

There are many different layers and levels that make up a user experience, some more obvious than others. People often jump straight to testing the more obvious UI levels like the appearance or interaction. When creating a new innovative product, it’s important to test the proposition or concept first.

You need to start with the questions: “Do people value my proposition?” and “Are people willing to pay money for my concept?” Different prototyping techniques allow you to test different levels, either validating or killing them.

Iterate

Testing one idea usually inspires more ideas. Learning is an ongoing journey.  Continue to test and try new ideas as you build a product. These could be for improvements, additional features or an entirely new product.

Fuel the feedback cycle. The shorter the feedback cycle, the sooner you can act and adjust your direction. You should be aiming for days or weeks at most, not months or years.

Don’t be scared to fail

Failure is often seen as a dirty word. This paralyses companies and prevents them from trying something new. Failure is an opportunity to learn. Culture needs to be changed to embrace the opportunity that comes out of failing.

Take lots of small bets

Think big, start small. Take lots of small bets. Experiment in small, inexpensive ways. The other way to say this is to ‘fail fast’. The trick to failing fast is to do it in a constructive way. Do it as quickly. Don’t bet everything on one idea. Embrace the learnings.

Get it out there

Launching a live product is the ultimate way to get feedback. You can only test some ideas by putting them in the hands of paying customers.

Be ruthless about what your Minimum Viable Product includes. What is the smallest set of features you can get in the hands of customers? This isn’t about being perfect or profitable. It’s about how can we get our customers using our product and how we get their feedback.

Prepare to learn and adapt…quickly

Innovators in today’s digital world aren’t the ones hiding away in research and development labs, privately tinkering away on an idea for years. Nor are they the ones who take two years to build and launch a product to the market.

Innovators are the ones constantly trying new ideas. They are the ones who are hooked on the feedback cycle and can’t get enough of that learning fix.

Originally published at Agile Australia in April 2015.

UX Designers: Why are we Wasting Time?

April 18, 2015

UX designers, we need to change the way we work. We need to stop wasting so much time. We have a well-deserved reputation for retreating to creative studios for lengthy design phases – only choosing to resurface once we have perfected solutions that are pretty, but not necessarily practical. We need to end the pursuit of design perfection and focus on delivering value faster. We need to embrace the Lean movement and streamline the way we work.

The Toyota Production System is essential reading for anyone serious about improving their craft. Everyone can learn from the lean manufacturing principles that Taiichi Ohno and Toyota developed.

Along with Agile, the Lean movement has helped revolutionise the software industry. These practices have transformed digital technology from a monolithic beast, into a disruptive force. This new digital world is reshaping our lives for the better.

Lean is all about continuously striving to improve your effectiveness and efficiency. It’s a focus on delivering value to customers and reducing waste. The Lean UX movement is a set of techniques that have emerged out of a recognition that UX Design practices can and have to improve.

I’ve drunk the Kool-Aid on Lean. I wholeheartedly agree with and embrace the Lean principles. They have changed my mindset. I could never go back to my old way of working; designing in my ivory tower, oblivious to the larger process I was part of. I have learnt to to focus on the value I deliver rather than the techniques I use. I’m actually embarrassed looking back on some of the projects I worked on. Thinking about some of the pretty, yet pointless documents I spent hours crafting makes me cringe. They may have made the client happy at the time, but looking back now, I know there are far better ways to work.

The rise of the Lean UX movement is a formal recognition that our discipline has fallen behind the rest of the digital industry. Traditional UX practices waste a lot of time and money compared to newer, more collaborative ways of working that the digital industry has embraced.

To be fair, these wasteful practices are a historical legacy. We have traditionally had to focus on pixel perfection in big upfront design phases. Once handed over to developers, our designs were never to be tweaked again. Or at least, not until after they launched six months later.

Times have changed. Now designers are the ones guilty of choosing to work in isolation. We often hide behind the ‘creative process’, using it as a justification to separate ourselves from the larger context in which we work. Instead we should be including our business and technical partners in the creative process. We should be using our skills to facilitate collaboration and inspire creative thinking in everyone.

We need to take the focus away from stand-alone UX methodologies and place it on delivering business value.

Here are 10 signs to watch out for that your practices aren’t as effective, efficient or as collaborative as they could be.

1. You start mocking up a solution before defining the problem

People love jumping into solution mode. It’s easy to get excited by an idea and want to play around with it. However, it comes with a risk that you lock in the wrong solution, rather than being open to alternative ideas. You become a hammer looking for a nail.

This doesn’t have to be a long-winded process. Sometimes it can be a 5-minute conversation. Other times, it can be a project in itself. It is important to be deliberate about what you are setting out to achieve and choose your tools and approach to suit the purpose.

2. You have committed to a lengthy interview schedule before you’ve even spoken to one user

Detailed UX research is an expensive overhead that you can often – but not always – live without. When going through the messy problem definition stage, talking to one or two users can provide the insight required to leap forward into concept development. Other times, it opens up a Pandora’s box of other questions to explore. Plan for the opportunity to do some initial interviews. Then consider where to go next.

If you start every project with a statistically significant number of participants, you miss the opportunity to streamline the process.

3. Your research findings report contains 10+ pages of text

This is a great way to produce shelfware that no one will ever read. No amount of text or slides will ever replace the richness of observing your target audience first-hand. Take your client/team out in the field with you, or you’re greatly reducing the value of your research. Seeing a user point out the flaws in your product is the quickest way to convince a CEO to drop his pet feature.

4. The person who came up with your product strategy isn’t involved in building it

This is another great way to lock in the wrong solution. If strategy is seen as a separate, stand-alone activity, you lose the ability to pivot or adjust your concept as you learn.

5. You spend more than one week designing an idea before seeing a user try it

Great ideas are great, but proven insights are better. Over-investing in an idea before confirming what your customer wants it is taking a gamble. It’s setting yourself up to waste a whole lot of design effort. Putting some rough, hand-drawn sketches in front of a user can be enough to confirm a good idea or kill a bad one before you invest in it.

6. Your design mock-ups have version numbers

If you need version controls for wireframes, you’re spending too much time in the upfront design stage. You should iterate in functioning code. Design in the actual medium it will be experienced in.

Okay, sometimes version controls are necessary in large, distributed teams with more complex workflows. However, there is still a point where you need to reconsider your approach. If your version numbers hit double figures, that is the probably the right time.

7. You have perfected your solution in Photoshop before sharing it with a developer

Designers often defer bringing technologists into the process as long as they can, rather than including them from the start. Technology should become an enabler, not a blocker.

Work with developers. If you’re not already designing with them, you are just wasting time. You can never specify everything in a design mock-up or wireframe. You never know exactly how something will work until it is in working code.

Architecture, Industrial Design, Engineering, Fashion. These industries all have ways for the people who design things to collaborate with those who build them. This allows for the differences that emerge when going from paper to the actual medium. Digital technologies should be no different.

8. You don’t know the developer who built your design

See previous point. Add double the guilt – particularly if you complain about them bastardising your design.

How do you expect them to deal with the details/oversights that weren’t captured in a static design mock-up?

Note: The answer to that is not adding more annotations to wireframes.

9. You design UIs and don’t feel an enormous sense of guilt that you can’t code

If you are one of those unicorns that can do both, you can skip to the next point (but know that I am jealous).

I wish I had the capacity to learn how to code. I wish I could design in code. Just imagine how empowering it is to build your own design rather than relying on someone else’s skills. Unfortunately I cannot.

I console myself with knowing enough HTML to at least have a basic understanding of how a page functions in a browser. Without this knowledge, I couldn’t sit down and pair with a developer to tweak a functioning product. It also means I know enough to have a conversation and understand why the design detail I suggested would take so much effort to build. This is an important bridge to collaboratively get to a better, more efficient solution.

If developers roll their eyes every time you approach them, it’s probably because you make their life hard. Learn a little bit about code. It will help a lot.

10. You don’t care what time your day ends

Work smarter, not harder. Working long hours is not sustainable and leads to tiredness and poor quality work.

I love working with people who have a life outside of work. That means they have motivation to get a job done and get out of the building. This creates a focus on the outcome and a desire to be efficient.

We work in a world of commercial realities and limited budgets. If you don’t value your own personal time, chances are you don’t value your client’s time either.

Originally published at www.thoughtworks.com on April 1, 2015.

Learn about Design Thinking and Agile Planning – my Agile Australia 2015 Workshop

April 10, 2015

The role of designers is changing. The growing adoption of Agile development practices has created an opportunity for designers to take the lead. Design thinking skills allow us to take ideas and quickly turn them into clear product visions that put users first. By stepping up as project kick-off facilitators, designers can drive rapid, customer-centric product development and make stakeholder management far easier.

If you want to learn more, come join my Agile Australia 2015 workshop on 16 June at the Hilton Sydney.

See detailed descriptions and register at www.agileaustralia.com.au/2015/workshops. Earlybird registration closes on Friday 24 April 2015, and get a further discount using our promo code, AA15-TW

Designing for Mobility 2014 Conference – I’m presenting

January 23, 2014

I’ve just been confirmed as a speaker at the Designing for Mobility 2014 Conference coming up on Friday 14th March in Sydney, Australia.

My talk will be about Building a bank’s biggest branch which will cover a case study of the Suncorp Mobile Bank App which our team launched in December after over a year’s worth of hard work. 

I’ll be co-presenting with Mr Toby Pritchard who I have been working closely with over the course of the project.

I’m looking forward to sharing the journey that we went on and some of the lessons we learnt along the way. Hopefully I’ll see you there.

Here’s the summary of our talk:

Building a bank’s biggest branch

Building an app is easy, right? 16 year-olds build million dollar apps in their garages every other day. But things get a little trickier when the app you’re building is for a bank.

Designing and coding an app with lots of cool new features is the easy part. Creating a whole new customer channel, that is effectively the bank’s most highly visited branch, comes with a whole different set of challenges including:

  • Shifting the businesses mindset from a one-off project to treating mobile as a first-class channel that needs to be supported and evolved on an ongoing basis.
  • Provide all the features that customer’s expect from web banking and deliver those in a lean fashion, with a mobile first approach.
  • Balancing security and fraud constraints whilst still making the app easy to use for customers.
  • Pushing the boundaries of traditional brand guidelines that didn’t translate to native mobile environments.

With a conservative financial institution that’s just beginning to understand how mobile is going to revolutionise its business, you have quite a mountain to climb before your designs can see the light of the app store.

This is the journey that Toby Pritchard (Suncorp) and Ben Melbourne (ThoughtWorks) have recently been through. In this case study presentation they’ll share the highs and lows they went through while rebuilding and launching the new Suncorp Mobile Bank App.

A tale of trust lost and broken elevators

September 20, 2013

Trust takes years to build, seconds to break, and forever to repair.

Customer relationships are built upon trust. Trust that a company will provide the product or service that the customer has paid for. Trust that the company will safely store the customer’s details and not misuse them. Trust that a company respects the customer and will do their best to make them happy.

This trust builds over time, with every interaction between the two. A simple a one-off online transaction, won’t build much. A daily face-to-face interaction will build a far deeper relationship.

If something happens to break this trust, it will be lost and take a long time to rebuild. Often something extraordinary is required to rebuild a broken relationship. If it ever can be.

In the world of user experience design, this is why attention to detail is important. Every time you confuse a user or make them work hard to complete a task you chip away at that relationship. Every time you do something sneaky or use a dark pattern you sour the relationship and turn your customer off your company, potentially losing them (and their friends) forever.

I encountered a perfect metaphor for this in my building recently.

The elevators have been unreliable for a long time. Lot’s of people have stories of being stuck in them. It’s an old building so it’s not surprising, but is still frustrating and scary.

They recently started replacing and upgrading the elevators. Which meant they took one out of action for a month or more.

Last week I turned up to work in the morning, coffee in hand, to find the doors to the elevator that had been out of service open. Bright light was gleaming out of the new elevator in to the dim foyer. Bright, clean, operating theatre temperature light was shining off its newly appointed stainless steel fittings.

And it was just sitting there with the doors open. Waiting for someone to get in and try out the shiny new interior.

Yet no one did.

The problem was that no one trusted it. After too many bad experiences no one was willing to believe that this shiny new elevator would finally work.  It’s sudden shiny new appearance only added to the distrust.

That morning, myself and two other people who arrived at the same time as me ended up waiting and taking one of the other older elevators.

That day someone from building management must have noticed this reaction by us occupants.

The next morning I was confronted by the same bright light shining out of the waiting elevator. This time it was adorned with signs urging me to trust it and step in to the light.

Untrustworthy elevators Shiny new elevator Tust me, I'm an elevator

Can you imagine the amount of distrust that an elevator has built up that people don’t automatically step in to it, let alone require a sign persuade them it is alright.

People accidentally get in to the wrong elevator all the time. It’s comical seeing the automatic response when someone steps in to an elevator going up when they are going down, just because it opened in front of them.

Bringing this back to designing user experiences, it’s strong metaphor for how important it is not to take a customer’s trust for granted. I’ve seen some extraordinary attempts to rebuild trust over the years but none of them wipe the slate clean again (Facebook’s data privacy issues are a great example). They always leave a lingering memory that stops a customer from ever completely losing themselves in their trust for you again. You’re far better off just not breaking your customers trust in the first place.

Finding time for the creative exploration process within Agile software delivery

July 14, 2013

When I first started working in an Agile environment I was excited to have finally reached the land of opportunity.

I was well aware of the problems with Waterfall. The big-upfront design phases.  The painful stakeholder reviews.  Having to endlessly review and revise wireframes. Countless annotations explaining every microscopic detail. Pesky developers not building things as specified.

I thought all these problems would magically disappear with Agile. All of a sudden developers would become my best friends. My wireframes wouldn’t need to be anywhere near as detailed. Document version numbers would stop getting in to double figures. My designs might get launched in the same year I created them.

However, I soon realised that these things didn’t just happen automatically. Agile opens up an exciting new way of working for designers, but it also presents a new set of challenges. To experience the benefits I had to adjust the way I worked.

These are some of the challenges I’ve faced and adjustments I’ve learnt to make to my UX practice along the way.

Challenge #1: Not having time to explore ideas and find innovative solutions

StreamrolledIn the Agile software development world dedicated design time becomes a precious commodity. The traditional user-centered design approaches start to encounter problems.  Taking the time to conduct in-depth user research, analyse the findings, write them up, and then design an extensive solution is an expensive luxury that is rarely affordable.

It’s hard to take the time when a team of developers is waiting for to you to finish so that they can start. As a UXer it feels like you have a steamroller bearing down on me while you’re screaming: ‘Stop, I need to explore more ideas!’

Challenge #2: Feeling like time spent exploring ideas is a waste when you’re trying to be Lean.

WasteThe traditional creative process involves playing with designs and seeing where they lead. More often than not the designs themselves get discarded, but the learnings from them fold back in to the overall solution.

Once you start to embrace a Lean way of thinking, taking the time to play around with different options, then discarding them can feel like your deliberately wasting time.

This test and learn approach is actually a core part of Agile, but we just have to adjust to do it in a different way.

Challenge #3: Getting out of the deliverable business.

Throwing deliverables over the wallOur traditional approach of producing shiny, all-encompassing deliverables doesn’t work in an Agile process. Not only do you not have the time but the documents are usually out-of-date before you finish them.

On the flip side, it’s not just a matter of producing lots of low-fidelity prototypes and then handing them over to developers. There can still be communication breakdowns.

We need to use prototypes as a communication tool, not just a deliverable.

We need to get out of the deliverables business.

Adjusting our practice

All of these challenges can be overcome. We just need to adjust our practices to get the most out of the Agile process. We need to focus on the value we deliver, not just the outputs. Once we do, we’ll find that it actually improves our ability to create awesome experiences.

These are some of the ways I’ve learnt to adjust my practice to be a better designer.

Adjustment #1: UX as a facilitator, not just a designer.

FacilitationChange the way you see your role in the team. We need to let go of control of every little detail and empower our team. We need to use our research skills to take the team on a journey of understanding and empathising with the user.

We need to become an Information Radiator. Not the sole source of knowledge. Give everyone the knowledge required to build a great product that meets the needs of the target audience. There will still be plenty of design work required by you as a designer, but you will be able to start sharing the load.

Adjustment #2: Design as a continuous activity.

design-dnaMake design an ongoing activity, not just a phase. Agile UX isn’t just about breaking our activities into small chunks, i.e. mini-waterfall. It’s about changing our approach and how we work.

Other fields are embracing the “Continuous” approach. Product management is embracing the Lean Startup movement.  Launching a MVP, testing and proving a proposition, then continuing to evolve it.

The development community is doing Continuous Delivery to make this possible.

The ability to release software regularly (i.e. daily rather than a few big releases a year) is allowing teams to reduce feedback cycles. Testing and learning rapidly.

As designers we need to embrace this mindset. We need to change our approach to embrace the benefits of this. Shortening our feedback cycles. Learning quickly. Finding out what experiences work and building on them.

Adjustment #3: Do just-enough, just-in-time.

Mona-Lisa sketchJust-in-time design does what it says in the tin. Don’t try to do everything upfront. Start by sketching a vision, then figure out the details as you go. Make decisions at the last responsible moment.

Change is relentless. If you do too much upfront, the requirements will probably change by the time you get your designs out there.

Tools & Techniques

These are some of the tools and techniques I’ve been using to make these adjustments to my practice.

Collaborative design workshops

Collaborate design workshops and sketchboardsTo start projects, run collaborative design workshops with stakeholders and the Delivery team. These involve getting everyone in a room and sketching out their ideas about what the product should/could be.

Put it all up on a wall where anyone can see and contribute to it. This creates a sketchboard – a lightweight shared vision of the product, which everyone is bought in to as they contributed to it.

This shared vision can inform the story writing process when kicking off projects. The sketches can also quickly be turned in to prototypes to test and validate the team’s ideas.

Use Iterations to shorten your feedback loops

The iteration cycle allows us to shorten our feedback loops. Each iteration allows us to design and build something new, quickly turning design ideas into working code. These functioning ideas when can then be tested, validated and improved. We no longer have to wait months/years to see if our design works, we can do it immediately.

Co-Location

Problems throwing things over the wall? There shouldn’t be one in the first place. Co-locate yourself with and become part of the development team. Multidisciplinary teams all working in the same place sharing ideas is a key principle of Agile. You don’t need to waste time typing up emails when you can have a quick conversation.

This also allows us to open up the kimono. Lots of developers have never worked with a UXer before. They want to learn to work with us as well. Co-location naturally allows for this to happen.

Design Walls

As the details start to emerge turn your Sketchboard in to a Design Wall. Fill in the details of your vision as you go.

Claim space alongside the Project team’s card wall. Put our wireframes/visual designs on the wall. This will ‘radiate’ details of the UX design and become a source of truth.

Lightweight production

Co-Location allows lightweight productionDo just enough, just in time. Don’t aim to capture every detail upfront. Capture just enough, and then have a conversation. Sometimes a paper sketch is enough, others a hi-fi mock is required. It’s the conversation that is important. The closer you can get to designing in code the better.

Conducting lightweight research

Bring user feedback in to the team as often as you can. Go guerrilla. Run three sessions every Friday, or every iteration. Make it easy for the team to watch. Capture you findings on Post-it notes, not in PowerPoint.

Often you can put the findings in to action straight away if the team is still working on the feature.

Add usability fixes to the backlog

If you can’t act on usability fixes straight away, become part of the Agile process and prioritise usability fixes like everything else. Write up stories for them and add them to the backlog. The Product Owner can make an informed decision about whether to prioritise them, or build something new. This helps avoid the tension of us designers trying to squeeze small fixes in to the workload in an unstructured way.

Creating great experiences

Some of these Agile ideas aren’t too dissimilar to existing UX techniques, often they just have a different name to what we usually call them. Some principles can be easily applied to the design process, with just a bit of willingness to learn and adapt.

As designers, Agile provides us some great lessons and opportunities to work in better ways.  At the same time we can add value to the process as well. Our ability to visually communicate ideas and share a vision helps drive the process and provide a customer focus which can often be lost in the mix of delivery activities.

They key goal is have an entire team collaborating together, finding ways to combine their skills and talents together to create great experiences.

This post is based on this presentation Agile UX conference presentation.

Recording mobile device usability testing sessions – guerrilla style

April 9, 2013

One of the challenges of running usability testing on mobile devices is how to record a video of the session.  On desktops recording sessions is easy. Applications like Morae or Silverback have been allowing us to do screen captures with a picture-in-picture window of a participant’s face for years now. They simply use the mouse pointer and webcams to capture of full picture of what the user is doing.

On mobile devices it is not so easy. On mobile devices it’s just as important to see what the users hand/fingers are doing as the targets they are tap on. When you combine this with the movement of handheld devices recording a usability testing session becomes somewhat more complicated.

On a recent project we ran a round of formal usability testing, where we ran ten participants through a competitor comparison of apps. To record these sessions we put together a somewhat ad-hoc, yet highly effective recording setup that used this option of mounting a camera of the device. Here’s an overview of how we made it work, including:

  • Choosing a video recording approach
  • The pop-up usability lab
  • The camera mount – Mr Tappy
  • The cameras
  • Recording software
  • Remote broadcasting of the sessions
  • Lighting
  • What would we change next time?

Choosing a video recording approach

There are 3 current options that I’ve seen used to capture a session on a mobile device:

  1. Attach a camera to the device itself.
  2. Mount a camera above the space where the device will be used.
  3. Record a screen capture of the device.

Each of these has their relative pros and cons:

Option Pros Cons
Attach a camera to the device itself
  • Stable view of the screen.
  • You can see the participant’s hands and finger movements.
  • Device agnostic. Device can be quickly and easily swapped as they don’t require anything to be locally installed.
  • The user has to overcome the weird feeling of an unusual camera mount attached the device.
  • The physical mount can get in the way of form factors or the way participants would naturally hold a device
  • Screen usually has slight perspective tilt due to the camera angle.
  • Camera recording resolution has to be balanced against physical weight of the camera mounted on the device.
Mount a camera above the space where the device will be used
  • You can see the participant’s hands and finger movements.
  • Device can be quickly and easily swapped.
  • The user can freely pick up and hold the device in a natural way.
  • Device agnostic. Device can be quickly and easily swapped as they don’t require anything to be locally installed.
  • Unstable view of the screen depending on how much the user moves the device around in their hand.
  • User has to keep the device within a certain area bounded by the camera view.
Record a screen capture of the device
  • The user can freely pick up and hold the device in a natural way.
  • High quality screen capture.
  • Little limitation on how the user needs to hold the device.
  • No visibility of the participant’s hands and finger movements.
  • Requires screen capture software to be run on the device, which is currently inconsistent between platforms.
  • Screen capture software can overload some devices.

Personally I prefer the first option of attaching a camera to the device itself. In most situations having a clear view of the screen and the hands outweighs the participant’s feeling a little bit weird about the camera hanging over the device.

The pop-up usability lab

Having chosen to go with the option of mounting a camera on the device, we set about turning two meeting rooms in to a full-featured mobile device usability testing lab.

Here are some pictures of what we put together.

The testing lab room setup and ready to go.

The testing lab room setup and ready to go.

A session in progress, with the participant using the phone and moderator guiding the session.

A session in progress, with the participant using the phone and moderator guiding the session.

The participant’s view of the device and recording equipment.

The participant’s view of the device and recording equipment.

The testing lab and observation room side-by-side.

The testing lab and observation room side-by-side.

A note taker observing the video feed from the next room, taking notes on post-its.

A note taker observing the video feed from the next room, taking notes on post-its.

The video output, the device view with a ‘picture-in-picture’ view of the participant holding the device.

The video output, the device view with a ‘picture-in-picture’ view of the participant holding the device.

What’s happening in the room?

The testing lab was made up of the following equipment:

  • Two cameras; one mounted on the device, the other in front of the user capturing their face and them holding the device.
  • Both cameras feeding in to a laptop, which was recording the session. We used a MacBook pro for this setup. This machine comfortably dealt the processing load for this setup.  It also had the 2 x USB inputs and 1 x HDMI output that we needed to connect to the cameras and TV.
  • Laptop sitting to the side where the facilitator could see it. This meant that the facilitator could sit back and watch what the user was doing through the screen rather than having to always be looking over the participant’s shoulder.
  • HDMI output cable running to a large TV in the room next door broadcasting the screen share and audio to observers.

The camera mount – Mr Tappy

The centerpiece of the setup was our camera mount – Mr Tappy. Mr Tappy is a simple mount, which you can attach a mobile device to, with an adjustable arm that extends over the top of the device to attached a camera to.

Meet Mr Tappy:

MrTappy-005 MrTappy-003 MrTappy-004 MrTappy MrTappy-002 MrTappy-001

The best part about Mr Tappy is that he can be used for all types of devices; phone, tablet, iOS, Android, Windows. The mount is flexible enough to accommodate any sized device. The only restriction is needing to attach some velcro to the back of the device for Mr Tappy to secure to.

Mr Tappy is the brainchild of Nick Bowmast, who I can happily say I used work with a few jobs ago. He took the idea that came from an old improvised mobile testing sled made out of perspex and turned it in to a high-quality product designed specifically for this purpose.  I highly recommend it.

One thing we learned was that it’s important to pick up and hand the device to the participant so that they start with it in their hand. The attached camera mount does make participants reluctant to handle the device. Handing it to them when you first ask them to start doing something on it helps get it in their hand and using it.

Still, a few participants just put it back down on the table and used it there. In our context it wasn’t too much of an issue as using a phone whilst it sits on a tabletop is not an uncommon behavior anyway.  Otherwise, it’s important to be aware of it that impacts what you are trying to test.

The cameras

For the camera mounted on the device we used the iCubie camera that Mr Tappy recommends. It’s the smallest and lightest camera that could be found at the time.

It’s downside is that it’s not super high-resolution. Therein lies the trade-off. We considered using a higher quality camera but they all just became too big and bulky to use on the mount. Instead we decided to concede the quality for being less obtrusive.

For the other camera we had on the table camera focused on the participant we used a Logitech HD Pro Webcam C910.  It was less important which specific camera we used for this. We used the Logitech as we had one on hand and knew it did good quality HD video and audio.

Recording software

The recording software was where we had to improvise the most. All the applications out there at the moment were either too expensive to bother with, didn’t allow us to capture the output from two different webcams at one time, or required too much post-production work to sync video feeds.

In the end we took an ad-hoc approach of simply having the two camera feeds open on the desktop using separate applications, then using a third application to do a screen capture of the whole desktop. Whilst not the prettiest video outputs you’ll ever see, it captured everything we needed it to.

Where it got tricky was finding the right combination of apps that didn’t have conflicts and worked with the various cameras. We had to do a bit of trial and error with different media players, plus trouble-shooting to make it work but we eventually found the right combination that worked. Here’s what we had running:

  • Device camera feed – Quicktime 7 Pro
  • Participant camera feed – Quicktime 10
  • Screen capture – Quicktime 10
  • Audio output for HDMI feed to TVLineIn

We had to ensure we left enough hard drive on the laptop to record 50-100gb of video. Full resolution videos of an hour session typically ended up at 30gb before we optimised them to a lower level. We kept an external hard drive handy to off-load videos.

We also had two laptops that setup so that we could quickly swap them if necessary. Occasionally this was necessary if a video was taking a while to be processed and saved.

Remote broadcasting of the sessions

The setup of the office, with two rooms next to each other and a big TV in one, worked out really nicely for our setup. An alternative option that we were also considering was to broadcast the session to a separate location using Skype (or similar) to screen share the video and audio to another location.

Lighting

Reflection/glare from the ceiling lights can be an issue in seeing the screen of the device through the camera mounted on Mr Tappy. To avoid this we turned off the lights directly above user (we unscrewed the fluorescent lights).

This creates a problem where the camera can’t deal with the contrast between the screen brightness and the dull background. To compensate for the lack of light we setup a lamp to the side of the user, which lit up their hands. We also turned down the device screen brightness to balance out the difference. The balance between all this needs to be tweaked depending on the exact room.

The unintended side effect of all this was to create a somewhat ‘intimate’ setting in the room for participants to walk in to. We made sure we called it out and explained why so that participants weren’t too put off by it.

Controlling the laptop and video feed

Given that the two rooms were directly next-door we connected a Bluetooth keyboard and mouse to laptop, which we put in the observation room. This allowed the observers to run the laptop and control the video (i.e. starting/stopping and trouble shooting), which helped reduce the load on the moderator in the room.

What would we change next time?

Overall we were very happy with the whole setup. I’d happily take the same approach and recommend it to others – hence why I just put this summary together.

Mr Tappy worked well as the camera mount and solved the problem of a camera mount. I’d still keep looking for a higher resolution camera than the iCubie though.

I’d also keep my eye out for a better media player solution. In fact, as I was putting this post together someone recommended SecuritySpy to me. Despite being made for a different purpose, looks like it would do the trick for a small license fee.

Otherwise, here are some of the approaches I’ve found that other people have used.

Follow

Get every new post delivered to your Inbox.

Join 404 other followers