Usability Testing in Practice

Our thanks to The Usability Company, for permission to visit their testing laboratories, watch real usability tests in progress and publish the details.

There are many variations on Web site usability testing, but a simple way to picture it is to imagine a real user, almost (but not quite) picked off the street, sitting in front of a PC and accomplishing a short list of tasks on a Web site. As they work through the tasks they talk about them, how difficult or easy they are, and a psychologist listens and watches.

The process is repeated with a handful of different users. When the sequence is complete, the psychologist who's watched the tests should have a good idea of the site's weaknesses and strengths. The weaknesses can then be rectified and the site becomes more successful.

That's a very simple picture. In practice there are many variations on testing, beginning with the basic division between testing live sites that already exists on the Web, and testing new sites still at an early stage of development. The two are approached in different ways.

Testing existing sites is the more common of the two, perhaps because there are relatively few new sites in the process of creation compared to the number that already exist, and also because sites need to be repeatedly tested through their lifetime as they change and incorporate new content.

Fist let's look in detail at testing existing sites, before moving on to brand new Web ventures, or "greenfield" sites, as they're known.

Heuristic Evaluations

The simplest and cheapest usability studies don't include testing. They're a straightforward audit of the site by a usability expert - also referred to as a heuristic evaluation. The definition of heuristic used here belongs more to psychologists than to programmers. It means expertise gained through experience, but without the programming implications of dogged improvement through trial and error, usually involving wasted time.

A heuristic evaluation of a Web site is an inexpensive way to identify its most obvious usability flaws. It may be the only option for companies that don't have the budget for proper testing, which can cost from ten thousand to a hundred thousand dollars.

Testing

Not everybody trusts experts - in the field of usability as elsewhere. As the joke goes, there was Adam and there was Eve, and then there was the snake, the first consultant. But most companies who are wary of experts will be convinced about opinions if they're coming from their customers, and that's the aim of usability testing. It aims to interpret the experiences of target users and turn this into information that's actionable.

It's not the same as market research, which might be interested in whether a site is attractive to the user or has a memorable logo. This kind of sentiment may come out of a usability test as a useful by-product, but the primary focus is on pure performance - how easy the site is to use.

Choosing Users

The aim is to recruit real people from the site's target demographic to carry out tasks typical of those they would do in the real world, for example logging in and purchasing items.

In practice, it's been found that only a small number of people are needed for the tests to be effective - between three and ten, depending whose expert opinion you choose to believe. The Usability Company opts for eight. Such small numbers might rankle with theoretical statisticians looking for statistical significance and expecting hundreds of testers, but experience shows that it works. The first eight testers will invariably identify a site's worst flaws, and after this number the law of diminishing returns rapidly kicks in. Additional testers rarely discover further issues of crucial importance.

Some clients prefer to test with their own registered users, but most prefer the testing company to find a selection for them. The Usability Company subcontracts this selection to agencies, which is more expensive than keeping a database of users itself, but avoids the problem of database junkies who turn up as a housewife one month and a stockbroker the next. The non-database "free-found" option works out slightly more expensive, but the cost of hiring this kind of participant remains a relatively small proportion of the total testing cost - not much above 5%.

The Tasks