Usability testing: from # of users to # of tasks ?


User-centered design of any system requires user testing.
The question of how many users should be tested had settled for a
while on about 5-8 users. Today there is a new proposal promoting
the swith of focus from number of users to the number of tasks.
Quoting from HFI (Human Factors International):

“In contrast, their analysis showed a significant positive correlation
between the number of tasks evaluated and the number of problems
uncovered. That is, the more tasks a team included in their testing protocol, the more problems they uncovered.

They conclude that other things being equal (e.g., quality of recruiting), the better predictor of the productivity of usability testing is the number of tasks participants (try to) complete, not the number of participants who try to complete them.”

Read the entire article at the HFI website >>
Daniel Montano
Keyword: Daniel Montano, Dan Montano, user experience design, information architect


2 thoughts on “Usability testing: from # of users to # of tasks ?

  1. In practice, it is obvious that if you test with more task, you will fin more problems. The real isse is the selection pf task.
    1) In my view, it is better to test with a limited number of task thar are representative of the user goals. Who care about not important functionality, why should we test it

    2) There is clerely an observable diminished law of return with the number of users. There is less return after 8 users

  2. reticula

    I had a grad course in “technology mediated collaboration” at the School of Information at the U. of Michigan. It’s one of their specialties, including ways to do testing on collaboration-ware, which is an entirely different animal than what I’d call “parallel play ware”. It’s also much harder to test, and, in my opinion, where electronic medical record systems’ noble goals breakdown, crash, and burn. CCHIT standards for EHR’s don’t even include multi-user testing scenarios, but the whole point of an EHR is to handoff wisdom about a patient from one doctor to another who is collaborating on their care. Interactions flow in loops, not trees.
    Single user testing of such systems, regardless how many “tasks” are tested, is like testing the concept “soccer” one player on the field at a time.
    If multiple people are trying to act as a single entity in handling some problem or task, the only realistic way to reveal the group-level problems is to test it with multple people at once. It’s not a question of performance or loading – it’s a question of behaviors that people exhibit in groups that they would never do by themselves. In medicine, it is sometimes said that there is no such thing as “one doctor.” There is a huge “audience effect” on behavior.
    All this is on top of the normal single-user testing normally done.
    And, the first question about any “fact” or “assertion” is “who said that?” All data are not equivalent. Standardizing all input into the same format is the kiss of death to usefulness of the result to doctors. It’s not even clear that such input should be taken out of voice and converted to text with the voice discarded, because in person doctors convey a great deal unspoken about the certainty or uncertainty they have about statements that appear definitive in text.
    For technology-mediated collaboration projects, see


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s