Dialog Design DialogDesign ved Rolf Molich English English Dansk Dansk
DialogDesign, Skovkrogen 3, 3660 StenlÝse, Denmark, dialogdesign@dialogdesign.dk, +45 4717 1731
Free Resources CUE Products Events Books About DialogDesign
CUE Papers CUE-1 CUE-2 CUE-3 CUE-4 CUE-5 CUE-6 CUE-7 CUE-8 CUE-9

The Evaluator Effect, or What You Get is What YOU See (CUE-9)

CUE-9 was a unique opportunity for usability professionals to assess and improve their usability testing skills.

CUE-9 was about the Evaluator Effect or Rashomon Effect, which names the observation that usability evaluators who analyze the same usability test sessions often identify substantially different sets of usability problems.

CUE-9 assembled experienced usability professionals to discuss the state-of-the-art in usability evaluation based on a common experience in evaluating the website for the US moving company U-Haul:

  • Watch five 30-minute videos from usability test sessions,

  • Write a short, anonymous report about your findings,

  • Submit your report,

  • Read similar reports written by other experienced professionals,

  • Meet experienced colleagues at the CUE-9 workshop. Compare and discuss findings. Learn from the similarities and differences.

Available Downloads

       CUE-9 Overview (2 pages)

       CUE-9 Detailed Description (7 pages)

       CUE-9 Participants (1 page)

       CUE-9 Reports, 19 reports from first workshop (33 MB)

       CUE-9 Reports, 16 reports from second workshop (34 MB)


CUE-9 workshops were held on 20 June 2011 in Atlanta, GA, USA, and on 11 September 2011 in Chemnitz, Germany. Thirty-five leading usability professionals participated in CUE-9, including Steve Krug, Chauncey Wilson, Carol Barnum, Tom Tullis and Nigel Bevan. The Atlanta workshop attracted 19 participants while the Chemnitz workshop attracted 16.

Advice from CUE-9

The analysis of CUE-9 results is underway.

Preliminary findings and advice:

      An evaluator effect was demonstrated but much of it can be explained by other means, for example the large number of usability problems on the tested site.

      Phrase your usability findings carefully. A considerable number of CUE-9 findings were hard or impossible to understand.

      Be careful when you report minor problems from a usability test. No one else may agree with you that the problem is worth reporting.

      We need more precise rating scales for usability problems. CUE-9 participants varied considerably in their ratings of identical problems.

Home    Free Resources    CUE    Products    Events    Books    About DialogDesign