Dialog Design DialogDesign ved Rolf Molich English English Dansk Dansk
DialogDesign, Skovkrogen 3, 3660 Stenløse, Denmark, dialogdesign@dialogdesign.dk, +45 4717 1731
Free Resources CUE Products Events Books About DialogDesign
CUE Papers CUE-1 CUE-2 CUE-3 CUE-4 CUE-5 CUE-6 CUE-7 CUE-8 CUE-9

CUE-3 - Avis Rent-a-Car

CUE-3 is a comparative usability inspection of the www.avis.com website conducted in August and September 2001. Eleven Danish usability professionals independently evaluated the site, each using their favorite inspection technique.

Available Downloads

·       The proposal and scenario for CUE-3. The proposal describes the background and rules for the study, and the scenario shows what each professional received at the start of the study 
(PDF, 27 KB).

·       All evaluation reports in one PDF file. Note that the reports from teams B and E are in Danish; the rest are in English
PDF, 5,959 KB).

·       CUE-3 paper by Morten Hertzum, Niels Ebbe Jacobsen, and Rolf Molich, “Usability Inspections by Groups of Specialists: Perceived Agreement in Spite of Disparate Observations,” CHI2002 Extended Abstracts, ACM Press, pp. 662-663, www.acm.org   (PDF, 19 KB). 

The Most Important CUE-3 Findings

Each of the 11 evaluators reported an average of 35 (16%) of the 220 problems. The overlap in reported problems between any two evaluators averaged only 9%. As many as 174 (79%) of the problems were reported by just one or two evaluators. Hence, the 11 inspections exhibit a substantial evaluator effect.

The evaluator effect would be less critical if severe problems were reported more consistently than cosmetic problems, which have little impact on a website’s usability. A problem was defined as severe if it appeared in one or more executive summaries. Each evaluator reported an average of 24% of the 33 severe problems. Seventeen (52%) of the severe problems were reported by just one or two evaluators. Hence, the evaluator effect persisted for severe problems.

The substantial differences in the individual reports stand in stark contrast to the perception the evaluators acquired during the group work. They left the group work with a strong, and reassuring, feeling of agreement. This became evident during the plenary session, as exemplified by the following quotes from five of the evaluators:

·      “I was surprised to see how little we disagreed.”

·      “A very high level of agreement.”

·      “It is not that subjective after all. There is consensus about what the problems are.”

·      “General agreement, but a number of concrete details differ.”

·      “We are all in agreement. We haven’t made the same observations, though.”

Nobody countered these statements.

Read more in our CUE-3 paper. You can download the paper using the link in the left column.

Home    Free Resources    CUE    Products    Events    Books    About DialogDesign