The purpose of CUE-8 was to compare practical approaches to usability task measurement.
Fifteen participating teams carried out independent, simultaneous measurements of the Budget.com car rental website and compared results.
CUE-8 results were presented and discussed at a workshop at the UPA 2009 conference in Portland, OR, USA, on 9 June 2009.
You can’t manage what you don’t measure –
but can you manage measurements
to generate reproducible and useful results?
Practitioner’s Take Away
- Adhere strictly to precisely defined measurement procedures for quantitative tests.
- Report time-on-task, success/failure rate and satisfaction for quantitative tests.
- Exclude failed times from average task completion times.
- Understand the inherent variability from samples. Use strict participant screening criteria. Provide confidence intervals around your results if this is possible. Keep in mind that time-on-task is not normally distributed and therefore confidence intervals as commonly computed on raw scores may be misleading.
- Combine qualitative and quantitative findings in your report. Present what happened (quantitative data) and support it with why it happened (qualitative data). Qualitative data provide considerable insight regarding the serious obstacles that users faced and it is counterproductive not to report this insight.
- Justify the composition and size of your participant samples. This is the only way you have to allow your client to judge how much confidence they should place in your results.
- When using unmoderated methodologies for quantitative tests ensure that you can distinguish between extreme and incorrect results. Although unmoderated testing can exhibit a remarkable productivity in terms of user tasks measured with a limited effort, quantity of data is no substitute for clean data.
Papers About CUE-8
Rent a Car in Just 0, 60, 240 or 1,217 Seconds? – Comparative Usability Measurement, CUE-8
by Rolf Molich, Jarinee Chattratichart, Veronica D Hinkle, Janne Jul Jensen, Jurek Kirakowski, Jeff Sauro, Tomer Sharon, Brian Traynor
JUS, the Journal of Usability Studies, November 2010
Rent a car in just 60, 120, 240 seconds – Comparative Usability Measurement
by Rolf Molich, Jurek Kirakowski and Tomer Sharon
90-minute session at the UPA 2010 conference in Munich, Germany, May 2010