Skip to main content


  • Kei Long Cheung (a1), Mickaël Hiligsmann (a2), Maximilian Präger (a3), Teresa Jones (a4), Judit Józwiak-Hagymásy (a5), Celia Muñoz (a6), Adam Lester-George (a7), Subhash Pokhrel (a4), Ángel López-Nicolás (a8), Marta Trapero-Bertran (a6), Silvia M.A.A. Evers (a9) and Hein de Vries (a10)...

Objectives: Economic decision-support tools can provide valuable information for tobacco control stakeholders, but their usability may impact the adoption of such tools. This study aims to illustrate a mixed-method usability evaluation of an economic decision-support tool for tobacco control, using the EQUIPT ROI tool prototype as a case study.

Methods: A cross-sectional mixed methods design was used, including a heuristic evaluation, a thinking aloud approach, and a questionnaire testing and exploring the usability of the Return of Investment tool.

Results: A total of sixty-six users evaluated the tool (thinking aloud) and completed the questionnaire. For the heuristic evaluation, four experts evaluated the interface. In total twenty-one percent of the respondents perceived good usability. A total of 118 usability problems were identified, from which twenty-six problems were categorized as most severe, indicating high priority to fix them before implementation.

Conclusions: Combining user-based and expert-based evaluation methods is recommended as these were shown to identify unique usability problems. The evaluation provides input to optimize usability of a decision-support tool, and may serve as a vantage point for other developers to conduct usability evaluations to refine similar tools before wide-scale implementation. Such studies could reduce implementation gaps by optimizing usability, enhancing in turn the research impact of such interventions.

Hide All
1. Park, AL, McDaid, D, Weiser, P, et al. Examining the cost effectiveness of interventions to promote the physical health of people with mental health problems: A systematic review. BMC Public Health. 2013;13:787.
2. Nicod, E, Kanavos, P. Commonalities and differences in HTA outcomes: A comparative analysis of five countries and implications for coverage decisions. Health Policy. 2012;108:167177.
3. Drummond, MF, Schwartz, JS, Jönsson, B, et al. Key principles for the improved conduct of health technology assessments for resource allocation decisions. Int J Technol Assess Health Care. 2008;24: 244258.
4. van Velden ME, Severens JL, Novak, A. Economic evaluations of healthcare programmes and decision making. Pharmacoeconomics. 2005;23:10751082.
5. Oliver, K, Innvar, S, Lorenc, T, Woodman, J, Thomas, J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14:2.
6. Macintyre, S, Chalmers, I, Horton, R, Smith, R. Using evidence to inform health policy: Case study. BMJ. 2001;322:222.
7. Drummond, M. Economic evaluation in health care: Is it really useful or are we just kidding ourselves? Aust Econ Rev. 2004;37: 311.
8. Garrido, MV. Health technology assessment and health policy-making in Europe: Current status, challenges and potential. Geneva: WHO Regional Office Europe; 2008.
10. Lim, SS, Vos, T, Flaxman, AD, et al. A comparative risk assessment of burden of disease and injury attributable to 67 risk factors and risk factor clusters in 21 regions, 1990–2010: A systematic analysis for the Global Burden of Disease Study 2010. Lancet. 2013;380: 22242260.
11. WHO. WHO Framework Convention on Tobacco Control. 2003. (accessed December 17, 2015).
12. European Commission. “Public Health: Tobacco Policy.” 2012. (accessed December 17, 2015).
13. Pokhrel, S, Evers, S, Leidl, R, et al. EQUIPT: Protocol of a comparative effectiveness research study evaluating cross-context transferability of economic evidence on tobacco control. BMJ Open. 2014;4:e006945.
14. van Gemert-Pijnen, JE, Nijland, N, van Limburg, M, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res. 2011;13:e111.
15. Eysenbach, G. The law of attrition. J Med Internet Res. 2005;7:e11.
16. Horsky, J, Schiff, GD, Johnston, D, Mercincavage, L, Bell, D, Middleton, B. Interface design principles for usable decision support: A targeted review of best practices for clinical prescribing interventions. J Biomed Inform. 2012;45:12021216.
17. ISO. 9241-11:1998. Ergonomic requirements for office work with visual display terminals (VDT)s – Part 11: Guidance on usability. 1998. (accessed January 16, 2018).
18. Bevan, N. Usability is quality of use. Adv Hum Factors Ergon. 1995;20:349.
19. Wichansky, AM. Usability testing in 2000 and beyond. Ergonomics. 2000;43:9981006.
20. Cho, V, Cheng, TE, Lai, WJ. The role of perceived user-interface design in continued usage intention of self-paced e-learning tools. Comput Educ. 2009;53:216227.
21. Becker, S, Mottay, FE. A global perspective on web site usability. IEEE Software. 2001;18:5461.
22. Ahern, DK. Challenges and opportunities of eHealth research. Am J Prev Med. 2007;32:S75-S82.
23. Nielsen, J. Usability inspection methods. New York: John Wiley and Sons; 1994;17:2562.
24. Voncken-Brewster, V, Moser, A, Van Der Weijden T, Nagykaldi Z, De Vries H, Tange H. Usability evaluation of an online, tailored self-management intervention for chronic obstructive pulmonary disease patients incorporating behavior change techniques. JMIR Res Protoc. 2013;2:e3.
25. Cheung, KL, Evers, SM, Hiligsmann, M, et al. Understanding the stakeholders’ intention to use economic decision-support tools: A cross-sectional study with the Tobacco Return on Investment tool. Health Policy. 2016;120:4654.
26. Centre for Disease Control and Prevention. 2011. Introduction to program evaluation for public health programs: A self-study guide. (accessed January 21, 2018).
27. Nielsen, J, Landauer, TK, editors. A mathematical model of the finding of usability problems. Proceedings of the INTERACT'93 and CHI'93 conference on Human factors in computing systems, Amsterdam, The Netherlands, 1993: ACM.
28. Jaspers, MW. A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. Int J Med Inform. 2009;78:340353.
29. Brooke, J. SUS-A quick and dirty usability scale. Usability Eval Ind. 1996;189:47.
30. Jaspers, MW, Steen, T, van Den Bos, C, Geenen, M. The think aloud method: A guide to user interface design. Int J Med Inform. 2004;73:781795.
31. Nielsen, J, editor. Reliability of severity estimates for usability problems found by heuristic evaluation. Posters and short talks of the 1992 SIGCHI conference on Human factors in computing systems, Monterey, California, 1992: ACM.
32. Brooke, J. SUS: A retrospective. J Usability Stud. 2013;8:2940.
33. Jeffries, R, Miller, JR, Wharton, C, Uyeda, K, editors. User interface evaluation in the real world: A comparison of four techniques. Proceedings of the SIGCHI conference on Human factors in computing systems, Montreal, Canada, 1991: ACM.
34. Lai, T-Y, Larson, EL, Rockoff, ML, Bakken, S. User Acceptance of HIV TIDES—tailored interventions for management of depressive symptoms in persons living with HIV/AIDS. JAMA. 2008;15:217226.
35. Bailey, RW, Allan, RW, Raiello, P, editors. Usability testing vs. heuristic evaluation: A head-to-head comparison. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 1992: SAGE Publications.
36. Rogers, EM. Diffusion of innovations. New York: Simon and Schuster; 2010.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

International Journal of Technology Assessment in Health Care
  • ISSN: 0266-4623
  • EISSN: 1471-6348
  • URL: /core/journals/international-journal-of-technology-assessment-in-health-care
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Type Description Title
Supplementary materials

Cheung et al. supplementary material 3
Cheung et al. supplementary material

 Word (46 KB)
46 KB
Supplementary materials

Cheung et al. supplementary material 2
Cheung et al. supplementary material

 Word (24 KB)
24 KB
Supplementary materials

Cheung et al. supplementary material 4
Cheung et al. supplementary material

 Word (31 KB)
31 KB
Supplementary materials

Cheung et al. supplementary material 1
Supplementary Video

 Video (144.2 MB)
144.2 MB


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed