Hostname: page-component-76fb5796d-wq484 Total loading time: 0 Render date: 2024-04-25T14:16:22.565Z Has data issue: false hasContentIssue false
  • English
  • Français

A Narrative Review of Methods for Applying User Experience in the Design and Assessment of Mental Health Smartphone Interventions

Published online by Cambridge University Press:  24 January 2020

Christopher Lemon Open the ORCID record for Christopher Lemon [Opens in a new window] ,
Kit Huckvale ,
Kenneth Carswell  and
John Torous
Christopher Lemon*
Affiliation:
St Vincent's Hospital, 390 Victoria Street, Sydney, Australia Faculty of Medicine, University of New South Wales, Sydney, NSW, Australia
Kit Huckvale
Affiliation:
Black Dog Institute, UNSW Sydney, Sydney, Australia
Kenneth Carswell
Affiliation:
Department of Mental Health and Substance Use, World Health Organization, Geneva, Switzerland
John Torous
Affiliation:
Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, USA
*
Author for correspondence: Christopher Lemon, E-mail: christopheralemon@gmail.com
Get access Rights & Permissions [Opens in a new window]

Abstract

Objectives

User experience (UX) plays a key role in uptake and usage of mental health smartphone interventions, yet remains underinvestigated. This review aimed to characterize and compare UX evaluation approaches that have been applied in this specific context, and to identify implications for research and practice.

Methods

A narrative review was conducted of UX-themed studies published in PubMed, PsycINFO, and Scopus up to February 2019. Eligible studies reported on data reflecting users' interactions with a smartphone intervention for any mental health condition. Studies were categorized into “situated” versus “construct-based” methods according to whether or not an established UX construct was used to acquire and analyze data.

Results

Situated approaches used bespoke UX metrics, including quantitative measures of usage and performance, as well as grounded interview data. Construct-based approaches such as assessments of usability and acceptability were based on conceptual frameworks, with methodologically stronger versions featuring construct definitions, validated measurement tools, and an ability to compare data across studies. Constructs and measures were sometimes combined to form bespoke construct-based approaches.

Conclusions

Both situated and construct-based UX data may provide benefits during design and implementation of a mental health smartphone intervention by helping to clarify the needs of users and the impact of new features. Notable however was the omission of UX methods, such as split testing. Future research should consider these unaddressed methods, aim to improve the rigor of UX assessment, integrate their use alongside clinical outcomes, and explore UX assessment of more complex, adaptive interventions.

Keywords

User experienceMental healtheHealthMobile healthApps
Type
Assessment
Information
International Journal of Technology Assessment in Health Care , Volume 36 , Issue 1 , 2020 , pp. 64 - 70
Copyright
Copyright © Cambridge University Press 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

World Health Organization (2016) Global diffusion of eHealth: Making universal health coverage achievable. Report of the third global survey on eHealth Global Observatory for eHealth. World Health Organization: Geneva.Google Scholar
Gulliver, A, Griffiths, KM, Christensen, H (2010) Perceived barriers and facilitators to mental health help-seeking in young people: A systematic review. BMC Psychiatry 10. https://doi.org/10.1186/1471-244X-10-113.CrossRefGoogle ScholarPubMed
Dorsey, ER, Chan, YF, Mcconnell, MV, et al. (2017) The use of smartphones for health research. Acad Med 92, 157–160.CrossRefGoogle ScholarPubMed
Sabesan, S, Kelly, J (2015) Implementing telehealth as core business in health services. Med J Aust 202, 231–233.CrossRefGoogle ScholarPubMed
Estai, M, Kanagasingam, Y, Xiao, D, et al. (2017) End-user acceptance of a cloud-based teledentistry system and Android phone app for remote screening for oral diseases. J Telemed Telecare 21, 44–52.CrossRefGoogle Scholar
Zaliani, S, Gilani, MS, Nikbin, D, et al. (2014) Determinants of telemedicine acceptance in selected public hospitals in Malaysia: A clinical perspective. J Med Syst 38, 111.CrossRefGoogle Scholar
George, D, Hassali, MA, HSS, A-S (2018) Usability testing of a mobile app to report medication errors anonymously: Mixed-methods approach. JMIR Hum Factors 5, e12232.CrossRefGoogle ScholarPubMed
Knowles, SE, Toms, G, Sanders, C, et al. (2014) Qualitative meta-synthesis of user experience of computerised therapy for depression and anxiety. PLoS ONE 9. https://doi.org/10.1371/journal.pone.0084323.CrossRefGoogle ScholarPubMed
Smith, MS, Lawrence, V, Sadler, E, et al. (2019) Barriers to accessing mental health services for women with perinatal mental illness: Systematic review and meta-synthesis of qualitative studies in the UK. BMJ Open 9, e024803.Google Scholar
Knaak, S, Mantler, E, Szeto, A (2017) Mental illness-related stigma in healthcare: Barriers to access and care and evidence-based solutions. Healthc Manag Forum 30, 111–116.CrossRefGoogle ScholarPubMed
Gronholm, PC, Henderson, C, Deb, T, et al. (2017) Interventions to reduce discrimination and stigma: The state of the art. Soc Psychiatry Psychiatr Epidemiol 52, 249–258.CrossRefGoogle ScholarPubMed
Bhugra, D, Tasman, A, Pathare, S, et al. (2017) The WPA-lancet psychiatry commission on the future of psychiatry. Lancet Psychiatry 4, 775–818.CrossRefGoogle ScholarPubMed
Razzouk, R, Shute, V (2012) What is design thinking and why is it important? Rev Educ Res 82, 330–348.CrossRefGoogle Scholar
Scholten, H, Granic, I (2019) Use of the principles of design thinking to address limitations of digital mental health interventions for youth: Viewpoint. J Med Internet Res 21(1), e11528.CrossRefGoogle ScholarPubMed
Yardley, L, Morrison, L, Bradbury, K, et al. (2015) The person-based approach to intervention development: Application to digital health-related behavior change interventions. J Med Internet Res 17, e30.CrossRefGoogle ScholarPubMed
Kylberg, M, Haak, M, Iwarsson, S (2018) Research with and about user participation: Potentials and challenges. Aging Clin Exp Res 30, 105–108.CrossRefGoogle ScholarPubMed
Bakker, D, Kazantzis, N, Rickwood, D, et al. (2016) Mental health smartphone apps: Review and evidence-based recommendations for future developments. JMIR Ment Health 3, e7.CrossRefGoogle ScholarPubMed
Torous, J, Firth, J, Mueller, N, et al. (2017) Methodology and reporting of mobile heath and smartphone application studies for schizophrenia. Harv Rev Psychiatry 25, 146–154.CrossRefGoogle Scholar
Feather, JS, Howson, M, Ritchie, L, et al. (2016) Evaluation methods for assessing users’ psychological experiences of web-based psychosocial interventions: A systematic review. J Med Internet Res 18(6), e181.CrossRefGoogle ScholarPubMed
Nicholas, J, Fogarty, AS, Boydell, K, et al. (2017) The reviews are in: A qualitative content analysis of consumer perspectives on apps for bipolar disorder. J Med Internet Res 19, e105.CrossRefGoogle ScholarPubMed
Torous, J, Nicholas, J, Larsen, ME, et al. (2018) Clinical review of user engagement with mental health smartphone apps: Evidence, theory and improvements. Evid Based Ment Health 21, 116–119.CrossRefGoogle ScholarPubMed
Torous, J, Andersson, G, Bertagnoli, A, et al. (2019) Towards a consensus around standards for smartphone apps and digital mental health. World Psychiatry 18, 97–98.CrossRefGoogle ScholarPubMed
Abrahão, S, Bordeleau, F, Cheng, B, et al. User experience for model-driven engineering: Challenges and future directions. In: Gray, J, Kulkarni, , eds. Proceedings ACM/IEEE 20th international conference on model driven engineering languages and systems. Piscataway, NJ, USA: IEEE Institute of Electrical and Electronics Engineers Inc., 229–236.Google Scholar
Law, ELC, Roto, V, Hassenzahl, M, et al. (2009) Understanding, scoping and defining user experience: A survey approach. In: Olsen, DR and Arthur, RB, eds. Proceedings conference on human factors in computing systems. New York, NY, USA: ACM Press, 719–728.CrossRefGoogle Scholar
Baumel, A, Birnbaum, ML, Sucala, M (2017) A systematic review and taxonomy of published quality criteria related to the evaluation of user-facing eHealth programs. J Med Syst 41(128). doi:10.1007/s10916-017-0776-6.CrossRefGoogle ScholarPubMed
Ben-Zeev, D, Wang, R, Abdullah, S, et al. (2016) Mobile behavioral sensing in outpatients and inpatients with schizophrenia. Psychiatr Serv 67, 558–561.CrossRefGoogle ScholarPubMed
Mackintosh, M-A, Niehaus, J, Taft, CT, et al. (2017) Using a mobile application in the treatment of dysregulated anger among veterans. Mil Med 182, e1941–e1949. https://doi.org/10.7205/MILMED-D-17-00063.CrossRefGoogle ScholarPubMed
Orlowski, S, Lawn, S, Matthews, B, et al. (2016) The promise and the reality: A mental health workforce perspective on technology-enhanced youth mental health service delivery. BMC Health Serv Res 16. doi:10.1186/s12913-016-1790-y.CrossRefGoogle ScholarPubMed
Attwood, S, Parke, H, Larsen, J, et al. (2017) Using a mobile health application to reduce alcohol consumption: A mixed- methods evaluation of the drinkaware track & calculate units application. BMC Public Health 17, 1–21.CrossRefGoogle ScholarPubMed
ISO 9241-11:2018 (2018) Ergonomics of human-system interaction—Part 11: Usability: Definitions and Concepts: Geneva.Google Scholar
Agnisarman, SO, Madathil, KC, Smith, K, et al. (2017) Lessons learned from the usability assessment of home-based telemedicine systems. Appl Ergon 58, 424–434.CrossRefGoogle ScholarPubMed
Sheehan, B, Lee, Y, Rodriguez, M, et al. (2012) A comparison of usability factors of four mobile devices for accessing healthcare information by adolescents. Appl Clin Inform 3, 356–366.CrossRefGoogle ScholarPubMed
Vilardaga, R, Rizo, J, Kientz, JA, et al. (2016) User experience evaluation of a smoking cessation app in people with serious mental illness. Nicotine Tob Res 18, 1032–1038.CrossRefGoogle ScholarPubMed
Bangor, A, Kortum, PT, Miller, JT (2009) Determining what individual SUS scores mean: Adding an adjective rating scale. J Usability Stud 4, 114–123.Google Scholar
Dubad, M, Winsper, C, Meyer, C, et al. (2018) A systematic review of the psychometric properties, usability and clinical impacts of mobile mood-monitoring applications in young people. Psychol Med 48, 208–228.CrossRefGoogle ScholarPubMed
ISO/IEC 9126-1 (2001) Software engineering—Product quality—Part 1: Quality model. International Organization for Standardization: Geneva.Google Scholar
Kortum, PT, Bangor, A (2012) Usability ratings for everyday products measured with the system usability scale. Int J Hum Comput Interact 29, 67–76.CrossRefGoogle Scholar
Ben-Zeev, D, Brenner, CJ, Begale, M, et al. (2014) Feasibility, acceptability, and preliminary efficacy of a smartphone intervention for schizophrenia. Schizophr Bull 40, 1244–1253.CrossRefGoogle Scholar
Brooke, J (1996) SUS: A “quick and dirty” usability scale. In: Jordan, PW, Thomas, B, Weerdmeester, BA, McClelland, IL, eds. Usability evaluation in industry. London: Taylor & Francis, 189–194.Google Scholar
Lewis, JR (1992) Psychometric evaluation of the poststudy system usability questionnaire: The PSSUQ. Proc Hum Factor Soc Annu Meet 36, 1259.CrossRefGoogle Scholar
Venkatesh, V, Davis, FD (2000) A theoretical extension of the technology acceptance model: Four longitudinal field studies. Manage Sci 46, 186–204.CrossRefGoogle Scholar
Lund, AM (2001) Measuring usability with the USE questionnaire. Usability User Exp 8, 3–6.Google Scholar
Povey, J, Mills, PPJR, Dingwall, KM, et al. (2016) Acceptability of mental health apps for Aboriginal and Torres Strait Islander Australians: A qualitative study. J Med Internet Res 18, e65.CrossRefGoogle ScholarPubMed
Shen, N, Levitan, M-J, Johnson, A, et al. (2015) Finding a depression app: A review and content analysis of the depression app marketplace. JMIR mHealth uHealth 16, e16.CrossRefGoogle Scholar
Fuller-Tyszkiewicz, M, Richardson, B, Klein, B, et al. (2018) A mobile app-based intervention for depression: End-user and expert usability testing study. J Med Internet Res 5, e54.Google ScholarPubMed
Maramba, I, Chatterjee, A, Newman, C (2019) Methods of usability testing in the development of eHealth applications: A scoping review. Int J Med Inform 126, 95–104.CrossRefGoogle ScholarPubMed
Bojko, A (2013) Eye tracking the user experience: A practical guide to research. New York: Rosenfeld Media.Google Scholar
Righi, C, James, J, Beasley, M, et al. (2013) Card sort analysis best practices. J Usability Stud 8, 69–89.Google Scholar
Speicher, M, Both, A, Gaedke, M (2014) Ensuring web interface quality through usability-based split testing. In: Casteleyn, S, Rossi, G, Winckler, M, eds. Web engineering. ICWE 2014. Lecture notes in computer science, vol. 8541. Cham: Springer, 93–110.Google Scholar
Woods, L, Cummings, E, Duff, J, et al. (2017) Design thinking for mHealth application co-design to support heart failure self-management. Stud Health Technol Inform 241, 97–102.Google ScholarPubMed
Naslund, JA, Aschbrenner, KA, Barre, LK, et al. (2015) Feasibility of popular m-health technologies for activity tracking among individuals with serious mental illness. Telemed e-Health 21, 213–216.CrossRefGoogle ScholarPubMed
Long, AF, Gambling, T, Young, RJ, et al. (2005) Acceptability and satisfaction with a telecarer approach to the management of type 2 diabetes. Diabetes Care 28, 283–289.CrossRefGoogle ScholarPubMed
Torous, JB, Chan, SR, Gipson, SY-MT, et al. (2018) A hierarchical framework for evaluation and informed decision making regarding smartphone apps for clinical care. Psychiatr Serv 69, 498–500.CrossRefGoogle ScholarPubMed
Stoyanov, SR, Hides, L, Kavanagh, DJ, et al. (2015) Mobile app rating scale: A new tool for assessing the quality of health mobile apps. JMIR mHealth uHealth 3, e27.CrossRefGoogle ScholarPubMed
Leigh, S (2016) Comparing applets and oranges: Barriers to evidence-based practice for app-based psychological interventions. Evid Based Ment Health 19, 90–92.CrossRefGoogle ScholarPubMed
Fulmer, R, Joerin, A, Rauws, M, et al. (2018) Using psychological artificial intelligence (Tess) to relieve symptoms of depression and anxiety: Randomized controlled trial. JMIR Ment Health 5, e64.CrossRefGoogle ScholarPubMed