(Re)Defining user engagement in the m-health space

Human–computer interaction (HCI) researchers define user engagement as capturing and maintaining the attention and interest of technology users,^{Reference Jacques9} and the cognitive, emotional and temporal investment made by users.^{Reference O'Brien and Toms10} Mental health interventions target not only behaviour, but emotional and cognitive processes,^{Reference Perski, Blandford, West and Michie11} and social connectivity may be fundamental for self-management.^{Reference Nicholas, Fogarty, Boydell and Christensen8} A holistic definition of user engagement with m-health apps goes beyond what people do to how these tools address needs for information/education, social support and personal agency.

HCI researchers have identified attributes of user engagement, including user attention, interest, motivation, control and system usability and aesthetic appeal.^{Reference Jacques9,Reference O'Brien and Toms10} Focusing on user-engagement attributes offers a targeted approach to measurement. For example, we might use eye-tracking or heat maps to gauge in-app attention, or brief self-report instruments to capture users’ sense of control at defined points in time during an interaction. HCI researchers have modelled user engagement as having natural ebbs and flows over the course of users’ interactions with digital tools. The process of model of engagement suggests that users move through points of engagement, periods of sustained engagement, disengagement and re-engagement and that some attributes are more salient at particular stages than others.^{Reference O'Brien and Toms10} Process-based models reveal that engagement is not an ‘all or nothing’ phenomenon. Thus, is it essential to not only measure interaction outcomes (i.e. total session duration, total app downloads) but users’ journeys through an app, i.e., ups and downs in interactivity and varying levels of emotional and cognitive involvement.

Qualifying the quantification of m-health app usage

Interpreting the ‘ups and downs’ of in-app user engagement involves looking at usage data differently, and connecting multiple data sources in significant ways. There are ‘unique cognitive, neurological or motor needs arising from mental illness’.^{Reference Ng, Firth, Minen and Torous5} People with chronic health conditions have fluctuating needs, yet apps do not take into account the diversity of individual lived experiences or different users, for example, young people, those newly diagnosed.^{Reference Nicholas, Fogarty, Boydell and Christensen8} High usage is not indicative of positive clinical outcomes but may actually reflect worsening mental health. App usage may also exacerbate negative mood by providing poor quality/too much information, technical challenges^{Reference Nicholas, Fogarty, Boydell and Christensen8} or increasing user's awareness of distressing symptoms.^{Reference Allan, Mcleod, Bradstreet, Beedie, Moir and Gleeson12}

Morton et al^{Reference Morton, Hole, Murray, Buzwell and Michalak13} found that people living with bipolar disorder experienced both negative and positive emotions toward self-monitoring; the practice made them feel that they were managing their condition effectively compared with others, but was also ‘an unpleasant reminder that they were living with bipolar disorder’. Nuanced interpretation is not revealed through usage data alone.

Condition-specific knowledge is critical, and participatory design approaches are essential for gathering insights from people with lived experience and healthcare providers. Participatory design draws upon different methods (focus groups, ethnography) to co-create and co-evaluate prototypes with users throughout the design life cycle.^{Reference Spinuzzi14} Such methods are needed to identify what goals an app should fulfil, how people want to use it and for what purposes, and how design features can reflect user preferences and goals. Such knowledge can aid in the development of appropriate engagement indicators that will facilitate the interpretation of usage and other data sources.

Apps contain different types of content (for example educational articles, quizzes, symptom trackers, social connection tools). Looking at how frequently/long users interact with individual content pages or features may be less informative than grouping features according to function or the need they are intended to serve, such as education, symptom tracking, social support. It may be productive to distinguish content that people typically access once, for example quiz, versus multiple times, for example sleep or exercise logs, and to consider the sequence of content interactions in terms of scaffolding engagement. Rethinking the analysis of usage data would allow for richer interpretations of why people use apps in naturalistic settings, such as for routine maintenance, affirmation, social support. It could also help tailor content according to recovery or illness stage to reduce cognitive load.^{Reference Morton, Hole, Murray, Buzwell and Michalak13}

Algorithms facilitate the identification of engagement patterns based on interactions with apps over time, often based on duration (short versus long term) and frequency (low versus high). Such analyses reveal that different users have different use trajectories, but they do not explain why the pattern is occurring or its significance to clinical outcomes. Diverse and varied streams of data, including user self-reports, discussion forum transcripts, social network data and data from symptom severity, functioning or quality of life measures can be used to make sense of usage data. Cluster analysis is another option, whereby usage patterns are examined in concert with clinical, sociodemographic variables or other data sources, such as in-app text messages between users and coaches to identify reasons for app use/non-use.^{Reference Chen, Wu, Tomasino, Lattie and Mohr15}

Users weigh the benefits of using m-health apps with factors such as usability, convenience, personal risk, for example privacy, and cognitive or emotional effort,^{Reference Morton, Hole, Murray, Buzwell and Michalak13} which may not be accounted for in in-app assessments. Empirical studies utilising self-report data may not use validated self-report inventories or use them systematically.^{Reference Ng, Firth, Minen and Torous5} Thus, incorporating different data sources must balance the burden and risk of self-reporting with the benefits, emphasise replicability in the selection and use of self-report instruments, and transparency in data collection and reporting.

Toward a more comprehensive view of in-app engagement

In-app engagement must seek to support the cognitive, emotional and behavioural changes necessary for desired mental health outcomes, including symptom reduction, recovery and quality of life improvements. For this to occur, use/non-use must be connected to broader out-of-app goals, and the value of negative emotions, behaviours and cognitions relative to overall self-management must be considered. For example, non-use may be indicative of improved mental health and the need for less reliance on digital interventions.

It is tempting to want a magic, uniform formula for measuring user engagement, and this has been the appeal of usage data.

We argue instead that user-engagement indices for m-health apps must be:

• Corroborative, where different measures including usage data, symptom severity assessment scales and subjective outcome assessments (for example quality of life) are used to determine what meaningful engagement with the app entails.

• Outcome, rather than output, oriented. If m-health apps are meant to improve intervention effectiveness, then how they are used becomes more important than how often they are used.

• Process based, where we expect to see ebbs and flows in usage. Rather than labelling app users as low or high engagers based on algorithmic (non)-use, we should adopt participatory design approaches (for example journey mapping) to appreciate how different users interact with different features of the app over time.

• Expert driven, meaning that the expertise of people with lived experience and clinicians is included throughout the design process to identify salient needs (for example social support) and goals (for example establishing a routine, symptom management) and how these can be met with the app, as well as to inform aesthetic and content design choices.

User-engagement indices should be developed in parallel with the app itself, and draw upon condition-specific knowledge and multiple data sources. The COPE approach (Corroborative, Outcome oriented, Process based, Expert driven) necessitates collaboration among people with mental health conditions, healthcare providers and user-experience designers to develop m-health apps. Documenting each element of this framework would result in greater transparency about how design decisions were made, what is being measured and why, and how the resulting app fits into the broader mental health landscape.