Limitations in the current assessment of social interactions

Knowledge about social networks and cognitive function has largely been built upon structural aspects of a person's network: the simple self-enumeration of the number of people or acquaintances in an older person's life, their geographic proximity, and/or the degree to which they navigate across these social ties (e.g. how often they are in contact, familiarity among contacts). Acquiring these simple metrics has been the province of standardized questions or scales for determining the extent and structure of a person's social network.

However, the accurate collection of these critical data is challenging. First, collecting the data using surveys suffers from the common challenges of self-report data. This is particularly critical for social engagement research related to cognitive function in that one may not remember well social partners or details of interactions over time. Second, social interactions are dynamic, changing over time. This may be especially true for elderly persons who experience more frequent loss of friends through illness and death, as well as their own transitions through retirement, and change in residence. Thus, the method for capturing these data matters. Survey methods may be insensitive because of choices made in terms of particular social activities queried or the sensitivity of the levels at the intensity of activity reported. For example, in a study attempting to harmonize social activity assessments across four major studies to examine the effect of social activities on cognition, simplifications had to be made to create a comparable metric of “social activity” (Brown et al., Reference Brown2012). In one study used in the analysis, activities surveyed were volunteering, playing cards, phone conversations, visiting others, attending church, dancing, and partying. In a second study, activities queried were eat at restaurants, visit friend or relative, give dinner party, attend church, meetings of service organizations, meetings of clubs, and do volunteer work. In addition, the estimates of activity were highly variable and skewed such that the scores were ultimately dichotomized.

At the other end of the “forced-choice” environment of survey instruments are more direct self-report activity assessments. As an example, the American Time Use Survey (http://www.bls.gov/tus/) conducted by the Bureau of Labor Statistics in the US falls into this category. In the survey, the interviewer collects a detailed account of the respondent's activities from 4 a.m. the previous day to 4 a.m. on the interview day. The interviewer uses pre-codes to quickly record commonly reported activities, but records the respondent's verbatim responses for all other activities as well. These methods, along with others such as day reconstruction or diary methods, present their own challenges. They may miss more consistent trends or patterns, being generally limited by the particular day or time of year that the information is collected.

Channels of data sources available

Until relatively recently, there have been few advances in improving the quality and type of information captured for gauging social interactions. In the best of all worlds, we would like to objectively account for all human interactions over time, marking their frequency, duration, and quality. This may appear a daunting task. However, the instances of opportunity for social interaction can be readily assigned into two major categories. These include face-to-face or in-person interactions and/or interactions carried out via remote media (telephone, Internet, and written communication). Considered in this way, capturing interactions conceptually becomes simplified to an exercise in assessing the channels of communication.

Recent advances in communication technology, remote sensing, pervasive computing, and data analysis have provided the opportunity to begin to more objectively and meaningfully assess the lines of communication that are vital to social interactions. In many ways much of this progress has been the result of many of the channels of interaction becoming more amenable to automatic data capture. Several developments in particular are of note. The first is the revolution in assessing social network connectivity afforded by the cell phone or programmable mobile phones (“smart phones”)(Raento et al., Reference Raento, Oulasvirta and Eagle2009). This development is still quite new given that cellular service and the wide adoption of mobile cell phones only began in the 1990s. Nevertheless, taking advantage of this technology has facilitated large-scale and long-term study of daily patterns of activity. This activity can be captured without either the researcher or the participant needing to actively engage in data entry. The phone itself is a natural part of everyday human interaction. Data provided from assessing human interaction by measuring aspects of cell phone use (e.g. time on calls, number of calls, an individual's location) are inherently ecologically valid. Some of the most notable work using this approach has been in enumerating, and in particular, examining the dynamics of social contacts in networks. For example, young adults in a university environment were asked about their typical proximity to other individuals in the study (Eagle et al., Reference Eagle, Pentland and Lazer2009). Participant self-reports were compared with average daily proximity (within 5–10 minutes) reported through the Bluetooth scans of the smartphones issued for the study. Although most proximity was not reported (69%), when a proximity event was reported, it was usually overestimated. The average reported proximity was 87 minutes per day whereas the average observed proximity was only 33 minutes per day. Using the phone-based observational data alone, it was shown that self-identified friend dyads had distinctive temporal and spatial patterns.

Other platforms (laptops, notebooks, tablets, and interactive TV) also provide the opportunity to assess social interaction through multiple wireless channels of cellular or on-line communication. Social interaction may be captured by assessing the duration or number of events generated on a computer or via phone communication. These may take several forms: “old-fashioned” email, texting, social media interaction, or use of the computer as a phone via voice over Internet protocol.

Ultimately, face-to-face interactions may be the most salient for individuals. Much of this activity may occur spontaneously and outside of the home setting. Here multimodal (auditory and visual) aspects of engagement can be captured with automated analysis of speech and facial expression (Zeng et al., Reference Zeng, Pantic, Roisman and Huang2009). In the auditory realm, one may use micro-recorders to capture person-to-person interactions. The Electronically Activated Recorder (EAR) developed by Mehl et al. (Reference Mehl, Pennebaker, Crow, Dabbs and Price2001) is a modified digital voice recorder that activates periodically for brief periods of time to sample and record ambient sounds. Participants wear the EAR while going about their usual activities to capture for future analysis aspects of social life that normally go unnoticed (e.g. subtle interaction preferences and linguistic styles). Using this device, Mehl et al., for example, showed that the amount of conversation male and female college students have is on average about the same despite the belief that women talk more than men (Mehl et al., Reference Mehl, Vazire, Ramirez-Esparza, Slatcher and Pennebaker2007). Currently this approach is limited to snippets or brief biopsies of time because the technologies are not capable of recording for long periods of time (typically < 48 hours) and are not totally unobtrusive.

Levels of social interactions assessable

Within these various communication environments, there is an obvious hierarchy of information about social interaction that can be considered. At the lowest level simple capture of time or duration of interaction can be a powerful metric. We have shown that simple daily total time spent on a home computer by elderly people without regard for the specific activity or nature of the potential personal interaction is a sensitive measure of change among people with mild cognitive impairment (Kaye et al., Reference Kaye2013), along with other unobtrusively monitored in-home activities such as walking speed and its variability (Dodge et al., Reference Dodge, Mattek, Austin, Hayes and Kaye2012). At a higher level of information is the question of what is the nature of the interaction and with whom, such as is the individual communicating through frequent email with friends or family or as a member of an on-line community? Yet another level of abstraction is the degree to which the interaction is meaningful or valued by the individual. This latter quality may be assessed beyond self-report, facilitated or augmented by several technology-aided approaches. Phone conversations (conducted via any device) may be automatically classified into personal or business conversations (Stark et al., Reference Stark, Shafran and Kaye2012). Email or texts may be auto-analyzed (with instantaneous local processing and immediate deletion for privacy and security concerns) for emotive content (Thelwall et al., Reference Thelwall, Buckley, Paltoglou, Cai and Kappas2010).

Since much more than vocal or conversational interaction is important to optimally assess social interaction, capturing other relevant activity through additional sensors worn by social contacts is a notable evolving approach. One important potential channel of information in this regard may be provided by the addition of video-based information to assess engagement especially within interpersonal exchanges. This application can become a powerful tool for advancing the science of meaningful engagement. For example, using video capture of interpersonal interactions, one may analyze facial expressions to assess the degree to which an exchange represents true amusement or simply being polite, a quality of exchange that would be near-impossible to quantify by self-report (Hoque et al., Reference Hoque, Morency, Picard, D'Mello, Graesser, Schuller and Martin2011). In general, video capture currently requires environments where the cameras are fixed such as in an environment already outfitted with such equipment or the videocam available on a personal computing device. Google Glass represents a potential mobile system for capturing relevant visual information during interactive activities.

Other developments expanding the types of information that may be captured add multiple domains of sensed information into one device. Thus for example, a developing device, the Hitachi Business Microscope (Takaguchi et al., Reference Takaguchi, Nakamura, Sato, Yano and Masuda2011; Akitomi et al., Reference Akitomi, Ara, Watanabe and Yano2013) combines six infrared transceivers, an accelerometer, a flash memory chip, a microphone, a wireless transceiver, and a rechargeable lithium-ion battery in a badge-like form factor that allows recording of social interactive information for up to two days at a time. Such additional sensors can provide more contextual information such as location, distance and duration of physical contact with others one may interact with over time. Advances in power management, miniaturization, and electronics will likely see many new devices and applications developed in the coming years.

In-home monitoring

Several limitations to current worn or carried devices include their form factor (and resulting obtrusiveness), as well as their largely untested use in older populations. However, since many elderly people do not carry cell phones or similar devices and spend the majority of their day in their homes, more passive sensing techniques in the residential environment provide a unique opportunity space for expanding the capacity to assess social interactions. We have already pointed out the potential value of recording multiple types of data such as audio, activity, and video streams using single carried or interactive devices. The home environment provides some potential additional advantages toward capturing useful data unobtrusively. This can be accomplished by strategically placing passive infrared and contact sensors around a home to capture simple engagement metrics such as time out of home (a measure of outside or “public” socialization) or time in certain locations in the home (e.g. time in bed) (Kaye et al., Reference Kaye2011). Importantly, this approach may also allow one to assess the degree to which external events in real time may affect social interactions such as the onset of or recovery from illness (Campbell et al., Reference Campbell2011). These approaches are most readily applied to persons living alone, a growing population (approximately a third of all adults over age 65) particularly vulnerable to the potential adverse outcomes of social isolation. Currently, the presence or proximity of visitors or loved ones cannot be easily determined with precision in indoor environments where individuals do not live alone, unless body-worn sensors or tags are used. This is likely to change significantly with new developments in technology (Wan et al., Reference Wan, Paul and Jacobs2012).