Study design

Since the purpose of this paper is to describe and illustrate the advantages of the behavioural technique, this section focuses more on questionnaire design and analysis than sampling; the method could be used with most sampling techniques. The SAM questionnaire differed from standard surveys in two main ways. First, the initial section of the survey adapted the DRM technique to prompt recall of all locations the respondent had visited during the previous 2 days. Second, it used the recollections of locations prompted by the DRM to direct the respondent to an appropriate subset of 13 branched survey modules that each covered a different type of location. Most participants only completed three or fewer of these modules. Thus, the branching was extensive and the bulk of the study questions was specific to the combination of locations that each respondent had visited.

At the start of the study, participants were given strong assurance that responses were completely anonymous. An adapted form of wording from the DRM (Kahneman et al., Reference Kahneman, Krueger, Schkade, Schwarz and Stone2004; Lades et al., Reference Lades, Laffan and Weber2021) then directed them to revisit each of the previous 2 (named) days:

Starting with activities carried out yesterday, participants typed into open text boxes, which were split into morning, afternoon and evening. These responses were not analysed; the purpose was solely to aid recall for the questions to follow. Following this exercise, participants were presented with a list of potential locations, such as shops, workplaces, other homes and pubs/restaurants/cafés, and asked to select any that they had visited. The list was designed and piloted to be fairly exhaustive but also contained two ‘other’ categories (one indoor and one outdoor). The process was then repeated for the day before yesterday. Responses to the two location lists determined which of the branched modules participants completed. In addition, all respondents answered questions about visits to their home and any walking, running or cycling in their neighbourhood. At this point, the study had not mentioned COVID-19.

The modules gathered details about the type of location participants visited and behaviour at that location. For instance, the module for shops recorded the type of shop, whether they had to queue and whether staff and other customers wore masks. Similarly, the module for cafés, pubs and restaurants asked whether the visit was to collect takeaway or, if not, whether the participant sat inside or outside. For locations visited more than once, one occasion was selected at random for these more detailed questions. For all locations, participants provided information on how many people from other households they had arranged to meet there or had met for more than 15 min, how long they were present, whether soap or sanitiser was provided for cleaning hands, whether they used it, whether they kept at least 2 m from other people, whether they wore a facemask and, for indoor locations, how well ventilated it was. Thus, for each location, questions proceeded in a narrative fashion for a specific occasion. As well as exploiting the DRM, the study therefore aligned with evidence that spatial context can serve as a scaffold for episodic memory (Robin, Reference Robin2018). Question-wording was kept factual and neutral, with no evaluative questions that probed opinions or attitudes about behaviours.

In this way, as well as prompting recall, the study design differed from a standard linear survey. Respondents only answered questions about locations they had visited during the previous 48 h and identified by the DRM-prompted recall method. Aggregation of behaviours was done by the research team during data analysis. For instance, rather than asking respondents directly how many close contacts they had during the previous days, or how often they had worn a mask in public places, the analysis aggregated information across separate answers about, say, a specific shopping trip, meeting in a café, visit to a relative's house, etc., where on each occasion the participant provided information about how many people they met, for how long, in what context and whether they and others wore a mask.

Of course, we cannot be sure that aggregating responses like this produced more accurate measures of behaviour than asking respondents to aggregate their own behaviour in response to more general questions. However, researcher aggregation of behaviour from diary methods has been shown in other domains to be more accurate than self-aggregation (e.g., Fonseca et al., Reference Fonseca, Victora, Halpern, Lima and Barros2002; Heeb and Gmel, Reference Heeb and Gmel2005; McNaughton et al., Reference McNaughton, Mishra, Bramwell, Paul and Wadsworth2005; Wutich, Reference Wutich2009) and the prompted recall of events in reverse chronological order can improve recall (e.g., Loftus and Fathi, Reference Loftus and Fathi1985). Moreover, behaviours relevant to the spread of COVID-19 are likely to be more complex for survey participants to aggregate. For a respondent to assess accurately, say, how often they had close contact or wore a mask during a recent reference period, they would need to undertake the sort of detailed memory search that our method explicitly walked them through, with the relevant episodes prompted. Furthermore, this modular approach allowed the research team to not only build up a picture on how protective behaviour varied across individuals and over time, but also across different types of locations.

Following the completion of the modules detailing behaviour, SAM asked a range of psychological and attitudinal questions that included, among other things, measures of personal worry about COVID-19, perceptions of risk, feelings of tiredness with restrictions and confidence in government. The survey finished with a set of standard socio-demographic questions.

The findings below derive from 36,000 observations collected in fortnightly waves of 1,000.Footnote ¹ Respondents were recruited via existing online panels of two of Ireland's national market research agencies, which alternated between waves. Members of both panels were paid €4 for completing the survey. Quota sampling was used to obtain samples of adults (aged 18+) that were nationally representative by age, gender, social grade (defined by the occupation of the chief earner) and region. Ideally, data would have been collected from either a fresh cross-section of participants in each wave or from a longitudinal panel of repeat participants. However, given budget and, more importantly, time constraints, from the fifth wave of data collection onwards, respondents on the panels were permitted to respond to more than one wave. Repeat participation was always at least 4 weeks apart, but this was recorded, so any findings could be checked to ensure that changes over time were not confounded with willingness to be a repeat participant.Footnote ² While the socio-demographic composition of the sample was matched to Central Statistics Office control totals on every wave of data collection, some hard-to-reach groups were doubtless underrepresented, in particular as the study was conducted only in English. The survey was programmed in Gorilla Experiment Builder (Anwyl-Irvine et al., Reference Anwyl-Irvine, Massonnié, Flitton, Kirkham and Evershed2020), spread across 7 days of the week for each wave, could be undertaken on mobile, tablet or PC and took approximately 15–20 min to complete.

Illustrative evidence for policy

This section provides examples of the types of findings that SAM generated and describes how they informed policy decisions. The aim is to present a subset of results to illustrate the advantages of the method. During the period concerned, the research team presented results from the latest wave of SAM to officials in the Department of the Taoiseach every fortnight. These meetings were two-way, with policymakers often raising specific questions they wanted to investigate using the SAM data. On one occasion, in March 2021, results were discussed directly with the Taoiseach. On multiple occasions, specific findings were sent to Cabinet Committees, to the Chief Medical Officer, to the Government Information Service and to Departmental Heads of Communication. Findings were posted on the Department's website in the form a slide deck and accompanying short commentary.Footnote ³

Figure 1 demonstrates how the research team were able to aggregate data across individuals and locations to give the government a real-time picture of how the frequency and locations of close contacts were changing.Footnote ⁴ For instance, from January to April 2021, Ireland was under ‘Level 5’ public health restrictions. People were confined to a 5 km radius of their homes (unless travel was essential), schools were closed, non-essential retail establishments were closed, workers worked from home wherever possible, hospitality (cafés, pubs, restaurants and hotels) and sporting venues were closed and gatherings and visits to other homes were not permitted. The SAM data revealed that despite these restrictions, aggregating across all locations, close contacts were increasing (Figure 1, left). Because the data were collected via modules for each location visited, we were able to uncover that the main reason for the increase was a minority who were having close contact while engaging in social visits to other homes, in violation of restrictions (Figure 1, right, blue line). Further analysis at the time showed that this behaviour was spread across socio-demographic groups and that people were less inclined to take precautions (e.g., ensuring good ventilation and wearing a mask) when in other people's homes compared to other locations. This evidence was used by the government communications team who used messages specifically about home visits. Care was taken to communicate widely that the large majority of the population was not going into other people's homes, given the prevalence of conditional co-operation in collective action settings (Chaudhuri, Reference Chaudhuri2011; Thöni and Volk, Reference Thöni and Volk2018). Insights about where close contacts were most likely to occur and where people were more likely to avoid them also contributed directly to decisions about the timetable for lifting the Level 5 restrictions, which prioritised outdoor meetings and activities. Similarly, when COVID-19 cases began to rise again in Ireland in the Autumn 2021, the SAM data pinpointed the contribution that close contacts at work were making as more employees returned to workplaces after the Summer holiday period (Figure 1, right, orange line). Again, this information allowed interventions and communications to be targeted. Overall, the granularity by location that the modular design of SAM afforded was of clear benefit to policymakers.

Figure 1.

Proportion of adults in Ireland who had at least one ‘close contact’ (spending 15 min or more within 2 m with an individual from another household) with and without wearing a facemask (left) and without a facemask by location (right). ‘Outside’ refers to outdoor locations such as parks, beaches and neighbourhood streets. ‘Hospitality’ refers to cafés, pubs, restaurants and hotels.

Each fortnight, similar analyses to Figure 1 were produced for the frequency of visits to different types of locations, where and how often individuals were meeting up with people from other households and, in each location, whether people were cleaning hands, maintaining social distance and wearing masks. Each of these measures could also be broken down by socio-demographic group and vaccine status. In general, the SAM data could be aggregated in multiple ways to provide a detailed picture of where, how and among whom transmission of the virus might take place. As researchers who also worked with data from Ireland's national COVID-19 tracking survey, we found this flexibility in the SAM data to be particularly helpful. As with many other high-quality COVID-19 tracking surveys (e.g. Blom et al., Reference Blom, Cornesse, Friedel, Krieger, Fikel, Rettig, Wenz, Juhl, Lehrer, Möhring and Naumann2020; Kapteyn et al., Reference Kapteyn, Angrisani, Bennett, de Bruin, Darling, Gutsche, Liu, Meijer, Perez-Arce, Schaner, Thomas and Weerman2020), Ireland's main tracking survey asked respondents themselves to aggregate behaviours over a reference period and multiple locations, providing a self-reported frequency. By instead asking detailed questions only about locations each individual had visited and then leaving the aggregation to the data analysis, the SAM data allowed us to address additional research questions.

In fact, the SAM data contained some interesting patterns in self-reported aggregate behaviour in the context of COVID-19. Figure 2 displays the number of locations that respondents reported visiting, broken down both by the day of the behaviour and the day of survey completion. The variations by weekday were of some interest to policymakers. However, we observed that respondents recalled visiting more locations when going back through yesterday than when thinking about the day before yesterday. Overall, this effect was highly statistically significantFootnote ⁵ and implied that respondents were forgetting approximately 1-in-20 locations visited the day before yesterday. When considered across all days, this effect size is not particularly large, but there were systematic differences across days. The effect was stronger for weekdays, presumably because weekend activity was more memorable. We collected data only in relation to the previous 2 days, always asking about yesterday first and only within the context of the DRM, so we cannot know if this pattern might generalise to longer periods and other methods. However, this pattern may be of interest to others seeking to collect similar data that requires recall of behaviour. Given the disparities when respondents considered just the previous 2 days, it is possible that self-reports of behaviour over, for instance, a 7-day reference period might be subject to considerable measurement error.

Figure 2.

Reported number of locations visited on each day of the week by day of survey completion. Error bars are 95% confidence intervals. The vertical axis range is set to approximately one standard deviation of the number of locations across individuals.

Other advantages of collecting multiple, relatively accurate measures of behaviour across contexts are the possibilities for investigating drivers of behaviour. It was apparent by Summer 2021 that movements in the behavioural measures collected by SAM were closely related to official COVID-19 case numbers, which were released by the Department of Health and reported on news programmes every evening in Ireland. Figure 3 displays the relationship across the period. The left-hand chart shows variations in monthly mean case numbers and a mean risk score for individual behaviour calculated from the SAM data. The right-hand chart is a scatter plot of month-on-month changes in the variables.Footnote ⁶ Throughout the period, we experimented with different ways of aggregating the behavioural data into a single metric to capture variation in the riskiness of behaviour, in order to make comparisons of this sort. This score displayed in Figure 3 varied from zero to four, based on whether an individual had close contact the previous day, visited two or more locations per day, met up with more than one person from another household per day and took mitigation actions (wearing a mask, keeping 2 m from others and cleaning hands) less than two-thirds of the time in each location they visited. These criteria were set to capture variation, as each of these behaviours was undertaken only by a minority of the population, but the majority undertook at least one of them. As more people were vaccinated throughout 2021 and public health restrictions were gradually lifted, there was a clear upward trend in this risk score. However, on a month-by-month basis, the change in behaviour was strongly and negatively correlated with the (logged) case numbers.Footnote ⁷ Of course, there would have been some reverse causality in this relationship. The extent to which increased activity led to higher cases (albeit with a time lag) would have dampened the estimated negative relationship.

Figure 3.

Relationship between riskiness of behaviour and new daily cases of COVID-19 (as released by Ireland's Department of Health). Risk scores (0–4) are based on whether an individual had close contact the previous day, visited two or more locations per day, met up with more than one person from another household per day and took mitigation actions less than two-thirds of the time. The left-hand chart displays the time-course of new cases and risk scores. The right-hand chart plots the change in risk score from the previous month against the change in (log) daily case numbers.

For policymakers, the fact that behaviour was tied to reported case numbers was important. There was a debate within policy circles about the advisability or otherwise of continuing to release daily case numbers publicly. The SAM data suggested that ceasing to do so might remove a factor that was moderating behaviour.

The SAM data were able to offer further insights into the psychology behind social activity. The behaviours recorded in the modules could be modelled to identify relationships with socio-demographic background characteristics and the psychological scales collected at the end of the survey. Such exercises were undertaken at multiple points during the study period. An example is provided in Figure 4, which reports output from a model estimated for the period under Level 5 restrictions (January to March 2021). The chart shows standardised coefficients for multiple psychological scales in a model where the dependent variable was the risk score described above. The strongest psychological determinant we recorded throughout SAM was people's response to a question about how worried they were personally about COVID-19. Further work showed that this variable was linked to news about case numbers and was less associated with worry about contracting the virus than with worry about friends and family and whether the health service would be overwhelmed. For policymakers, this analysis therefore further supported ongoing communication of the epidemiological situation, including hospitalisations. Another variable of note was how tired people were with sticking to restrictions. This was a constant theme in public debate, as some politicians expressed doubt about the public's willingness to keep complying. These views were often reinforced by comments from industry lobby groups that stood to gain from restrictions being relaxed. They were also bolstered by some researchers, assuming that declining compliance must reflect some kind of ‘pandemic fatigue’ (World Health Organization, 2020; see also Harvey, Reference Harvey2020). While the SAM data suggested that fatigue was a factor, Figure 4 points to other important psychological factors. Notably, these included a scale that directly asked respondents which was more important to them, the burden of the restrictions or preventing the spread of the virus. Other important factors were whether people viewed the public health restrictions as easy to follow, whether they viewed the overall set of measures as coherent or contradictory and whether they thought other people were following recommendations to prevent the spread of the virus and confidence in the government's handling of the pandemic. Most of these factors were chosen for inclusion in the survey and for analysis based on how they mapped onto the literature on drivers of co-operation in collective action problems (e.g. Fehr and Schurtenberger, Reference Fehr and Schurtenberger2018): the value placed on the common gain (worry, burden vs spread), belief that the co-operative strategy will lead to the gain (ease and coherence of restrictions and confidence in the government) and belief by conditional co-operators that others will also comply. Other factors were included in the survey because they were receiving public attention at the time. These were generally less influential, including deterrence for non-compliance and whether the public understood how the virus transmitted, as captured in Figure 4 by variables assessing people's belief that they might get caught and fined for violations and their self-reported understanding of transmission (similar results were found for an objective measure of comprehension too).

Figure 4.

Example of output from a model of psychological determinants of the risk score during ‘Level 5’ public health restrictions based on SAM data collected from January to March 2021 (n = 4,000). The risk score is as defined in Figure 3. Standardised coefficients (ordinary least squares) estimate individual-level relationships between the relevant psychological scale and the risk score, controlling for socio-demographic background characteristics. Error bars are 95% confidence intervals.

Figure 4 is illustrative; multiple exercises of this sort were undertaken with the SAM data across the 18-month period and the drivers of behaviour changed somewhat at different stages of the pandemic.Footnote ⁸ However, the SAM results changed policymakers’ perspectives and assumptions about why people were behaving as they were, providing the government in Ireland with a different picture of what was behind public behaviour than they were getting from the contemporaneous public debate. The fact that the data consistently showed how important it was for compliance that the overall range of public health restrictions was perceived as coherent informed the order with which restrictions were lifted during Summer 2021 and how this was communicated. The stronger influence of worry relative to fatigue gave the government confidence that the public would stay the course if the epidemiological situation warranted ongoing caution.

SAM data were also used to assist regulations and communications with industrial sectors. Responses provided information about which kinds of shops and workplaces were making greater efforts to provide safer environments. A specific example arose in relation to the mandatory checking of vaccine certificates in cafés, pubs and restaurants, which was introduced in late July 2021. Respondents who had eaten out were asked in the relevant survey module whether they had been subject to a certificate check. Figure 5 shows that initial compliance was fairly poor. After SAM revealed a still higher proportion of establishments not performing checks, industry representatives were publicly called into meetings with the government to address the issue. Non-compliance declined markedly thereafter.

Figure 5.

Proportion of occasions that indoor diners reported not having COVID-19 vaccine certificates checked on the door of pubs, restaurants and cafés by month.

The findings shown in this section offer a selection of the sort of information that SAM supplied directly to the Department of the Taoiseach, which had overall responsibility for pandemic policy. Many other findings informed policy and government communications. One important element was myth-busting. For instance, the SAM data repeatedly demonstrated that non-compliance with restrictions and more risky social activity was not concentrated within specific socio-demographic groups, despite some public discussion that pointed fingers at young people, men and specific geographic locations. Another finding was the consistently higher price being paid by younger adults in terms of lower mental well-being while restrictions were in place. As the vaccination roll-out progressed, SAM provided detailed comparisons of the behaviours of vaccinated and unvaccinated people who were subject to different restrictions on activities. The detailed information on social activity by location was also used to develop a risk model to determine where rapid testing systems would generate the most benefit (Mallon et al., Reference Mallon, Horgan, McAloon, Lunn, Little, Beck, Bennett, Shaver, McConway, O'Regan and Whelan2022).