3.1. Concurrent vs retrospective collection

We will now explain why we recommend collecting open-text data retrospectively, after collecting the quantitative response (and probably on a separate survey page). The more historically widely used ‘think aloud’ protocol involves participants verbally explaining their thought processes concurrently, that is, while they work through an experiment (Eccles & Arsal, Reference Eccles and Arsal2017; Sela et al., Reference Sela, Ab and Pau2000). The process we describe here not only asks participants to write their thought process rather than speak it, but also to do this retrospectively (albeit only seconds later), after they have already completed that part of the task and provided their quantitative response. Due to some high-profile studies (e.g., Haidt, Reference Haidt2007; Johansson et al., Reference Johansson, Hall and Sikström2008; Nisbett & Wilson, Reference Nisbett and Wilson1977), there is a prevalent view within cognitive science and JDM in particular that if you ask people why they gave a particular response after the fact, they will simply ‘confabulate’ or make up reasons. We believe this is an erroneous leap of logic, unfortunately common in work on human biases: just because some experiments show that some people confabulate sometimes, in certain situations, it does not follow that all people will do this always, in all situations. More generally, we agree with a growing view that our discipline has exaggerated the level of bias in human reasoning (e.g., Lejarraga & Hertwig, Reference Lejarraga and Hertwig2021; Madsen et al., Reference Madsen, de-Wit, Ayton, Brick, de-Moliere and Groom2024; Weiss & Shanteau, Reference Weiss and Shanteau2021). We typically conduct our experiments in contexts where participants have little reason to feel defensive or ‘on the spot’ (unlike the above-cited work), with minimal risk to their ego or identity if they cannot explain their reasoning. Consequently, we have seen little reason to be concerned about this issue. Many of our participants are open to admitting complete uncertainty about their choice or response (e.g., ‘I don’t know’) or reporting intuitive responses (e.g., ‘I went with my gut’) without seeming to feel the need for a well-articulated answer.

However, if one wishes to study phenomena where this might be an issue, an additional layer of skepticism should be applied by both the researcher and any reviewer or reader. For example, one might face such an issue if studying decision-making by professionals such as doctors, asking them to explain their reasons for making a medical decision, especially if it led to a bad outcome. While much of their decision-making is likely to be intuitive and based on extensive experience (e.g., Hall, Reference Hall2002), they may feel social pressure to provide a well-thought-out explanation. This example highlights that confabulation may, in fact, be less of an issue with written reports collected via an online survey, which could be a benefit over a verbal report approach in some cases. It is hard to imagine a lower-pressure situation than conducting an online survey at home pseudonymously for a faceless researcher (Reips, Reference Reips2002).

However, there are good reasons to think that writing your thought process concurrently, that is, while solving a task, is likely to change cognitive processes, possibly more so than verbalizing. Writing is slower, and it is well known within the therapeutic world that expressing cognitive processes through methods like journaling opens up a complex range of metacognitive processes, allowing individuals to examine and reconsider their thinking (e.g., Hubbs & Brand, Reference Hubbs and Brand2005). This might improve performance in a range of cognitive tasks. Tentatively supporting this, the first author has unpublished data showing that accuracy rates on a simple Bayesian problem solving task rose from 5.1% (n = 99) to 16.7% (n = 60) by providing brief ‘think aloud’ training plus a ‘concurrent’ open-text box (using binary logistic regression: Z = 2.3, p = .02), which required participants to explain their reasoning before being allowed to proceed to the next page to provide their numerical response. Details of this simple study, including the dataset and materials, can be found at the online repository for this paper: https://osf.io/cyjhd/. This runs in contrast to the largest review of the effect of concurrent ‘think aloud’ protocols, which Fox et al. (Reference Fox, Ericsson and Best2011) found did not typically affect performance, possibly supporting a difference between verbalizing and writing here.

This is the primary reason we recommend retrospective collection of qualitative data after the quantitative response has been made: collecting it concurrently may affect your quantitative data in unpredictable ways and make it less comparable with other purely quantitative studies. In many previous studies, we have included the open-text box on the same page immediately below the quantitative response. However, Lerner and Tetlock (Reference Lerner and Tetlock1999) and Ranyard and Svenson (Reference Ranyard, Svenson, Schulte-Mecklenbeck, Kühberger and Ranyard2011) make the good point that if participants anticipate having to explain their thought process, this could still influence their reasoning (albeit likely smaller than being forced to write out their thinking beforehand). For a similar reason, they recommend not advertising to your participants that they will be providing qualitative data. However, there is a tension and trade-off here. If the open-text box, which many participants experience as an effortful task, comes as a surprise, they may be less willing to engage with it, and you may get poor-quality, low-effort responses, or they may even drop out. You are also increasing the time between quantitative collection and their explanation, potentially allowing more time for memories to fade or crystallize. We therefore do not have a definitive position on this, and your choice may depend on whether you want to minimize the possible impact on the quantitative data or maximize the quality of your open-text data. Future studies comparing both methods would be valuable, and researchers starting out with these methods may want to use both during piloting to see if there are any obvious impacts on either kind of data.

3.2. Frameworks for judging the validity of the data

Generally, it is crucial to be aware of and think deeply about the type of cognitive process you are attempting to study and the likely limitations of written, retrospective reports for this type of process. We tend to operate within a pragmatic framework broadly in line with Ericsson and Simon’s (Reference Ericsson and Simon1980) model. Participants will be able to report the cognitive process of interest, if they have (1) insight into it (i.e., if it is available to their conscious mind), if they (2) put in the required effort to introspect, and if they (3) choose to be honest in reporting it. These are the three key principles we use to consider potential limitations in our data. The first author has conducted work where the cognitive process of interest simply and clearly emerged directly from the open-text data (e.g., Dewitt et al., Reference Dewitt, Hsu, Lagnado, Connor-Desai and Fenton2019, the example in Figure 2), while in other work, participants’ responses did not mention the target cognitive process (e.g., Dewitt, Adler, et al., Reference Dewitt, Adler, Li, Stoilova, Fenton and Lagnado2023). Whether participants can report the cognitive process of interest can be informative in itself, providing clues about the nature of the process. For example, the first author concluded from that latter work that there was some hidden assumption at play, informing the design of future experiments.

Gricean principles of conversation (Grice, Reference Grice1975) have also been valuable in understanding the limitations of this kind of data. Participants engaging in these open-text responses are effectively having a ‘one-shot’ dialogue with us as researchers, aiming to communicate what they believe are the most critical aspects of their reasoning within the constraints of time and effort they are willing to invest. Consequently, much of their thinking might remain unexpressed, and they often write only one or two sentences. In our research area, if a participant mentions a cognitive process, we typically consider it to be approximately true, unless there is a reason to suspect confabulation. However, it would be a mistake to infer that the absence of a mention implies the absence of an experience or thought. Participants might omit certain details, assuming they are self-evident or not crucial to communicate at that moment, or the cognitive process may have occurred but be less than fully conscious. This creates an asymmetry in their evidential value: they can be very powerful evidence for the presence of a cognitive process (e.g., see Figure 4) but are generally a poor form of evidence for the absence of one (although see Figure 6 for a counterexample). Concurrent collection may be less prone to this asymmetry (Kuusela & Paul, Reference Kuusela and Paul2000).

Figure 4

Indicative quote from the ‘legal detainment’ running example.

In general, we must remain aware that open-text data, like quantitative measures (e.g., Toomela, Reference Toomela2008), are an indirect representation of the individual’s cognitive process mediated by a communication process. As Ranyard and Svenson (Reference Ranyard, Svenson, Schulte-Mecklenbeck, Kühberger and Ranyard2011) also remind us, the mental representations of a person engaged in a decision are likely in continuous flux during the decision process, and even after. We likely capture a relatively crystallized and ‘cleaned’ version of what actually happened in their mind using this method. The benefits are that these data are easy to obtain and do not compromise the validity of the quantitative data, but if one wants to fully map out those changing representations during a decision process, they may wish to turn to the concurrent ‘think aloud’ approach or other process tracing methods (Schulte-Mecklenbeck et al., Reference Schulte-Mecklenbeck, Kühberger and Johnson2019). Generally, inferences from such data must be analyzed and communicated to readers critically and tentatively, with consideration to how likely these issues are given the phenomenon and participants under study and the method used to collect the data.

4.1. Regarding the use of large language models

Before we explain this process, which at its core involves two human beings, we are aware, we may need to address the potential involvement of large language models (LLM’s). LLM’s are a valuable tool for researchers (e.g., Demszky et al., Reference Demszky, Yang, Yeager, Bryan, Clapper and Chandhok2023), including for analyzing very large amounts of text data that would be unfeasible for a human being (Bhatia et al., Reference Bhatia, Walasek, Slovic and Kunreuther2021; Feuerriegel et al., Reference Feuerriegel, Maarouf, Bär, Geissler, Schweisthal and Pröllochs2025). However, there is also much disagreement over the role they should have within science (Binz et al., Reference Binz, Alaniz, Roskies, Aczel, Bergstrom and Allen2025). They have the potential to reduce workload and even provide an alternative perspective; however, while we will give suggestions for how they could be integrated at various stages, we argue that they should never replace the two human coder roles due to the need for independent and accountable oversight.

The use of an LLM to replace the second coder, that is, to ‘validate’ the first coders’ scheme and coded responses, which Bijker et al. (Reference Bijker, Merkouris, Dowling and Rodda2024) suggest as a future possibility, would be, we think, highly problematic. As will become clear throughout the following sections, the role of the second coder is not as a passive automaton, but as a curious person with an independent will, tasked with pushing back against the first coder and questioning and critiquing their coding. Much of the advice in the following sections will involve ways to maximize the independence of the second coder to increase your reader’s confidence that they provided this robust check. Second coders also provide a second person ‘in the room’ that should be named in the paper (and ideally, a named author) and could be approached by an inquisitive reader to check that the first coder/author’s depiction of the process is accurate.

LLM’s are not well-suited to this role. First, considering off-the-shelf versions, there is, of course huge scope for iterative prompt engineering to achieve a desired outcome. LLM’s passivity (and even sycophancy) makes them ripe for intentional or unintentional manipulation by the first coder. While a researcher can provide extensive documentation on their use of prompts, this is a huge amount of work for a reader/reviewer to process, and it is easy to conceal result-hacking behavior. While an LLM specifically developed or trained for this kind of purpose (e.g., Alaswad et al., Reference Alaswad, Kalganova and Awad2023; Binz et al., Reference Binz, Akata, Bethge, Brändle, Callaway, Coda-Forno, Dayan, Demircan, Eckstein, Éltető, Griffiths, Haridi, Jagadish, Alexander Kipnis, Kumar, Ludwig, Mathony, Mattar and Modirshanechi2024) is likely to be more accurate at categorizing responses, this provides even more opportunity for the primary researcher’s desired outcomes to become embedded in the model. This would require a huge level of scrutiny and expertise by readers and reviewers to be confident that training did not in some way produce the desired result.

However, there are two ways in which we can see LLM involvement being valuable. First, the LLM could be involved in the development of the coding scheme in tandem with the first coder. We add a cautionary note here, as creating the coding scheme (the role of the first coder) is an act of directed creativity driven by a deep understanding of the purpose of the program of research, and so, while they could prove useful as a collaborative partner, we would not recommend just handing this over to an LLM.

As a second route, and one we would prefer, for large datasets, the method advocated in this guide could be adapted, as Rodger et al. (Reference Rodger, Sanna, Cheung, Raihani and Lagnado2025) did, by the first coder creating the coding scheme in a random subset of the data, then using an LLM to apply it to the whole dataset. This introduces a new issue beyond the first coder’s potential researcher bias: the LLM’s accuracy. These need to be checked separately. For the former issue, we would recommend having the second coder independently validate the scheme in that initial subset before it is handed over to the LLM. Following this, as Rodger et al. did, both first and second coders should then check a new random subset of the LLM’s coding as a validation check on its accuracy.

One final reason to advise minimizing the handing of this analysis over to an LLM is that, while the process can be fairly time-consuming (typically half an hour per 100 responses), we strongly believe that this is time well spent. We have learned a huge amount about our domain of expertise by deeply engaging with participants’ explanations of their thought processes. We do our best to find patterns in the data and present these to our readers, but they will never learn as much from our report as we learned from those hours spent reading themFootnote ³ . This applies to second coders too, who are typically students and, in our experience, learn a lot from the process.

4.2. Stage one—Creation of the coding scheme by the first coder

Stage one involves creating a coding scheme, undertaken by the first coder. While this task is usually performed by an individual, it can also be completed by a team acting as a collective first coder (or as mentioned above, a human-LLM team). In this case, none of the team members should subsequently assume the role of the second coder.

There are no ‘rules’ governing stage one. This phase can be compared to the ’theory generation’ phase in the scientific method, where the focus is on idea generation without predefined constraints. Any strong guidelines provided in this section cover the process for validating that coding scheme using a second coder (akin to the ‘testing phase’ within science). Figure 6 contains some information on how the coding scheme was developed for the running example.

Figure 6

An example of a ‘no information/evidence’ quote from the running ‘legal detainment’ example.

However, some advice can be offered. As with any analysis, it is crucial to keep your broad research question at the forefront, as it provides the purpose and direction for your work. We typically do this analysis in a simple spreadsheet program, creating new columns to represent codes (see Figure 7 for a depiction of a coding scheme in development). In the creation of codes, utilizing existing theoretical frameworks can be helpful. For example, the second author collected open-text data during a study examining information search strategies. By considering known search strategies, such as confirmation bias, during the coding process, participants’ responses were mapped onto well-established cognitive processes (Liefgreen et al., Reference Liefgreen, Pilditch and Lagnado2020). Of course, you might be studying something novel without previous literature to use as a guide. Either way, we recommend you initially simply read all your qualitative data with your research question in mind, either making notes on the key categories you observe or creating columns to represent them (you can modify/delete these later). Your aim is to identify similar cognitive processes or representations across participants’ responses that are relevant to your research question/might be interesting to tally. Your label for each code should capture the core essence of the cognitive process you are aiming to catalogue. Typically, we only end up with 5–10 codes, which is very different from a typical content analysis on larger forms of data, which can easily stretch to 100+. Some of our more complicated work has stretched to 20 codes, but this can become unwieldy and hard to communicate to your reader, and we recommend aiming toward the lower end of this scale (around five). If more complicated code structures are necessary to represent the data, you can organize them into nested groups. However, we recommend avoiding using the word ‘theme’ to prevent confusion with thematic analysis.

Figure 7

A screenshot of a coding scheme in development with main codes ‘Guilty more likely’, ‘Guilt less likely’, and ‘No information/evidence’ as well as ‘Unclassified/other’. The final scheme ended up with five main codes.

While in early versions of implementing this process, we thought that the first coder should define their coding scheme before doing any coding, we have found over the years that this process is often more iterative than linear. Applying your developing coding scheme to your data, that is, simultaneously creating code columns and assigning responses to those codes, helps you identify what works and what does not, allowing you to refine it as you go. It is crucial, however, that once the coding scheme is finalized that the first coder revisits all the data coded during the development phase to ensure that every participant has been coded according to that final scheme. This is typically the process that we use: working through the qualitative data within a spreadsheet, creating columns to represent codes and assigning responses as we go, changing the codes’ titles, removing, or combining them as seems fit. The process is not complete until the first coder is happy with all the codes and that all the responses have been coded with the final version of the coding scheme. Many examples of such spreadsheets can be seen on the online repository: https://osf.io/cyjhd/. As we will see in a future section, the process of communicating this to the second coder may also prompt further changes.

4.2.1. Unclassified/other

While your ‘main’ codes will differ across studies, we recommend you always include a special code labelled ‘unclassified/other’ (Figure 7: far right, highlighted grey). While this code is somewhat like a ‘bin’ it is not simply for junk responses. If a person writes anything other than the research language (English for us), including spam responses such as ‘dahsdiowqn’, we will delete all their dataFootnote ⁴ . However, as long as the response is in English, we do not remove them from the study even if their response indicates low engagement, and we advise against deleting participants based on their qualitative data, other than clear spam like above. Generally, data removal on this basis is too subjective and could lead to p-hacking if these participants are also excluded from the quantitative analysis. Examples of responses coded as unclassified/other can be seen in Figure 8.

Figure 8

An example of some quotes coded as ‘unclassified/other’ in the running ‘legal detainment’ example.

So, the ‘unclassified/other’ code is not just for junk responses and is separate from the decision to remove data. Many participants might say things like ‘It just felt right’, which is a perfectly reasonable response, but provides little insight into their cognitive process and is unlikely to be assigned to a main code. Alternatively, they might express a highly unique view, which, while interesting, is not shared by anyone else. Coding is a process of identifying common cognitive processes, but if you wish to include a paragraph on unique but interesting responses at the end of your qualitative section, this is perfectly acceptable.

Once you have decided on your main codes of interest, the ‘unclassified/other’ code serves as a place for everything else, including some perfectly valid and interesting responses. There will always be more interesting elements in a dataset than you can condense into a readable format for your audience, and qualitative coding is already quite time-consuming. Therefore, you need to pick a focus and stick to it. It is important to note that the coding scheme presented to the reader reflects what the first author found interesting with respect to their research question. The second coder (more on this soon) provides some validation that these elements are genuinely present in the data. However, this does not imply that there is not something else interesting in the data, and providing your data in an open repository could allow other researchers to use your data to answer a different research question.

4.2.2. Confidence threshold for assigning main codes

We recommend that you adopt a conservative approach to coding, placing a response in ‘unclassified/other’ if you do not feel (subjectively) ‘confident’ that it should be assigned one of your main codes (see Figure 8 for examples and Figure 9 for further thoughts on the ‘confident’ threshold). While it may be tempting, especially when some part of you wants a particular result, to include more responses in your main codes, doing so actually just introduces more ‘noise’. By restricting the responses to those you are confident about, your relative frequencies between codes (or across conditions), which will be your main focus of analysis, should be less noisy. If both you and your second coder adopt this approach, it also increases the likelihood of achieving decent inter-coder agreement, avoiding potential complications (see stage three for when things go wrong).

Figure 9

Am I confident enough?

However, this is a relatively ‘latent’ form of content analysis, with the coders using their judgement as to whether they think a given cognitive process, represented by a code, occurred. Participants will often express themselves in a casual or rushed way and so you should not only be assigning a code if the participant literally writes it out word for word. Your subjective confidence as an intelligent human being with expertise in the domain should be your guide, and the independent second coder provides a check on you reading between the lines too much. Furthermore, it is important to recognize that this process is fairly time-consuming (very roughly half an hour per one hundred responses, but one tends to get quicker as the analysis continues), and so we do not advise spending minutes agonizing over whether you are confident enough on a particular response. Instead, if you cannot decide within around 10 seconds, you are probably not confident enough, and you should just put it in unclassified/other. If you spend minutes on each response, you will probably come to loathe this process and never want to do it again. Remember that the final set of codes will be determined by both you and the second coder, and so many ‘incorrect’ decisionsFootnote ⁵ on your or their part will be picked up this way. In the end, some small number of responses will be in unclassified/other that should perhaps have been in one of the main codes, but these should be minimal and will make little impact on your overall frequencies. An example of a more ambiguous ‘unclassified/other’ coded response can be seen in Figure 10.

Figure 10

An example of a more ambiguous quote coded as ‘unclassified/other’.

4.2.5. Exploratory vs hypothesis-driven

The vast majority of the time, this kind of qualitative analysis and any associated comparisons of coding frequencies between conditions should be exploratory. We agree with the Scheel et al. (Reference Scheel, Tiokhin, Isager and Lakens2021) criticism that our discipline is often too eager to test hypotheses when we are still actually in exploratory mode, and also with papers like Glass (Reference Glass2010) that research questions should be primary drivers rather than hypotheses. However, if you do have a strong hypothesis with respect to, for example, coding frequencies across conditions, there is no reason it cannot be preregistered like any quantitative hypothesis. However, if you intend to make such strong claims with your qualitative data, you should anticipate greater scrutiny of your analytical approach.

A hypothesis-confirmatory qualitative analysis would typically really need to follow an exploratory study, as even with a strong theoretical foundation, it is difficult to predict what participants will say. However, if you have conducted an initial exploratory study, it might be reasonable to predict in a follow-up study that certain responses will occur more frequently if you manipulate certain variables or change the framing (e.g., see Figure 11).

Figure 11

An example of distinguishing between hypothesis-driven and exploratory coding.

4.3. Stage two—Second coder recruitment, communication, and coding

While recruiting and working with a second coder may seem straightforward, there are many pitfalls to avoid. The purpose of this process is to validate your coding scheme and provide greater confidence to your readers that the coding scheme and associated code frequencies are not merely a fiction of the first coder’s mind. To achieve this, it is essential to minimize the influence of the first coder’s desired outcomes on the second coder, thereby maximizing their independence (see Figure 12 for some thoughts on this).

Figure 12

An example of deciding on the level of independence needed for the second coder.

This independence can be challenging to achieve because the first coder will often recruit the second coder and has various kinds of ‘power’ over themFootnote ⁶ . For instance, second coders are typically added as authors on papers, so they may feel pressure to please the first coder (usually the first author) to secure their authorship. Additionally, more senior authors often recruit students, creating a seniority disparity. Lastly, the first author is usually the authority on the paper and coding scheme, having made it, and possesses the overall vision for the paper and analysis. To address these issues, an environment must be established where the second coder is not only given permission but is actively encouraged to hold the first coder accountable. We recommend including a statement like the following when sending coding instructions to the second coder.

When choosing a second coder, you face a tension. Ideally, your second coder would have no prior involvement with the paper and should not be too familiar with you either. However, this must be balanced with the need for the coder to comprehend the study, especially if it is complicated. It also must be balanced with pragmatic constraints of who is available. Independence becomes more crucial if you intend to make strong claims regarding coding frequencies. If your qualitative component is purely exploratory and tentative, the independence requirement can be somewhat relaxed (e.g., see Figure 12). However, if you are aiming to make strong claims about coding frequencies across conditions to support a theoretical position, it is essential to find a second coder unaware of those aims and to maintain that unawareness throughout the process. This might require looking beyond your immediate lab, especially if you have presented early ideas locally. In terms of competency, we recommend recruiting someone within the same field who ideally has some previous qualitative experience to understand the requirements. If they have no experience with this or similar processes, having them read this paper would be a good idea. Above all, you should be transparent in your paper about the second coder’s background and any previous involvement in the study.

4.3.1. Second coder instructions

You will need to send the second coder written instructions as well as a prepared spreadsheet for them to conduct their coding independently. There are two options we have used here for communicating this to the second coder:

1. 1) Text only: if independence is the primary concern, and the coding scheme/code definitions are relatively straightforward, we recommend just sending the instructions along with the second coder spreadsheet by email or other text channel.

2. 2) Text + calibration meeting: if communication clarity is the primary concern, for example, if the coding scheme is complicated or the definitions are quite nuanced, you could have a calibration session with your second coder where you go through a few responses together to agree on what they would be. This meeting should cover the minimum number of responses possible because these responses should not be included in any final coding frequencies.

Whichever of these two routes you choose should be clearly stated in the paper, and for route two, the number of responses used for calibration (and therefore removed) should be reported. If the calibration meeting leads to any changes or clarifications in the definitions of the codes, the second coder’s instructions should be edited to reflect this. We have several examples of second coder instructions and accompanying spreadsheets ready for the second coder (e.g., with condition / quantitative data hidden) on the online repository: https://osf.io/cyjhd/ (also see Figure 13 for an example of a single code). When sending these, the same balance of understanding vs independence continues. The second coder will need some basic background on the study (and potentially the phenomenon under study, depending on their expertise) to do their job effectively, but you also do not want to overly influence them, and certainly not to give away what you expect to find (e.g., a large number of participants saying X, or more participants saying X in one condition than another). Again, it is important to make sure that all information you send to your second coder is clearly available to your readers in an online repository so they can judge for themselves if the process was robust.

Figure 13

An example of a code definition and example code sent to the second coder for the running ‘legal detainment’ example.

After providing some background, your instructions should include your ‘codebook’, containing your coding scheme. This codebook should contain definitions for each code (typically a few sentences) along with one to two exemplar quotes that you think perfectly capture the essence of each code. Importantly, not all responses need to be as perfectly articulated as these exemplars: participants often vary in their clarity of expression (more on this soon). As mentioned previously, creating these instructions may prompt you to rethink your coding scheme, which is normal and should be embraced. It is much better to get it all straight now than face a large number of disagreements in stage three.

Second coders, especially inexperienced ones, sometimes struggle with how much to ‘read between the lines’. Generally, it is better to be conservative but otherwise to just use their best judgment. We instruct them to apply the code if they, personally, are ‘confident’ (see Figure 9) that it is correct. If they feel like they are ‘guessing’, they should just place the response in ‘unclassified/other’. However, this is not a surface-level form of content analysis, where the person has to literally say an exact phrase or word. Both coders use their natural human intelligence (and professional domain knowledge) to decide whether they think the person is trying to express the cognitive process of interest, and the agreement between them provides some (not absolute) validation of the final set of coded responses.

There is an issue of communication in all of this. The first coder has an idea of what their code means in their head and attempts to communicate this, but this may not always be entirely effective. The second coder should be encouraged to ask questions if they are completely unclear about a code’s meaning, but should otherwise do their best autonomously to maximize their independence. If you are going down route one (text only), you should avoid having repeated back-and-forth conversations with your second coder about whether particular responses fit a code; they must code independently to the best of their ability. If this does not seem to be working, it might be best to go down route two and have a calibration meeting.

4.4. Stage three—Disagreement resolution

After the coding is complete, the first and second coders should meet to discuss responses they disagree on and compile a set of ‘final’ agreed coded responses. Before beginning the discussion, it is beneficial to remind the second coder that they have your permission to push back and hold you accountable.

During the discussion, it seems to be a good rule that if the two coders cannot agree within a short time (e.g., one minute), then the response should just be put into ‘unclassified/other’. This mirrors the personal ‘10 second’ version of this rule as a lone coder if one is struggling to decide. This achieves the two goals of reducing the chance that the first coder dominates the second and also reduces the chance of a disagreeable dynamic emerging. If both coders remember that it is not ‘bad’ for a response to go into ‘unclassified/other’ because you do not really want to pollute your main coding frequencies with more ambiguous responses, then this should ensure a smoother process. While the time recommendation here might seem short, there may be many responses to deal with, and you will see the need to move relatively swiftly once you begin this process. Typically, this meeting will take 30 minutes to an hour, but it could take a lot longer if you exhaustively debate every disagreement.

In the final spreadsheet you upload to your online repository, you should show both coders’ initial codes and the final agreed code after discussion. In the online repository associated with this paper (https://osf.io/cyjhd/), we provide several example spreadsheets, including one for the running ‘detainment’ example and another for a different study (Dewitt, Adler, et al., Reference Dewitt, Adler, Li, Stoilova, Fenton and Lagnado2023). We have also included a template spreadsheet which is set up to display which codes between the first and second coder require discussion, accompanied by a ‘read me’ document with some tips on how to use it. Also see Figure 14 for some possible metrics for inter-coder agreement.

Figure 14

Examples of statistics that can be used for calculating inter-coder agreement.

This transparency allows an inquisitive reader to see what transpired during the disagreement resolution discussion. For example, in Dewitt, Adler, et al. (Reference Dewitt, Adler, Li, Stoilova, Fenton and Lagnado2023), there were 28 disagreements for one code, all of which stemmed from the second coder assigning the code while the first coder had put them in unclassified/other. This might suggest that the second coder was being less conservative, with a lower confidence threshold for assigning the code than the first coder (i.e., ‘over-coding’). The spreadsheet shows that nine out of the 28 times, the final agreed code reflected the second coder’s initial coding, while in 19 cases, it reflected the first coder’s. Although this slightly favors the first coder, it does not seem overly concerning, and a reader could look to see if this pattern was there across multiple codes. It is good practice and a sign of transparency to make a note of such things for inquisitive readers, for example, by using the comment function in an appropriate part of the spreadsheet.

One approach to minimize the first coder’s dominance during disagreement resolution (and save time at this phase) is, before the discussion, to just assign any response that either coder placed in ‘unclassified/other’ to that category in the final agreed set of codes. We used this method in Dewitt, Adler, et al. (Reference Dewitt, Adler, Li, Stoilova, Fenton and Lagnado2023), resolving 31 of 91 disagreements this way, significantly reducing the proportion resolved through potentially biased discussions. While this need not be a universal rule, as there are cases where a coder genuinely errs by categorizing responses as ‘unclassified/other’, this highly conservative approach can make this stage more robust, especially if low inter-coder agreement is achieved at the end of stage two.

4.4.1. When things go wrong

What if you encounter very low inter-coder agreement, discover that your second coder has completely misunderstood your code definitions, or realize during the discussion that you need to change your coding scheme (e.g., the second coder thinks you missed an important code, or two of your codes are actually the same thing)?

While we have never faced major issues of this sort, we have heard from other researchers who have. Following a structured process like the one outlined here should minimize the chance of these issues occurring (if you want to get some practice now, check out Figure 15). Small changes can often be fixed with full transparency in the paper and the online repository. For example, if during the discussion process, the first coder realizes that the second coder’s interpretation of a particular code is actually more accurate than the first coder’s, the spreadsheet might show a large proportion of discrepancies resolved in favor of the second coder’s initial code. This might be followed by a change to the coding definition as it is written in the paper, which is not a problem.

Figure 15

An opportunity for the reader to get practice as a second coder (https://osf.io/cyjhd/).

However, if your process has gone completely awry and your second coder is no longer the independent checker you need, you may need to start over. In this case, consider the process so far as a ‘revision’ of your coding scheme or a ‘dry run’. This is perfectly acceptable as long as it is explained openly. Essentially, you just unintentionally hired someone to help refine your coding scheme before engaging in the second coder process, so the person you thought was your second coder has now become part of your first coder team. You now need to hire a completely new second coder to start again. Clearly detail this in your paper, explaining that you hired a second coder, realized your coding scheme was unclear (report the inter-coder agreement), revised the scheme, and undertook the process again. This is not shameful and does not need to be ‘dressed up’ as anything else; it is just part of the real, messy process of research.

6.1. Stage one—First coder creation of coding scheme

There are no strict ‘rules’ for the creation of the coding scheme, and this may even involve computational assistance such as an LLM in some form. Like scientific theories, ideas can come from anywhere; however, it is a good sign of openness if the first author describes this process thoroughly.

• • If the authors are making claims about coded response frequencies with respect to condition (e.g., more people saying X in a given condition), or a quantitative response (e.g., higher average response among those coded as Y), was coding conducted blind to condition / quantitative response?

6.2. Stage two—Second coder recruitment, communication, and coding

Please read the stage two section for further details, but here are some key indicators to look out for:

• • Have the authors explained who the second coder was (they should ideally be named if not an author on the paper) and how they were recruited?

• • Is there a potential power discrepancy or a shared desire for a particular outcome that could affect independence?

• • Have the authors clearly explained the communication process with the second coder?

• • Have they uploaded the instructions they sent to the second coder?

6.3. Stage three—Disagreement resolution and final codes

Please read the stage three section for further details, but here are some key indicators to look out for:

• • Have the authors described the process for disagreement resolution?

• • Have the authors uploaded a spreadsheet showing the qualitative data for all participants, including initial codes for both coders and final agreed codes?