3.1.1 Identifying information of interest

Prior to drafting the coding manual, authors must consider the overall purpose of their research synthesis. While authors are identifying information of interest, they must also consider their access to resources for conducting the synthesis. For example, an author who is a doctoral student performing a research synthesis for a class project may have access to fewer technological resources (e.g., subscription-based software for coding) and human resources (e.g., availability of coders) than a group of seasoned systematic reviewers who are supported by grant funding to conduct a research synthesis. In addition, the more items a coding manual has, the more hours of training and study coding will be required. Nonetheless, one benefit of gathering information on more items can be a more comprehensive dataset, potentially allowing authors to answer other research questions (including exploratory questions) in additional research syntheses and publications. Considering resources in conjunction with the overall purpose of the project is helpful for establishing the size and scope of the coding manual.

We recommend at a minimum that authors begin by developing a list of items that the coding manual will need to include for authors to (a) describe the included studies, (b) develop tables and figures, (c) code studies for risk of bias assessment, and (d) sufficiently answer the research questions posed in the research synthesis.Reference Li, Higgins, Deeks, Higgins, Thomas and Chandler⁴ Authors may use an established framework—such as the population, intervention, comparator, and outcome (PICO) framework used to determine eligibility criteria for primary studies—to define coding categories that are likely to ensure the gathering of many of the essential details to answer their research question(s), write their final report, and create any necessary tables.Reference Thomas, Kneale, McKenzie, Brennan, Bhaumik, Higgins, Chandler, Cumpston, Li, Page and Welch¹⁷ Authors can directly transfer their eligibility criteria to items, code response options, and code descriptions in their coding manual. Authors can then expand their coding manual by thinking about what other items are needed to comprehensively describe the included studies’ population, interventions, comparators, and outcomes. It is also helpful for review authors to develop a template of the tables they plan on including in the report of the completed research synthesis.Reference Li, Higgins, Deeks, Higgins, Thomas and Chandler⁴ Sometimes, information presented in tables is purely descriptive or not used in analyses (e.g., a summary of the included study population characteristics) and could be missed in the development of a coding manual if the author is only focused on developing items that are specific to answering the research question(s). Authors should also consider what supplemental tables or figures they will present and ensure that their coding manual is aligned with those materials.

3.1.2 Consult guidelines and stakeholders

To develop items for study coding, authors should explore commonly used guidelines for reporting, risk of bias assessment, and methodological quality assessment for their general discipline. For example, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement and Journal Article Reporting Standards (JARS) checklists are resources that authors may use to identify the types of items to include in a coding manual.^8, Reference Page, McKenzie and Bossuyt⁹ PRISMA and JARS provide reporting standards for research syntheses that can be used to better understand the essential components that need to be coded and included in the write-up of a research synthesis. The Cochrane Collaboration also has several resources and open access tools related to risk of bias that authors can use or adapt to align with the types of studies in their synthesis.^18– Reference Higgins, Savovic, Page, Elbers, Sterne, Higgins, Thomas and Chandler²⁰ Authors should also consider using disciplinary-specific guidelines for their field. For instance, education researchers might consider the What Works Clearinghouse guidelines that focus on standards and procedures for assessing the methodological quality and internal validity of studies.⁸

We recommend that review authors engage various stakeholders while drafting the coding manual to determine the types of items they would be interested in learning more about on a particular topic. For example, if the research synthesis is focused on assessing the effectiveness of home-based interventions for young children, we recommend that authors have informal discussions with parents, service providers, staff at parent organizations, or family-centered businesses (e.g., libraries, museums) to identify salient items to include in a coding manual. Having informal discussions can shed light on current trends or challenges in practice that may not be familiar to researchers. In addition, if authors are new to their research area (e.g., graduate students), we suggest engaging additional experts in the field to review the coding manual and provide feedback about the items and codes.

3.1.3 Consider multiple data sources

In many cases, information about the same sample of participants may be reported in two or more different reports (e.g., one dissertation and one peer-reviewed journal article report findings from the same sample; two conference proposals and one peer-reviewed journal article report findings from the same sample). Before coding studies, authors should prespecify what procedures they will use to handle multiple data sources, link and keep track of individual reports presenting findings from the same sample, and plan procedures for extracting data from multiple sources. Ideally, we recommend that authors code information about a study from all available reports on that participant sample, given that different reports from a study sample may provide unique information (e.g., one article may present primary impact analyses needed to estimate effect sizes whereas a separate article may present valuable contextual information about implementation). However, this approach may not be feasible due to project resource constraints, or in scenarios where the authors have strong rationale for prioritizing information reported in some reports over others. For instance, authors may decide to prioritize coding of information from the ‘main report’ of a study’s confirmatory analyses focused on primary outcomes (vs. supplemental reports focused on secondary outcomes, secondary follow-ups, or secondary exploratory analyses from the same sample). Given the range of approaches that might be used to handle multiple data sources, authors should therefore clearly identify which reports contributed information that was included in their research synthesis. In scenarios where studies report findings for overlapping or dependent effect sizes or participant samples, authors will need to track and document these dependencies carefully so that they can be attended to at the analysis stage.Reference Olkin and Gleser^21– Reference Van den Noortgate, López-López, Marín-Martínez and Sánchez-Meca²⁴ The issue of multiple data sources spans the entirety of a research synthesis from the literature search stage to the data sharing stage; therefore, we recommend that authors review specific guidance on this issue as described in Mayo-Wilson et al.Reference Mayo-Wilson, Li, Fusco and Dickersin²⁵

3.2.1 Items should be single-barreled

Items on the coding manual should be single-barreled, meaning that they require a coder to respond to or record one individual element of the study. For example, the item “Was fidelity of implementation measured and reported above 80%?” is not single-barreled because a coder must identify and record responses to two different components of fidelity of implementation, including “Was fidelity of implementation measured?” and “Was fidelity of implementation reported above 80%?” These two items should be separated in the coding manual.

3.2.2 Determine the code response option format for each item

Authors will need to consider the best code response option for each item in the coding manual. Many items on a coding manual will be forced response, meaning that the coder is forced to choose one or more code response options (e.g., select “yes” or “no”; select 1 from a set of nominal codes; select all that apply). With items that are forced response, authors of the coding manual must also anticipate the structure of the code response options.Reference Li, Higgins, Deeks, Higgins, Thomas and Chandler⁴ Other items may allow coders to capture qualitative data in a free response format using text coding (e.g., an item that asks coders to “describe any services provided to the control condition”).

Authors will need to consider the code response option for each item separately because one response type (i.e., select one, select all that apply, open coding) cannot suit all items. For example, consider responding to the item “What mathematics content domain did the academic intervention address?” with the following code response options: early numeracy, whole number computation, word problem solving, fractions, geometry, and algebra. A “select one” code response option may not be an appropriate option for an intervention focused on word problem solving that includes computation of whole and rational numbers. In this case, a “select one” code response option forces coders to decide between two or more options that are all acceptable. This is likely to lead to unreliable responses between coders, as well as an incomplete data set related to this item. In this example, a code response option of “select all that apply” may be a better option.

Authors should also consider the tradeoffs associated with varying levels of detail related to the code responses for each item. For example, it may require more effort up front and during the coding process for coders to capture a specific code response across a set of items (e.g., separate binary yes or no codes for each grade level in school). This level of detail may not be needed for analyses in which authors plan to collapse grade levels across common grade bands (e.g., K-5, 6-8, 9-12) and may require more work at the data cleaning stage. However, if authors conduct detailed coding for items, their data may also be more accessible to future researchers who need or are interested in answering different research questions that require access to more specific code responses.

We recommend that authors also consider the advantages of open text coding for some items. Open text coding is particularly useful for items with too many possible response options to be accurately captured using a forced response list, for example, the study location or intervention characteristics. Open text coding also allows coders to expand on information already documented with forced response items, such as by providing a brief description after a forced response has been recorded (e.g., after selecting the grade levels included in the school, describe any other factors relevant to the school’s context), and attend to items that are included in the coding manual for descriptive purposes only. Additionally, open text coding can capture relevant information that may not have been considered at the time the coding manual was developed. For example, it can be difficult to know or anticipate all the outcome measures that may be used in a corpus of academic or behavioral intervention studies while developing the coding manual. Including an open text response option to briefly describe the outcomes measures that were noted as “other” in the forced response might be necessary. Text coding can also assign “summative, salient, or essence-capturing” information, which is not possible with forced response coding.Reference Saldaña²⁶ This type of outcome may be particularly important to different stakeholders, who, for example, want to know about the context within which an intervention took place or how an outcome was measured.Reference Sharma, Gordon, Dharamsi and Gibbs²⁷ Indeed, open text coding may be particularly valuable in qualitative or mixed-methods research syntheses that aim to provide rich contextual information about a study’s implementation barriers and facilitators.

Although useful and often necessary, open text coding may pose some challenges, such as ensuring data can be enumerated for meta-analytic or other quantitative summaries. Further, reviewing the results of open text coding requires additional time to first assign codes, then categorize the coded data, and eventually identify themes. Establishing reliability for open text coding may also pose challenges for researchers, as it requires more exploratory and iterative coding behavior from coders, and after categories have been determined, coders often need to recode studies to validate the coding scheme.Reference Villiger, Schweiger and Baldauf⁷ Thus, our recommendation is to minimize the amount of open text coding for analytic purposes and to reserve this type of coding for items that are exploratory or descriptive or considered high priority for stakeholders.

3.2.3 Ensure code response options are mutually exclusive and mutually comprehensive

When items in the coding manual are forced response in which coders are required to “select one option,” authors should ensure that response options do not have any overlap. For example, if coders are responding to the item “How many times per week was the intervention implemented?” the following code response options have overlap: (a) once per week; (b) 2–3 times per week; (c) 3–5 times per week. In this example, there is overlap if the study reports that there were exactly 3 sessions per week (i.e., coders could select either option b or c and both would be correct). In this case, authors might consider developing code options without a range; 1 = one time per week; 2 = two times per week; 3 = three times per week; and so on. And of course, for any items planned for inclusion in quantitative analyses (e.g., meta-analyses), higher levels of measurement (i.e., interval or ratio measurement vs. nominal or ordinal) are always desirable, when possible.

Authors should also ensure that coding manuals with forced response options provide all possible responses. For example, studies may report the duration of an intervention as a range. In the example above where coders must “select one” a study with an intervention range reported as 2–3 times per week is not represented in the code response options. Therefore, we recommend that in many cases, authors also allow for a code of “other” to be selected. When coders select a code of “other” for any item, they should be required to add a qualitative explanation about why they used the code of “other” instead of the provided code response options.

3.2.4 Avoid ambiguous code response options

Authors should avoid creating items with code response options that are descriptively ambiguous or not quantified. For example, code response options such as “none,” “some but not all,” “almost all,” and “all” are not quantified for coders. Two coders with the same training may interpret the use of “some but not all” and “almost all” in different ways that lead to an unreliable description of the results of the item. When possible, provide exact operational definitions for each of the code response options, or add quantifiers (e.g., a percentage that aligns to each code response option). When it is not possible to add quantifiers to code response options that are ambiguous, authors should consider adding a coder “inference level” item that indicates how confident (or not) they were when coding that item. See Wilson et al.Reference Wilson, Lipsey, Tanner-Smith, Huang and Steinka-Fry²⁸ for an example.

3.2.5 Establish the difference between “no,” “not applicable,” and “not reported”

Our final recommendation for authors related to developing items on a coding manual is to establish clear guidance for coders on the difference between code response options of “no,” “not applicable (NA),” and “not reported (NR).” NR is important to include, especially when one of the code response options is also “No.” For example, consider the item, “Did the authors monitor the fidelity of the implementation delivery?” Providing coders with the code response options of “Yes” and “No” is insufficient. A study that does not report whether it monitored implementation fidelity should not automatically be recorded as “No,” because fidelity monitoring may have occurred but simply was not reported in the final manuscript. A code response option of NR, in this case, allows authors to determine if they need to contact authors about missing data that may be essential for the results of the research synthesis. In contrast, more straightforward items such as “Did the study include children in preschool?” are often appropriate for a binary “yes” or “no” code response options. In this case, if the authors report that all students were in kindergarten and first grade, a coder can reasonably assume that preschool children were not included in this study and indicate “no.” Authors of coding manuals must carefully consider the distinction between missing information and information that can be inferred.

Similarly, authors should consider how “NA” code response options represent information that is conditional or may not apply to a particular study. For example, a common socioeconomic status (SES) measure in education research syntheses, “What percentage of students were reported as being eligible for free or reduced-price lunch (FRL)?” is only applicable to studies conducted in the United States. Other countries use different indexes of SES to describe participants and FRL may not be relevant. Recording a code of “NR” for studies conducted outside of the United States for this type of item is not accurate. It is important that the coding manual clearly states when codes of “NA” should be used and provides clear instructions to the coder.

3.3.1 Organize items according to how information is presented in studies

We recommend that authors organize the coding manual and coding form in a manner that allows for efficient study coding by grouping items that are similar or likely to appear in the same part of the study together. For example, primary study authors typically report all participant demographic variables together, either in text or in a table; these items should therefore be presented together in the coding manual and coding form. If demographic items are spread throughout the coding form, it becomes time consuming for a coder to repeatedly leave and return to the same sections of a study multiple times to record information in the coding form. Likewise, when coding effect size data, means and standard deviations are often presented together so it is ideal to have these two statistics presented side-by-side in the coding form versus in separate sections. This strategy can also reduce errors in study coding.

3.3.2 Consider the relational structure of the data

Authors will also have to consider the organization of the coding manual and coding form in terms of the structure of the data. Consider a research synthesis focused on social-emotional learning interventions. Some coding items may be related to student participants (e.g., sample size, demographics) and responses to these items may be reported in studies at the group level (e.g., intervention group, comparison group). Whereas other items, such as intervention features or study design, may be presented at the study level. Further, other items, such as characteristics of the outcome measures, may be coded at the level of the outcome measure or even at the effect size level. We recommend that authors separate areas of the coding form according to the different levels that information may be presented (e.g., study, participants, conditions, outcome measures, effect sizes). Separating the coding form in this manner may be more efficient for coders and lead to fewer coding errors; however, each part of the coding form must include the study’s unique identifier (and additional unique identifiers for each level of coding items in the relational data structure) to permit efficient merging of data during the analysis stage. Designing the coding form in this way will facilitate the creation of relational databases that are multifunctional and machine-readable.Reference Haddaway, Gray and Grainger²⁹

3.4.1 Compare different software

We recommend that authors compare different software based on their suitability to the research question and coding manual. For example, authors should take into account the types of data planned for collection and which stages of the review will be conducted with the software. Authors should look for specific features such as an easy-to-use interface for coders and project managers; collaboration features, such as assigning multiple coders; and customizable features, such as customized coding forms and data levels. Finally, authors should examine how the software organizes, presents, and exports data for use.Reference Zhang and Neitzel³² Features like availability of product support may be important to consider based on the team’s familiarity and comfort level with the software.

3.4.2 Select and program software

After reviewing different software, authors should decide on the software that fits most of their needs and begin programming their coding form into the software following the coding manual. When programming the coding form, make sure all item names and code response options are consistent with the coding manual. Adding some descriptive information from the coding manual to the coding form can help keep coders consistent and increase efficiency in the coding process. For example, authors might include an eligibility criteria table for easy reference. Maximizing the use of conditional items (i.e., coding fields that are only applicable and shown to coders when a specific criterial is met) can also greatly increase coding efficiency and reduce coder fatigue. Depending on the software, it may also be possible to program automated data collection fields that populate based on previous responses or create filters that control the flow of references through the project. It can be extremely helpful to review any user guides or product support information when programming to see what options exist for increasing efficiency in the coding process.

3.5.1 Pilot, discuss, and revise the coding manual and coding form

We recommend that authors pilot the coding manual and coding form with at least two studies, and when possible, select studies that report different types or amounts of information (e.g., select one peer-reviewed journal article and one dissertation; select one older study and one newer study that also provides supplemental files). When piloting, members of the review team should use studies that have been identified for inclusion in the current research synthesis or excluded studies that are closely aligned with the focus of the research synthesis. Piloting the tools for study coding allows the authors to identify any challenges with the coding manual or coding form. For example, during the pilot phase, coders may notice missing code response options or identify items needing additional details in the code descriptions. They may also identify missing items altogether and suggest additions to the coding manual. Coders may also identify challenges with the coding form, such as misalignment between the coding manual and coding form. We suggest that all members of the team (i.e., author(s) of the coding manual and coding form, and coders) meet to discuss potential edits and revisions. Meeting allows the authors of the coding manual and form to ask questions for clarification and gain additional understanding of any challenges with the tools. As final decisions are made, we recommend documenting changes to the coding manual and coding form and clearly communicating all changes to the research team (e.g., sharing a copy of the coding manual with final edits highlighted using track changes). After the meeting, the authors should review all comments and salient points from the discussion with the coders to make final decisions about the items, code response options, code descriptions, and coding form. Finally, we recommend that authors minimize changes to the coding manual after study coding begins.

3.5.2 Identify needs for training

This stage of developing the coding manual and coding form also supports the primary author in identifying aspects of study coding that need additional emphasis or practice during training. Authors can determine which aspects of the coding manual and coding form were most challenging for the coders and can share related resources with coders before training. At a minimum, we suggest that the coder training be led by one or more content area experts according to the review topic, and that training includes: providing coders with background knowledge about the content area of the review, providing a review of the coding manual and coding form, and allowing for coding practice. Similarly, to prevent misapplication, training in coding items related to risk of bias or study quality assessment should be provided by trained methodologists with experience in the tool.Reference Igelström, Campbell, Craig and Katikireddi³³ Training should also require that coders meet a minimum threshold of reliability with the coding manual and form. For more detailed procedures on training coders, we recommend reviewing Giesen and Roeser’s practices for establishing a team-based approach to coding data and Cooper et al.’s handbook (specifically Section 9.6 on training of codersReference Wilson, Cooper, Hedges and Valentine³⁴).Reference Cooper, Hedges and Valentine^6, Reference Giesen and Roeser³⁵

3.5.3 Prospectively sharing the finalized coding manual and coding form

Just as with other forms of research, an important step at this phase in a research synthesis project is to register the study and prospectively share the protocol and analysis plan.Reference Hardwicke and Wagenmakers^36, Reference Kerr³⁷ The version of the coding manual and coding forms finalized after piloting should be among the materials prospectively shared.Reference Moher, Shamseer and Clarke^38, Reference Shamseer, Moher and Clarke³⁹ Some research synthesis publishers (e.g., Campbell Systematic Reviews and Cochrane Systematic Reviews) require a prospectively registered protocol be published prior to the publication of the full research synthesis. However, researchers who are not publishing their completed reviews in these outlets should register their review elsewhere, such as PROSPERO or the Open Science Framework (OSF).^{40– 47} Given the multitude of options for where to register their study materials, the choice may be driven by disciplinary norms and preferences of the review authors. It may be difficult to anticipate all of the scenarios across a corpus of studies that may impact the development of the coding manual and coding form. To balance this challenge with prospective registration, we recommend that authors describe any changes made to the protocol after registration (e.g., in a “differences between protocol and review” section of the final report). The information provided to amendments to the coding manual should include the specific change, the reason for the change or revision, and the stage at which the change was made.Reference Page, McKenzie and Bossuyt⁹

3.6.1 Include at least two coders

During study coding, ideally at least two independent coders should be trained and responsible for study coding.⁴⁸ Larger projects may require more coders to avoid assigning any one coder too many studies; however, more coders may also lead to delays in all coders establishing reliability with the coding manual and coding form. Smaller projects (e.g., a doctoral thesis) should at minimum include one primary coder and, when possible, one secondary coder for the purpose of assessing the reliability of the coding process. While coding, we recommend that coders read the entire study and all supplemental materials associated with it; if a study presents findings from a given sample in multiple reports, we again recommend coders read all reports associated with that study. This ensures that the coders do not miss any information, including information presented in supplemental reports, or information buried deep in appendices or supplemental materials.

3.6.2 Randomly assign studies to coders

Ideally, studies should be randomly assigned among coders, with the team leader making manual adjustments as necessary to handle any potential conflicts of interest. For example, if any of the coders are authors of the included studies, the team leader should adjust the random assignments to ensure that no coder receives their own records to code. The same suggestion applies to articles in which a coder may be a former student or collaborator with the first author of an included study. This process removes any potential bias of coders coding information that is known only to them, and was not published in the report, because they are an author of an included study or a close collaborator to the first author of an included study. For large teams, the number of studies assigned to coders should be balanced such that all combinations of coder pairs are as equal as possible. Alternatively, coders may be assigned parts of the coding manual to code studies based on their areas of expertise.Reference Cooper, Hedges and Valentine⁶ This process could be particularly beneficial for coding items that are highly technical or require in-depth content knowledge. For example, graduate students who have expertise in social–emotional interventions may have content knowledge to support most study coding for items related to basic study details, the participants, and intervention features. However, some graduate students may not yet have training to understand the different ways a study will report effect size information or have sufficient knowledge about different methodological issues that need to be critically appraised as part of risk of bias assessments. With highly technical items, authors can separate the coding manual into sections according to areas of expertise and then randomly allocate coding assignments within each section.

3.7.1 Set coding deadlines

Rather than waiting to receive all study codes by one final deadline, we recommend the team leader create a schedule of due dates that requires coders to turn in codes for specific studies throughout the coding process. Depending on the size of the review, the team leader may collect codes for studies weekly or bi-weekly, or after a certain number of studies are coded, for example, every five studies. This requires that coders begin coding studies as close to the training as possible, rather than waiting several weeks, or even months, to begin coding. This will likely result in greater adherence to the coder training, coding procedures, and ultimately coding reliability. Setting coding deadlines also allows a team member to calculate reliability of the coding throughout the coding process (and provide additional training or corrections needed to maintain high reliability), rather than wait until the end of coding to calculate reliability (see Section 3.8.1). Moreover, setting intermediate deadlines can help maintain momentum for the overall task.Reference Giesen and Roeser³⁵ Setting intermediate deadlines also allows the team leader to monitor questions from coders, identify and address misconceptions, and regularly communicate with the team.

3.7.2 Establish regular communication

Regardless of the scope or size of the synthesis, the team leader should carefully manage the study coding process with regular and consistent communication.Reference Giesen and Roeser³⁵ Even with a well-thought-out coding manual and coding form, there are likely to be unexpected challenges that arise during study coding. Coders may ask the team leader for clarification on different items or response options as these challenges arise. The team leader can then communicate with the team to ensure that the clarifying information is applied consistently by coders. We recommend the research team meet at least weekly or bi-weekly to update the team on coding progress, ask clarifying questions, and discuss challenging items. This, of course, must be carefully balanced with the requirement that all coding is done independently by coders. Although changes to the coding manual should be minimal after independent coding begins, any changes to the coding manual (e.g., additional detail or examples to support coders) should be discussed during these meetings and documented.

3.7.3 Optimize the coding form

To improve the efficiency of study coding and coder engagement, team leaders should continually refine the coding form based on feedback from coders. While coding all studies, coders may encounter unexpected study-specific nuances that require special consideration or recognize patterns of reporting that are not reflected in how the original coding form was programmed. Coders should also be encouraged to provide feedback about the optimal presentation of information, such as how and where to place coding fields and whether additional definitions or examples would be helpful to include within the coding form. Team leaders can then incorporate coder feedback into the coding forms to make them more user-friendly, reduce the likelihood of errors and coder fatigue, and expedite the study coding process.

3.7.4 Assess data quality

During the coding phase, team leaders can perform data quality checks to ensure the integrity and accuracy of the coded data and increase the efficiency of data analysis. Assessing data quality can also reveal unanticipated issues with how coders interpret and use the coding manual or coding form. The team leader should review the coded data to verify all fields, code response options, and missing data are populating correctly. This can be done by exporting the data to ensure that missing data codes show up appropriately and as distinct from skipped fields. Additionally, the team leader should confirm that the item names and code response options in the coding form and the data export are consistent with the coding manual. Last, by exporting the data, the team leader can verify that the data from all levels (e.g., study level, group level, effect size level) can be merged efficiently for meta-analytic data analysis.

3.8.1 Identify studies for double-coding

Views differ in the research synthesis field on the number or percentage of studies that must be double-coded. Although double-coding of all studies is ideal, research synthesis methodologists appreciate that some reviews may not have the resources to do so (particularly for large reviews), or that some coding manuals may be sufficiently simple to allow single-coding after a piloting phase that establishes high inter-rater reliability.Reference Shea, Reeves and Wells^49– Reference Higgins, Lasserson, Thomas, Flemyng and Churchill⁵¹ One common and acceptable approach to completing double-coding is for teams to examine reliability of the study coding process at the beginning of coding (such as while piloting the measure or after the initial coder training), and once all coding is completed, calculate reliability, and then reconcile disagreements.Reference Villiger, Schweiger and Baldauf⁷ However, there may be instances when research teams want to consider a different approach to examining reliability and reconciling disagreements, such as with very large reviews or when several coders on the research team are new to study coding. In these instances, we recommend that research teams examine reliability during the coding process, such as bi-weekly. This approach allows reliability to be calculated and for disagreements to be reconciled in a manner that may lead to greater reliability of coder behavior throughout the study coding process. For example, through this process, coders may determine that an item with frequent disagreement is the result of a coder misunderstanding information in the coding manual. Instead of allowing a coder to code all studies with this misunderstanding, the team member leading study coding can re-train the coder on specific items. This approach requires more organization at the front end of a review to ensure that the studies for double-coding are identified early and that those studies are being coded at approximately the same time by both coders. In cases where it is not feasible to double-code all studies, we recommend that authors consider double-coding a random subset of essential items. For example, authors could double-code all studies for information required to calculate effect sizes as well as items related to pre-specified moderator variables of interest. Regardless of approach, when only a subset of studies is double-coded, these studies should be randomly identified, assigned to a second coder, and coded independently in duplicate.

3.8.2 Discuss disagreements and determine the final code

For disagreements that appear to be an error by one coder, coders can likely resolve the disagreement easily with a brief conversation between the two coders. We recommend that coders assigned to the same study are provided with a list of disagreements prepared by another author (e.g., the lead coder, first author) and shared prior to the meeting. This gives coders time to review the items with discrepancies and their own original codes. Coders can then determine if the coding discrepancy was due to an error, such as accidentally selecting the wrong code, or if they missed information in the study report. After coders have had a chance to review the discrepancies, then they should meet to discuss the disagreements, and one coder should be the record keeper of the final coding decision that is returned to the lead coder. For disagreements that cannot be easily resolved, we recommend that a third team member participate in a discussion about the disagreement to determine the final code.Reference Li, Higgins, Deeks, Higgins, Thomas and Chandler⁴ We also recommend that team leaders be mindful of any power differentials among team members that could cause one coder to be overly submissive/assertive during coding disagreement discussions (e.g., one coder is a graduate student and another is a postdoctoral scientist). Ideally, during team trainings and meetings, the team leader should explicitly discuss and actively model a team culture that values and rewards healthy discussion and debate.

3.8.3 Contact study authors for missing data

After determining the final code, coders should determine if there is any essential information missing from the published report. For example, missing information to calculate effect sizes (e.g., sample size) or specific moderator variables of interest. If there is missing essential information, coders could contact study authors to request this information. When contacting authors, we recommend the coder minimize burden of this request by including a list of missing information for the study author to respond with or a template for them to fill out. We also recommend including the specific report with missing information attached to the request to both increase the chances that the author will reply and decrease the chance the author sends information for the wrong published report. Finally, we recommend providing study authors with a clear deadline for their response, providing a reasonably appropriate timeframe to respond (i.e., at least 2 weeks). See Pigott and PolaninReference Pigott and Polanin² for more information about and options for handling missing data.

3.9.1 Ensure accessibility of coding manual and coding form

Prior to sharing, authors should review the materials and make them as user-friendly as possible. For example, the coding manual should be shared in a standard, accessible format such as a PDF and arranged clearly with section and question headers. Providing comprehensive metadata on the coding form (e.g., item names and question descriptions) in a concise format, such as Excel, makes it easier for the user to understand the data collected on the coding form. Authors should also consider creating a detailed README file to increase the accessibility of materials. This file should explain the purpose of the review, provide a clear explanation of each material file and brief description of what information the file provides, and give step-by-step instructions to navigate through the review process and corresponding files. Consistent question headers and item names should be used throughout all materials.

3.9.2 Organize and share materials (metadata, persistent identifiers, data structure)

After the research synthesis materials are prepared for sharing, the author must select an appropriate open access platform to host their materials. Repositories like the OSF that assign persistent identifiers, such as DOIs, allow for long-term availability and provide a consistent citation over time. Domain-specific or disciplinary-specific repositories can also increase the visibility of the materials to relevant researchers. Once a repository is selected, the accessible review materials should be clearly organized within the repository (e.g., grouping related documents, clear file names and structure). The README file should be the first document the user sees when accessing the repository; it should include all information needed for a user to find and understand what is in each file. All project files (including but not limited to the coding manual and coding form) should be included in the repository and corresponding README file.