Conceive the project idea and proposal

The research questions addressed and research plan proposed by harmonization initiatives need to be Feasible, Interesting, Novel, Ethical, and Relevant (FINER). ^{Reference Cummings, Browner, Hulley, Hulley, Cummings, Browner, Grady and Newman20} As in any research project, this involves defining elements including the objectives to be pursued and research questions addressed; the specific exposures and outcomes required to answer the research questions; and the suitable population size and characteristics (e.g., mothers, children, age range, area of residence, primipara). In addition, the logistical, operational, technical, methodological, ethical, and legal elements specific to harmonization and co-analysis of IPD across independent cohort studies generally need to be outlined. Relevant elements to be considered depend on the objectives and scale of the initiative but can comprise defining the proposed governance model; the criteria used to select participating studies; the data infrastructure to be implemented; the operational and methodological approach to harmonization; and the statistical methods foreseen to validate and analyze harmonized data. Ideally, the protocol should also include a first evaluation of the harmonization potential across participating studies to outline the true potential of the project to answer the research questions addressed. Specialized catalogs documenting the design and content of mother-and-child studies are available to the research community to facilitate such evaluation. ^{Reference Jaddoe, Felix and Andersen8,Reference Zeitlin, Sentenac and Morgan10,Reference Bergeron, Massicotte and Atkinson11,Reference Larsen, Kamper-Jørgensen and Adamson21}

Apply for funding

Harmonization initiatives can require significant investment in time and expertise, both from the participating cohort studies and from the team coordinating the project, and the budget should reflect this reality. Costs relate to many factors including the scope of the initiative, the complexity of the governance and data infrastructure, the number of cohort studies involved, the quality of study-specific data and metadata (information about data), and the number and type of harmonized variables to be generated across studies. Based on the survey results, relatively small-scale initiatives (e.g., aiming to harmonize 10–15 variables across four studies) can require 2–6 person-months (number of months, for the equivalent of one person working full time) to generate a validated harmonized data set, while large-scale initiatives (e.g., aiming to harmonize 150 variables across more than 10 studies) from 15 to over 80 person-months. Generally, most of the staff’s time resources are dedicated to data inventory, cleaning, management, and processing.

For many initiatives, the time and costs required to obtain data should also be considered. Depending on the context, data access procedures (from submitting a demand for access to being ready to initiate harmonization) can take from a week to more than a year per study. In addition, study-specific data access fees might be applicable and may easily exceed 2,000€. Implementing a complex data infrastructure (e.g., distributed across several studies) could also be required by large initiatives. Setting up such an infrastructure demands time from technical experts (e.g., in data management and security) and can take several months. An overview of the timeline and costs of the PAE project is provided in Supplementary material S2. Finally, as in individual cohort studies, large-scale harmonization initiatives might also need long-term funding to ensure a sustainable platform and the maintenance and management of access to the harmonized data sets generated.

Initiate activities and organize the operational framework

The success of large long-term harmonization initiatives often depends on building a collaborative, interdisciplinary team of experts and implementing flexible but efficient operational and governance models. Large initiatives bring together data users (investigators requesting harmonized data to achieve their research goals), data producers (stakeholders from participating cohort studies), and experts from specialized domains (e.g., longitudinal data analysts, ethicists, computer scientists, epidemiologists, clinicians). Team members generally come from different research groups, and each member brings its own professional background and level of expertise. To optimize projects operations and research outcomes, efforts might thus be required to build a unified approach and common understanding of various concepts.

Building consensus is not always necessary (e.g., in a small initiative with a narrow research question). However, to ensure efficient launch of activities, the team needs to rapidly delineate the practical requirements and operational details related to the research agenda, the data infrastructure to be implemented, and the data harmonization and analysis framework. Table 1 provides various examples of questions that could be addressed by the team.

Table 1. Examples of questions that could be addressed to help delineate analytical approach, practical requirements, and operations of a harmonization initiative

Assemble information on studies

It is generally important to gather precise information about the characteristics of the studies actually enrolled so as to ensure the quality of the harmonized data set. Data comparability is affected by heterogeneity of the study-specific populations and data content. Access to comprehensive information on study designs, population characteristics, data collected, duration and timing of data collection events, and the standard operating procedures used can be required to confirm the eligibility of studies and estimate harmonization potential. Study-specific inclusion criteria are different for each research question addressed, but could include study-specific design (e.g., cohort studies), number of participants (e.g., at least 500 mothers recruited at baseline), sampling/recruitment frame (e.g., representative sample of pregnant women in a geographic area), years of recruitment (e.g., mothers recruited after 2010), number and frequency of data collection events (e.g., at least two data collection events during pregnancy), data/samples collected (e.g., smoking status, cord blood), specific time of collection (e.g., fasting glucose collected before 12 weeks of pregnancy), and potential to access IPD (e.g., IPD can be transferred to a central repository; or IPD can be analyzed remotely but cannot physically be shared/transferred or copied). Harmonization initiatives generally select studies before initiating the project. However, large-scale ones can address a broad range of research questions, each requiring inclusion of different subsets of studies presenting specific characteristics.

Define variables to be generated and evaluate harmonization potential

A DataSchema, or list of core variables (e.g., outcomes, risk factors) to be generated using study-specific data items, generally needs to be outlined. Selecting and defining these variables is probably the most scientifically challenging step of the harmonization process. It can require participation of researchers with specific domain expertise (e.g., nutrition, mental health), investigators or data managers from member studies, and personnel with technical expertise in data harmonization. The information collected across cohort studies is generally not standardized, the wording of questions and measures used to evaluate the same constructs (e.g., level of physical activity, alcohol consumption) typically differ, and there is variation in the format, structure, and naming conventions of variables. In addition, the research questions addressed often require the analysis of longitudinal data (e.g., several data collection events during pregnancy or throughout the life of the child), but the collection events are likely to differ between and within studies in keyways that affect compatibility. As an example, Table 2 outlines information about mothers’ binge drinking during pregnancy collected by five Canadian cohorts. Various DataSchema variables could be created using the data collected by these studies. These include, but are not limited to, a unique ‘’binge drinking status during pregnancy’’ variable defined as binge drinking at least once during pregnancy (yes/no) or a ‘‘current binge drinking status’’ (yes/no) variable paired with the ‘'time when binge drinking status was collected’’ (number of weeks of pregnancy). While there is rarely a unique or perfect solution, it is important to implement a rigorous and transparent decision-making process to select and define the DataSchema variables. The process should be guided by the scientific needs of the project, including specific requirements related to the statistical analysis planned.

Table 2. Example of information about frequency of binge drinking during pregnancy collected by five mother-and-child cohorts

3D: 3D Study – Design, Develop, Discover; AOF: All Our Families; APrON: Alberta Pregnancy Outcomes and Nutrition; FAMILY: Family Atherosclerosis Monitoring in Early Life; OBS: Ontario Birth Study.

Various elements can be used to define the DataSchema variables. These include the: nature of the variable (e.g., smoking status, highest completed level of education); value type (e.g., integer, text, decimal); format, including the specific units (e.g., kg) or list and description of the response options (e.g., 0 = Never; 1 = Almost never; 2 = Sometimes; 3 = Often; 4 = Very often); targeted individual or entity (e.g., the information is about the mother, father, neighborhood); targeted time period (e.g., first trimester, last 30 days, at birth); interdependence with other information needed to interpret the variable (e.g., birthweight and duration of pregnancy); acceptable sources of information (e.g., information obtained from questionnaire, registry, medical files); acceptable informants or who can provide the information (e.g., participant or proxy); acceptable time of collection (e.g., smoking status during first trimester of pregnancy can be collected at birth); acceptable question wording (e.g., binge drinking defined as 5 drinks or more); and acceptable procedures or devices used to generate the measure (e.g., weight needs to be measured, not self-reported by the participant).

Following the selection and definition of each DataSchema variable, it is possible to evaluate the potential (or not) for each study to generate it. According to the Maelstrom Research guidelines for retrospective data harmonization, ^{Reference Fortier, Raina and Van den Heuvel4} the harmonization potential is considered complete (fully achievable) if study-specific variables can directly generate the DataSchema variables (identical) or could be transformed to do so (compatible). The harmonization potential is however deemed impossible if study-specific variables cannot generate the DataSchema variables that have been defined (incompatible) or if the information is simply not collected (unavailable). It is also possible to define the harmonization potential as partial when it is possible to generate the variable but with an unavoidable loss of information. Evaluating the harmonization potential will often lead to adjustments in the initial DataSchema variable definition proposed (e.g., response options for binge drinking categories are adjusted to allow harmonization of more studies). Once finalized, the process will provide a clear overview of the harmonization potential across studies (which variables can be generated by which studies) and the study-specific data required to generate the DataSchema variables. Such processing and documentation can be generated using simple tools (e.g., Excel) or specialized resources.

Develop the data processing infrastructure

In parallel to documenting cohort studies and exploring harmonization potential, it is essential to determine the operating model and build the infrastructure required to host, manage, and analyze the data. For small-scale initiatives, the operating model may be simple and the data infrastructure rudimentary. For example, it could be limited to study-specific data sets uploaded on a server accessible by a single user who generates the DataSchema variables to answer a specific research question and never shares or reuses the harmonized data. However, a more sophisticated approach is often required.

Define the data harmonization and analysis operating models

While various ethical, legal, methodological, and operating factors must be considered, data access and location are fundamental to inform the operating models to be implemented to harmonize and analyze data (Fig. 2). If transfer of study-specific IPD to external third parties is acceptable, data may be transferred to a central server and the harmonization process centralized. ^{Reference Fortier, Dragieva, Saliba, Craig and Robson22,Reference Wey, Doiron and Wissa23} But this is not always possible and may be unsuitable. Alternatively, study-specific data may remain on study-specific servers, and harmonized data generated by study-specific teams. ^{Reference de Moira, Haakma and Strandberg-Larsen24} Each approach presents advantages and challenges (Table 3) and directly impacts operational decisions (e.g., number of servers required, distribution of personnel) and the data infrastructure required (e.g., type of access required, level of security, and computing capacities).

Fig. 2. Influence of the level of access to individual participant data on the data harmonization workflow.

Table 3. Advantages and challenges related to processing collected data under the harmonized format centrally and by study-specific teams

In turn, the possible operating models for statistical analysis are informed by the level of access to the harmonized IPD. Study-specific (analysis performed by studies followed by a meta-analysis of study-level estimates), pooled (data hosted on a central server and analyzed as a collective whole), or federated (centralized analysis, but the individual-level participant data remain on local servers) IPD analysis can be achieved (Fig. 2). Again, each approach presents advantages and challenges. ^{Reference Blettner, Sauerbrei, Schlehofer, Scheuchenpflug and Friedenreich25–Reference Carter, Francis and Carter27} Study-specific IPD analyses followed by a meta-analysis of aggregate data (i.e., two-step IPD meta-analysis ^{Reference Riley, Lambert and Abo-Zaid28} ) is often the approach selected. ^{Reference Taylor, Elhakeem and Nader29} The approach may reduce efforts to obtain and analyze data, as only aggregate data are required for combined analysis and as meta-analytical methods for aggregate data are well established. However, standardizing analyses among studies may require substantial effort, and statistical power and flexibility to explore interactive or heterogeneous effects (for example, across studies or subgroups) can be limited. In contrast, a pooled analysis approach (i.e., one-step IPD meta-analysis ^{Reference Riley, Lambert and Abo-Zaid28} ) typically offers statistical power and flexibility, with the potential for greater insights into interactive or heterogeneous effects and interpretation of results (such as of pooled estimates). ^{Reference Voerman, Santos and Inskip14,Reference Benet, Albang and Pinart30} However, it may necessitate high-performance processing environments to allow analysis of large amounts of data, and it often comes with substantial efforts to obtain access to IPD. The trade-offs between these first two approaches and strategies for choosing an approach have been discussed in detail elsewhere. ^{Reference Riley, Lambert and Abo-Zaid28,Reference Stewart, Altman, Askie, Duley, Simmonds and Stewart31} Finally, federated data analysis can represent a valid option. ^{Reference Gaye, Marcon and Isaeva26,Reference Doiron, Burton and Marcon32} The approach may support one- and two-stage meta-analyses, but it requires implementation of a distributed and interoperable data infrastructure supporting unified co-analysis of the harmonized data across studies. Additional information on these approaches is provided in Supplementary material S3.

Explore study-specific data

Once access is granted, data are generally prepared (preprocessing under a defined format might be required) and explored to ensure quality and deepen proper understanding of each study-specific data set. For example, the completeness, content, and format of the study-specific data can be verified. Issues observed at this stage often lead to adjustment of the harmonization potential estimated. Poor data quality may also lead to the exclusion of a data set.

If data is processed under the DataSchema format by the study teams, this step may be facilitated and limited to extracting the required study-specific data. However, if harmonization is achieved by a central team, the study-specific data must be rendered accessible to the team and generally explored with close communication with study teams. Ensuring consistent quality and validation procedures across study-specific data sets can be challenging and must be adapted to each project. Different standard operating procedures, methodological approaches, and tools have been proposed by the research community ^{Reference Cai and Zhu35–Reference Schmidt, Colvin, Hohlfeld and Leon37} to support quality assessment of study-specific data. An example of minimal procedures that can be used is outlined in Supplementary material S4.

Process study-specific data under the harmonized format

To enable analysis, it is necessary to convert the heterogeneous study-specific data items into the DataSchema variables format. Where appropriate (when harmonization is deemed possible), data processing is accomplished through algorithmic recoding or statistical modeling of study-specific data. The approach selected for each variable will depend on the scientific objectives of the project, the nature and format of the DataSchema variable, the study-specific data items available, the potential to access study-specific IPD, and whether the data processing is achieved centrally or by study-specific teams. Supplementary material S5 provides an overview of possible approaches (algorithms and statistical models) and considerations in their use. Figure 3 illustrates possible algorithmic processing applied to generate a variable on binge drinking status using the variables outlined in Table 2.

Fig. 3. Example of harmonization potentials and algorithms used to generate the variable “binge drinking during pregnancy”. Variable definition: Label: Binge drinking during pregnancy (yes/no); Definition: Indicator of whether the mother ever binge drank at least once during pregnancy; Value type: Integer; Format: 0 = No, 1 = Yes; Targeted individual: Mother; Targeted time period: Throughout pregnancy; Acceptable time of collection: Can be collected in second or third trimester; Acceptable question wording: Binge drinking defined as five or more drinks or four or more drinks on one occasion or in one day. See Table 2 for information on the study-specific variables collected.

Establishing an efficient processing and quality assurance workflow and ensuring accuracy and consistency in decision making is challenging, especially for large-scale initiatives or if harmonization is achieved by different teams. Processing should be guided by the DataSchema variable definitions, and decision making (e.g., treatment of missing values) and quality assurance should be consistent across all data sets.

Estimate quality of the harmonized data

Once the harmonization process is finalized, it is often essential to explore the data sets generated to understand variable quality. This can include generating basic quality control checks (e.g., validating processing algorithms) and descriptive statistics (e.g., participant distributions, proportion of missing values) to evaluate the consistency across cohort studies (Supplementary material S6). When relevant, assessments of heterogeneity can be performed (e.g., testing for a statistical effect of study-specific question formats on the harmonized data generated). However, in practice it can be difficult to distinguish heterogeneity due to harmonization assumptions as opposed to population differences. A more comprehensive examination of relevant heterogeneity (and how to account for it) thus often needs to be performed at the stage of analysis.

Preserve the harmonized data sets and related documentation

Once validated and deemed of acceptable quality, the harmonized data set and its related documentation can be made available, and this, ideally in adherence to the FAIR⁷ principles. Complying with the FAIR principles involves making data and metadata accessible to the scientific community, enabling long term access, ensuring their interoperability, and providing sufficient information to enable optimal use and reuse of the harmonized data. Documentation provided could include the harmonization protocol, selected information about cohort study designs and standard operating procedures, the DataSchema variable definitions, the harmonization potential across studies, the processing scripts or statistical models applied to generate harmonized data, and summary statistics on participant distributions or missing values. For large initiatives, creating a centralized metadata portal can provide user-friendly access to such information. However, maintenance of such portal, as well as long term preservation of the harmonized data sets in one or multiple secured data warehouses, can be challenging (e.g., to retain competent staff, maintain and when required scale up the infrastructure, etc.).

Analyze data to answer specific research questions

Using harmonized data often involves working with an infrastructure where data are not available across all studies (missing values when harmonization is considered impossible), co-analyzing data available at different time points across studies, managing the heterogeneity of effects across studies, and using data that are not as precise as the study-specific data collected. Effectively, harmonizing heterogenous data often results in data reduction (e.g., transforming continuous variables into dichotomous) and subsequently to a potential lack of precision and reduction of power leading to underestimation of effects. ^{Reference Cohen38}

Substantial time can be necessary to explore the harmonized data and conduct preliminary analysis. ^{Reference Avraam, Wilson and Burton39,Reference Raab, Nowok and Dibben40} It might be required to explore the impact of the harmonization potential of each DataSchema variable on the reduction in sample size and/or the diversity of variables included in the analysis. Harmonized data sets might have complex or limiting patterns of missing data that need to be examined; for example, it may be difficult to obtain the complete harmonized data across the same studies for the same DataSchema variables, leading to a trade-off between including more studies or more covariates in an analysis. Further exploration of the heterogeneity existing across studies and the potential effect of the harmonization process on the variables generated could also be suitable. ^{Reference Friedenreich41,Reference Curran and Hussong42} Various approaches are then available to analyze data and the analytical models determined by the research questions addressed, data infrastructure, and variable content.