Highlights
What is already known?
-
• Around 50% of all quantitative research studies remain unpublished. The proportion of non-publication in qualitative research was only investigated in small studies.
-
• Dissemination bias occurs when the non-publication of research is selective and based on the findings and main messages, potentially reducing the certainty in evidence syntheses. While this phenomenon is well-documented within quantitative research, its extent in qualitative health research remains unclear.
What is new?
-
• In this retrospective cohort study, we followed 1,123 conference abstracts across various medical and health-related disciplines.
-
• Almost one quarter were still unpublished 6–8 years later. Half of the studies that were published after the conference were published after 11 months.
Potential impact for RSM readers
-
• Many qualitative studies are never published. This represents research waste and missed opportunities to understand non-dissemination and dissemination bias in qualitative research and to develop methods to aid the assessment of the plausible impact of dissemination bias on qualitative evidence syntheses.
1 Background
Dissemination bias describes a systematic error occurring from the selective non-dissemination of studies and individual findings.Reference Song, Parekh and Hooper 1 In the context of qualitative research, dissemination bias has been defined as a systematic distortion of the phenomenon of interest due to selective dissemination of qualitative studies or the findings of qualitative studies.Reference Booth, Lewin and Glenton 2 Also referred to as publication bias, dissemination bias involves more than the non-publication of studies. Dissemination bias considers when, where and in what format research or individual findings are published and covers underlying mechanisms or biases that impact on the accessibility of a study.Reference Song, Parekh and Hooper 1 Given its evidently harmful impact on quantitative research,Reference Chan, Song and Vickers 3 , Reference Turner, Cipriani, Furukawa, Salanti and de Vries 4 it is pertinent to consider dissemination bias in qualitative research as well, where the extent of, reasons for and implications of this phenomenon are, as of yet, relatively uncharted.Reference Booth, Lewin and Glenton 2
Until 2016, only one study had investigated non-dissemination in qualitative research.Reference Petticrew, Egan, Thomson, Hamilton, Kunkler and Roberts 5 In 2016, a large international cross-sectional survey gathered data regarding non-dissemination and dissemination bias in qualitative research from the scientific community.Reference Toews, Glenton and Lewin 6 Notably, almost 70% of the participating researchers reported that at least one of their qualitative studies had not been published in a peer-reviewed journal. Respondents perceived that non-dissemination of qualitative research has negative impacts on health policy. In a further study, a cohort of qualitative studies was followed.Reference Toews, JLZ and Stadelmaier 7 The authors found that one-third of qualitative studies presented at nursing research conferences did not result in full publication (including full texts in scientific journals as well as in grey literature) 5 years after presentation. Previous literature had supported the notion that dissemination bias in qualitative research is likely to lead to poor decision-making, potential research waste, and missed opportunities to support healthcare decisions, planning, and implementation.Reference Toews, Booth and Berg 8 Dissemination bias has therefore been suggested as a candidate domain of the GRADE-CERQual approach for assessing confidence in findings from qualitative evidence syntheses.Reference Booth, Lewin and Glenton 2 Still, there is insufficient evidence in this field to draw clear conclusions on its impact and consequences. The present study is linked to the development of the domain “Dissemination bias” in the GRADE-CERQual approach. It is intended to serve as a foundation for further research and conceptualization of the “Dissemination bias” domain within the overall approach. This study aims to extend the results of previous research by mapping and quantifying the extent of non-dissemination in qualitative health research, the time to publication and the factors associated with non-dissemination in a representative sample of qualitative studies in health and healthcare.
2 Methods
2.1 Study setting and data sources
Using a retrospective cohort study design, we followed a cohort of health-related qualitative studies initially presented as conference abstracts to investigate their full publication status later on. We used purposive, non-probabilistic sampling to build our study sample. We included studies that were presented at conferences between 2016 and 2018. We selected this time period to allow sufficient time for full publication. The follow-up period from conference presentation to the search for full publication was 6–8 years. Study methods build on methods of a previous studyReference Toews, JLZ and Stadelmaier 7 and refined these based on peer-review feedback. The study is reported according to the STROBE statementReference von Elm, Altman and Egger 9 and registered in OSF (https://osf.io/u7gtw).Reference Weber 10
Abstracts were retrieved via searches in Web of Science Conference Proceedings Citation Index – Science (CPCI-S) and Conference Proceedings Citation Index – Social Science & Humanities (CPCI-SSH) via the Web of Science Platform (Clarivate) on September 21, 2022 (see Supplement 1 of the Supplementary Material for search strategy). We selected this database due to its large coverage of conference abstracts in health and healthcare research. The search strategy was based on validated search strategies for related databases and adapted for this study.Reference Rogers, Bethel and Abbott 11 , Reference Rosumeck, Wagner, Wallraf and Euler 12
To be included, abstracts needed to fulfill one of the following criteria: (i) focused on describing an original qualitative study related to health and healthcare, (ii) described a mixed-methods study or a program description with at least one qualitative element, or (iii) described an evidence synthesis of qualitative research findings. We excluded abstracts reporting exclusively quantitative studies, broad methodological or theoretical discussions. Also, studies lacking a clear focus on health and healthcare research were excluded. A lack of reporting detail was not considered as a reason for exclusion. See Supplement 2 of the Supplementary Material for further information on inclusion and exclusion criteria of studies.
To identify eligible studies, each abstract was screened by at least two independent reviewers. In total, eight reviewers (MT, AB, SL, HMK, CG, JN, IT, and JN) participated in the screening process in Covidence. 13 We extracted author information and study characteristics (see Table 1) from the database record of all included studies by using a semi-automated approach with repeated, regular consistency checks in R-Studio. 14 Data that could not be extracted by the semi-automated approach were extracted by two researchers (MW and CS) between February and August 2023. We focused only on selected data, that is, on the first author, to ensure better manageability of the data and analyses. This information was used to match full publications and, subsequently, to explore potential factors associated with dissemination. Multiple publications of one study were consolidated within a single unit of analysis and a full account of the reported information was considered in the data extraction and analyses.
Sample characteristics
2.2 Searching for full publications and matching
We used two approaches, a survey and systematic literature searches, to identify full publications of the studies initially presented as conference abstracts. For the survey, contact information extracted from the database record was used to contact the study authors. The survey was created and distributed via personalized e-mail using the SoSci SurveyReference Leiner 15 platform to ascertain a full publication of the respective studies. Detailed results of the survey are published elsewhere.Reference Weber, Lewin and Meerpohl 16
For the literature searches, Google Scholar; MEDLINE via OVID; ProQuest & Google were searched in a stepwise approach for full publications/texts of the included abstracts. We conducted the searches between August 2023 and August 2024. We used individual search strategies for each study using a combination of author names, text words describing the phenomenon of interest, and the study methods as reported in the conference abstract. We combined search concepts using Boolean operators (AND and OR) and the applicable database settings, respectively, to find candidate publications. In case of either zero or too many (>20) search hits, we broadened or narrowed the search strategy by searching exact phrases and/or adapting the Boolean operators. Potentially matched full publications were screened for eligibility against author- and study information. We considered a full publication a match if the phenomena of interest and the method(s) of inquiry were the same in both records, and at least one of the conference abstract authors were authors in the full publication.
We considered a full publication to be any form of full text adding substantial detail to the original conference abstract information and reporting methods, final findings and to the interpretation or discussion of the findings. Substantive full publications included journal articles (both peer-reviewed and not peer-reviewed and extended versions published as conference proceedings, brief reports, and research letters), books, and book chapters (including collections of conference proceedings), grey literature (theses or dissertations published by organizations outside traditional commercial or academic publishing) and any other form of extended versions (e.g., conference proceedings published via standalone electronic libraries, conference organizers or universities, letters). For each matched pair of outputs, we extracted the month and year of publication and type of publication for the full publication.
2.3 Statistical analysis
All study data were documented in MS Excel 2016 and analyzed with R. 14 First, data were summarized with univariate descriptive statistics. The proportions of studies with and without full publication were analyzed. Fisher’s exact test was calculated to examine associations between dichotomous study- and author characteristics and the proportion of abstracts with a matching full publication. Logistic regression analyses were used to analyze associations between differently scaled (categorical and ordinal) study- and author characteristics (see Supplement 3 of the Supplementary Material for further details) and full publication. For ordinal scaled categories, we selected the factor with the most cases as reference post hoc in the logistic regression. When testing for associations, we excluded unspecific categories (e.g., unclear; not reported) in order to be able to assign associations more clearly. Results are reported as odds ratios (ORs) together with their 95% confidence intervals (CIs).
Median time to publication was calculated with a time-to-event analysis and illustrated using a cumulative incidence plot. For time-to-event analyses, when the exact conference month was not reported but the year was known, we substituted December for the missing value as the most conservative estimate. Additionally, we evaluated associations of study and author characteristics with duration to publication using Cox regression analyses (R packages survivalReference Therneau 17 and survminer 18 ). All analyses were bivariate, focusing on exploring simple associations between a single independent variable and the dependent variable. Dichotomous data are presented as frequencies and proportions, while continuous data are summarized as means and standard deviations (SDs) or as medians, depending on the distribution of the data.
3 Results
We found 7,502 records in our search. A total of 428 records were automatically excluded as duplicates. After screening 7,074 records, we included 1,123 conference abstracts for follow-up. A total of 5,951 records were excluded because they did not fulfill our eligibility criteria (see Figure 1). The majority of the included studies were presented at conferences in North America or Europe. The majority of the abstracts’ first authors were female. Most authors were affiliated with a North American or European institution. In most included studies, data were retrieved from a single population group and most studies employed one method of data collection. About one-third of studies reported that they had received funding, of which approximately three-quarters were publicly funded. Almost all of the abstracts reported the study aim and conclusions. We defined the majority of the studies as cross-sectional rather than longitudinal. See Table 1 for more detailed information.
Flow chart of conference abstracts and follow-up.
3.1 Proportions of non-dissemination and time-to-publication
For 256 of the 1,123 (22.80%) included conference abstracts, no full publication could be retrieved within at least 6 (for studies presented in 2018) and up to 8 years (for studies presented in 2016) after presentation. Conversely, for 867 of 1,123 (77.20%), we retrieved at least one matching full publication. We retrieved 113 of the 867 matching full publications (13.03%) via the survey. Of all full publications, 642 (74.05%) were published in a scientific journal, 35 (4.04%) in a book or book chapter, 27 (3.11%) in grey literature, and 163 (18.8%) as other publication format. The median time to any full publication was 11 months (95% CI 10 to 12 months) (see Figure 2).
Cumulative incidence plot with 95% CIs illustrating the time to full publication of qualitative studies presented at conferences.
Note: Due to the full publication rate of 0.77, the median is visualized at 0.385 on the scale for proportion of publication in this cumulative incidence plot. “0” on the scale for months represents the conference date.
3.2 Characteristics associated with dissemination
Table 2 contains results for associations of the conference abstract characteristics with full publication. We found that studies from authors affiliated with institutions in Australia were more than four times more likely to result in a full publication than those from North America (OR 4.47; 95% CI 1.58 to 18.74). Other author characteristics showed no statistically significant associations.
Conference abstract characteristics and their association with full publication
Note: Missing data and unclear information were excluded for the analysis of associations.
Abbreviations: OR, odds ratio; CI, confidence interval.
a Calculated with Fisher’s exact test.
b Calculated with logistic regression.
* p < 0.05.
The following study characteristics showed statistically significant associations with a full publication: Oral presentations were over three times more likely to result in a full publication than poster presentations (OR 3.40; 95% CI 1.57 to 8.20). Studies that used two qualitative data collection methods were one and a half times more likely to result in a full publication than studies that used one qualitative method only (OR 1.53; 95 CI 1.01 to 2.38). Conference abstracts that reported no funding were less likely to result in a full publication as those which reported funding (OR 0.71; 95% CI 0.51 to 0.99). Publicly funded research was over twice as likely to result in a full publication when compared to privately funded research (OR 2.24; 95% CI 1.16 to 4.28).
Other study characteristics showed no statistically significant associations.
3.3 Characteristics associated with time-to-publication
We found that studies from authors affiliated with institutions in Australia and Asia were associated with shorter time to publication. Furthermore, oral presentation as presentation format, community care as level of care, reporting of the funding as well as not reporting of study aim, and study conclusion showed associations with shorter time-to-publication. Other study or author characteristics showed no statistically significant associations with time-to-publication. Full results and plots of the Cox Regression are presented in the Supplement 4 of the Supplementary Material.
4 Discussion
We conducted a retrospective cohort study to investigate the subsequent full publication of qualitative health- and healthcare-related studies initially presented as conference abstracts. We found that for almost one-quarter (22.8%) of conference abstracts no full publication could be retrieved after up to 8 years. Half of the included studies resulted in a full publication within around 11 months after the conference. The presentation format, geographic location of the first authors’ affiliation, number of methods used for data retrieval, reporting of the funding as well as funding type seemed to be associated with a full publication. Moreover, Australian and Asian affiliated authorship, presentation as presentation format, community care as level of care, reporting of the funding as well as not reporting of study aim, and study conclusion seemed to be associated with a shorter time-to-publication.
This study showed lower proportions of non-dissemination than previous studies with a similar research question. A studyReference Toews, JLZ and Stadelmaier 7 with a similar design of qualitative research initially presented at a nursing conference showed a slightly higher proportion of no subsequent full publications (33.7%). In another study of qualitative research, Petticrew et al.Reference Petticrew, Egan, Thomson, Hamilton, Kunkler and Roberts 5 uncovered a non-publication proportion of 55.8% of abstracts. Both studies had slightly shorter follow-up timeframes (5–7 years) than our present study. To contextualize our findings, in quantitative health-related research, the proportion of published studies ranged between 45.8% and 53.8% with a follow-up time between at least 2 and up to 10 years.Reference Schmucker, Schell and Portalupi 19 , Reference Scherer, Meerpohl, Pfeifer, Schmucker, Schwarzer and von Elm 20
These differences might be explained by more exhaustive search methods and structural improvements in research publishing. We conducted extensive follow-up searches in multiple databases and grey literature and sought direct author contact to obtain full publications. We extended the duration of follow-up in comparison to other studies to up to 8 years. Furthermore, we drew our study sample from an electronic database with a broad thematic scope. Therefore, we included abstracts from different conferences with different abstract submission and presentation criteria. In our sample, a substantial part of full publications was published simultaneously with the conference in form of proceeding papers (see cumulative incidence plot in Figure 2), which we considered a full publication according to the previously defined inclusion criteria as they clearly reported results and exceeded the conference abstracts. We searched for publications in scientific and grey literature to capture different forms of publications of completed research, not just journal publications. All these factors might have increased the yield of our search for (subsequent) publications. Besides methodological factors that might have promoted a higher detection of subsequent publications, it can be hypothesized that the proportions of subsequent publications of conference abstracts have, in fact, increased. This will have to be investigated in future research.
Structurally, the research publication system has evolved in recent years increasingly encouraging registration of different types of research, using research protocols, thereby increasing scientific rigor.Reference Thaler, Kien and Nussbaumer 21 , Reference Haven and Van Grootel 22 This might be reflected in a higher conversion rate of studies presented as conference abstracts to full publications.
The time to publication (median 11 months) was similar to the previous study of Toews et al.Reference Toews, JLZ and Stadelmaier 7 For comparison, a corresponding study of clinical trialsReference Showell, Cole, Clarke, DeVito, Farquhar and Jordan 23 showed that median time to publication of clinical trials was 4.8 years from enrolment of the first participant and 2.1 years from trial completion. The phenomenon of qualitative studies being published before the conference date was also observed in Toews et al.Reference Toews, JLZ and Stadelmaier 7 as well. Similar observations have been reported in specific clinical fields, for example, in pediatric radiologyReference Meshaka, Laidlow-Singh, Langan, Arthurs and Shelmerdine 24 and pediatric surgery,Reference Oetzmann von Sochaczewski, Katzer, Ganschow and Muensterer 25 and forensic imaging,Reference de Heus, de Bakker and Decker 26 although this aspect has typically not been analyzed systematically across research areas. Consequently, direct comparisons between disciplines remain limited.
In other (health-related) areas, such as medical and health informatics, assistive technologies, or bioinformatics, we found that the full publication of results as conference paper is more common or even required by the conference submission guidelines. Regarding the associations, oral presentations compared to poster presentations were also found to be associated with higher chances of full publication as well as time-to-publication in a studyReference Toews, JLZ and Stadelmaier 7 with a similar methodological approach. Other significant associations did not correspond with the previous study.Reference Toews, JLZ and Stadelmaier 7 Due to the exploratory approach, we are not able to draw final conclusions, and advise against doing so.
5 Limitations
We drew a large, representative, and systematic sample from a search in Web of Science—a database that contains the highest number of conference abstracts in health and healthcare research. In comparison to other studies on this topic that drew samples from conference books and proceedings, sampling from a database is a novel and efficient approach that allows for more representative findings. Furthermore, we explicitly searched for full publications in the grey literature. We initially planned to search the EThOS database but it was not accessible during the search period. Therefore, we relied on Google searches for grey literature and did not search institutional repositories as this was too time consuming.
We followed a comprehensive process to match full publications with the relevant conference abstract. Nonetheless, matching through the phenomenon of interest, author, and qualitative method of inquiry was sometimes challenging, due to the brevity of reporting in conference abstracts.
Due to the large dataset and therefore feasibility, data extraction and searching for full publications and matching was not conducted in duplicate. A second study author (IT) checked all extracted study data for accuracy. The search strategies for full publications were randomly checked for accuracy and tested, likewise the matching full publications were checked. The latter two pertained to about 30% of the dataset.
We encountered issues of missing or unclear information (e.g., non-standardized reporting about the level of care or the methods used) that made it difficult to code study and author information clearly. This may have affected the validity of the respective associations between author and study characteristics and full publication. Missing information during the extraction and unclear cases during the matching were double-checked and discussed continuously throughout the process.
Moreover, we used study and author characteristics to investigate which factors might be associated with subsequent non-dissemination. The underlying bivariate analysis was inherently exploratory. As a result, confounder control or complex model selection procedures were not necessary. This approach may have potentially led to an overestimation of associations. Although we carefully selected characteristics, our analysis has multiple comparisons possibly resulting in false positive findings.
Overall, there is a chance that the proportions of non-dissemination of qualitative studies is substantially different from our findings. We were unable to account for those qualitative studies that were conducted but never presented at a conference. The impact of missing studies is likely to remain greater than studies omitted through sampling. In general, it is challenging and time consuming to establish the extent of non-dissemination from conference presentations without record linkage and transparent reporting. Mandating study identifiers for qualitative research, and requiring study protocols and/or a study registry for qualitative studies might enhance transparency and simplify matching to full publications.Reference Thaler, Kien and Nussbaumer 21 , Reference Carroll, Toumpakari, Johnson and Betts 27 It is important to emphasize that this would not aim to solve non-dissemination or dissemination bias, but might help provide a clearer and more precise picture of non-dissemination and sensitize interest holders for the potential impact of dissemination bias.
We recommend that future studies carefully balance sample size with feasibility of a study, recognizing that extending the search scope or restricting abstract inclusion may still lead to underrepresentation. Studies may extend the coverage to more than one database while, at the same time, narrowing the inclusion criteria for the abstracts. A large sample of studies with standardized specifications may improve the structure and clarity of data extraction potentially allowing for a closer focus on the nature and content of the study findings which are of key interest for investigating dissemination bias. Nevertheless such methods remain exploratory in nature. Further insights of reasons for non-dissemination beyond those that are applicable to research in general,Reference Toews, Glenton and Lewin 6 and that are related to the message and findings, remain of persistent interest if we are to understand more about actual dissemination bias in qualitative research.
6 Conclusions
Our study extends previous research, building a knowledge base for understanding the proportion of non-dissemination in qualitative research. We found that the proportion of non-dissemination in qualitative research is still considerable, too. Ascertaining the true proportion of non-dissemination is challenging as some studies may never be presented at conferences. We therefore assume that dissemination bias might potentially impact on qualitative evidence synthesis and thereby on decision-making that uses qualitative evidence.
Conceptually, using unpublished trials and systematic reviews of intervention effects as a reference, the impact of dissemination bias in qualitative research may be similarly grave on the confidence we may have in findings of evidence synthesis. Definitive data on dissemination bias remain elusive given the difficulties in identifying examples which can be tracked and the challenge of finding evidence for its existence, especially in qualitative research. In order to build a framework for understanding dissemination in qualitative research, more research is needed on how dissemination bias influences the qualitative evidence base and how we may begin to identify these distortions.
Acknowledgements
We thank John Nyirenda and Charlotte Schaffer for supporting the screening of references and data extraction. Preliminary results of this study were presented as a poster at the Global Evidence Summit in Prague, September 2024.
Author contributions
Conceptualization: A.B., S.L., H.M.M.-K., C.G., J.N., J.J.M., I.T.; Data curation: M.W., M.T., W.S., I.T.; Formal analysis: M.W., M.T., W.S.; Funding acquisition: J.J.M., I.T.; Investigation: M.W., M.T., W.S., I.T.; Methodology: M.W., M.T., W.S., I.T.; Project administration: I.T.; Resources: M.T., W.S.; Software: M.W., M.T., W.S.; Supervision: M.W., I.T., Validation: M.W., M.T., W.S., A.B., S.L., H.M.M.-K., C.G., J.N., J.J.M., I.T.; Visualization: M.W., M.T., W.S.; Writing—original draft: M.W., I.T., Writing—review and editing: M.W., M.T., W.S., A.B., S.L., H.M.M.-K., C.G., J.N., J.J.M., I.T.
Competing interest statement
A.B., S.L., H.M.M.-K., C.G., and J.N. reported being part of the GRADE-CERQual project group. The remaining authors declare none.
Data availability statement
Data and analysis code are available in the Open Science Framework (OSF) at https://osf.io/36ypd. This study is registered in OSF at https://osf.io/u7gtw.
Funding statement
The design of the study was funded by the Research Commission of the University Medical Center Freiburg (TOE2199/22).
Supplementary material
To view supplementary material for this article, please visit http://doi.org/10.1017/rsm.2026.10085.
Open access
