Hostname: page-component-7bb8b95d7b-2h6rp Total loading time: 0 Render date: 2024-10-04T13:14:38.517Z Has data issue: false hasContentIssue false

Involvement of information specialists and statisticians in systematic reviews

Published online by Cambridge University Press:  25 April 2023

Siw Waffenschmidt*
Affiliation:
Information Management Department, Institute for Quality and Efficiency in Health Care, Cologne, Germany
Ralf Bender
Affiliation:
Medical Biometry Department, Institute for Quality and Efficiency in Health Care, Cologne, Germany
*
Corresponding author: Siw Waffenschmidt; Email: siw.waffenschmidt@iqwig.de
Rights & Permissions [Opens in a new window]

Abstract

Background

Systematic reviews (SRs) are usually conducted by a highly specialized group of researchers. The routine involvement of methodological experts is a core methodological recommendation. The present commentary describes the qualifications required for information specialists and statisticians involved in SRs, as well as their tasks, the methodological challenges they face, and potential future areas of involvement.

Tasks and qualifications

Information specialists select the information sources, develop search strategies, conduct the searches, and report the results. Statisticians select the methods for evidence synthesis, assess the risk of bias, and interpret the results. The minimum requirements for their involvement in SRs are a suitable university degree (e.g., in statistics or librarian/information science or an equivalent degree), methodological and content expertise, and several years of experience.

Key arguments

The complexity of conducting SRs has greatly increased due to a massive rise in the amount of available evidence and the number and complexity of SR methods, largely statistical and information retrieval methods. Additional challenges exist in the actual conduct of an SR, such as judging how complex the research question could become and what hurdles could arise during the course of the project.

Conclusion

SRs are becoming more and more complex to conduct and information specialists and statisticians should routinely be involved right from the start of the SR. This increases the trustworthiness of SRs as the basis for reliable, unbiased and reproducible health policy, and clinical decision making.

Type
Commentary
Copyright
© Foundation for Quality and Efficiency in Heath Care, 2023. Published by Cambridge University Press

Background

Systematic reviews (SRs) of primary treatment and diagnostic studies form the basis of health technology assessments (HTAs) and evidence-based clinical practice guidelines. They thus inform health policy and clinical decision making, ensuring reliable, unbiased and reproducible health care decisions both on the individual and population-based level (Reference O’Rourke, Oortwijn and Schuller1).

SRs are usually conducted in a professional environment in specialized research teams working at HTA agencies, universities, and international organizations, also in collaboration with each other. Examples of HTA agencies are the UK National Institute for Health and Care Excellence (NICE), the French Haute Autorité de Santé (HAS), the German Institute for Quality and Efficiency in Health Care (IQWiG), and the Canadian Drug and Health Technology Agency (CADTH). Examples of universities are the Specialist Unit for Review Evidence at Cardiff University (UK), the Department of Health Research Methods, Evidence & Impact at McMaster University (Canada), and the Institute for Evidence-Based Healthcare at Bond University (Australia). Examples of international organizations are the Cochrane Collaboration and JBI.

The above organizations have in common that SRs are conducted by a team of researchers with specific areas of expertise, as no single researcher would be able to cover all the areas of expertise required. An SR team should routinely include methodological experts such as statisticians and information specialists to ensure the production of high-quality, that is, accurate and reproducible, SRs (Reference Metzendorf and Featherstone2;Reference Uttley and Montgomery3). Depending on the topic of the SR, other experts such as ethicists, epidemiologists, and health economists may also be involved.

However, although the call for routine involvement of these experts in SRs is supported by research findings, inadequate resources often hamper the implementation in small research groups (Reference Waffenschmidt, van Amsterdam-Lunze and Gomez4). This especially applies to information specialists – even though it is a “core methodological recommendation” to involve them, they “tend to fly somewhat under the radar” (Reference Metzendorf and Featherstone2).

In the guideline for preparing the implementation of the HTA Regulation of the European Parliament (5), the European HTA network EUnetHTA requires that one HTA team member has expertise in information retrieval and one has expertise in statistical analysis (6). These requirements for European HTA were the rationale for our commentary on the role of information specialists and statisticians in HTA to provide background information on these professions and to highlight the importance of their involvement.

The commentary describes the qualifications required for information specialists and statisticians involved in SRs, as well as their tasks, the methodological challenges they face, and potential future areas of involvement.

Tasks and qualifications

Information Specialists

Information specialists often work alone, as professional teams usually exist only in larger organizations. The importance of collaborating with peers has resulted in the establishment of information specialist networks such as the Cochrane Information Specialist Group (7) and the HTAi Interest Group Information Retrieval (8).

The main responsibility of information specialists is to provide the evidence base for SRs; in detail, the tasks include (Reference Waffenschmidt, van Amsterdam-Lunze and Gomez4;Reference Littlewood, Fenton and Bridges9Reference Spencer and Eldredge12):

  • being involved in the formulation of the research question,

  • selecting the type of information retrieval (e.g., comprehensive or focused), information sources (e.g., bibliographic databases, study registries), search techniques (e.g., Boolean search, citation searching), and study selection tools (e.g., EndNote, EPPI Reviewer) to be used,

  • developing and peer-reviewing search strategies,

  • conducting searches and update searches,

  • reporting information retrieval methods and results,

  • performing librarian tasks (e.g., full-text ordering, subscription management, reference list checks, reference and repository management, provision of advice on copyright and other questions relating to scholarly communications, dissemination of completed reports), and

  • further developing information retrieval methods (e.g., validation of different search techniques, including automation and digitalization; development of search filters).

Qualifications

Information specialists come from a variety of backgrounds with different types of training and levels of experience (Reference McGowan and Sampson11;Reference Booth and Beecroft13;Reference Tran14). Only a few countries, such as the United Kingdom, the United States, and Canada, offer formal university degrees (e.g., Master of Library and Information Science (15)). Information specialists from other countries such as Germany sometimes also hold a degree related to health care or other sciences, but have no formal university degree in library and information science (Reference Knüttel, Krause, Semmler-Schmetz, Reimann and Metzendorf16). However, a variety of training options exist to acquire the skills needed. In our opinion, the following minimum requirements should be met to qualify as an information specialist in an SR team:

  • A master’s degree in library and information science or another health-related or scientific qualification.

  • Several years of experience in information retrieval/evidence-based medicine/health science.

  • An understanding of health care.

  • Evidence of continued education and training in information retrieval methods.

Statisticians

Statisticians work in similar settings as information specialists (organizational group or alone). Various networking options are given, for example, by working groups within national and international societies such as the International Biometric Society (IBS), which contains a large number of countries, the International Society for Clinical Biostatistics (ISCB), and other statistical networks such as the Statistical Methods Group of the Cochrane Collaboration.

During the twentieth century, there was an explosive growth in the development of statistical methodology, leading to a strong increase in the involvement of statisticians in biomedical and epidemiological research (Reference Ellenberg17). Statisticians are nowadays involved from the planning phase of a research project to the data analysis and finally to the interpretation and publication of the results. Statisticians are coresponsible for an appropriate and efficient study design, an adequate data analysis, and a correct interpretation of the findings (Reference Zapf, Huebner, Rauch and Kieser18;Reference Zapf, Rauch and Kieser19).

Consequently, international regulatory guidelines for conducting clinical trials for drug approval require that statistical expertise should be utilized throughout all stages of a trial and that a “qualified statistician” takes responsibility for all statistical aspects (Reference Zapf, Huebner, Rauch and Kieser18;Reference Gerlinger, Edler and Friede20). As in clinical trials, a statistician should also be involved in all stages of an SR and is responsible for the following tasks (Reference Zapf, Rauch and Kieser19;Reference Higgins, Li, Deeks, JPT, Thomas, Chandler, Cumpston, Li, Page and Welch21Reference Schünemann, Higgins, Vist, Higgins, Thomas, Chandler, Cumpston, Li, Page and Welch25):

  • Assessing the risk of bias of the results of the included studies.

  • Choosing appropriate effect measures and summarizing the study results by means of appropriate methods for evidence synthesis (e.g., meta-analysis, network meta-analysis).

  • Exploring heterogeneity.

  • Assessing the assumptions of the chosen method for evidence synthesis.

  • Performing required subgroup and sensitivity analyses.

  • Presenting the SR results by means of appropriate graphs (e.g., forest, funnel, and network plots) and tables (e.g., summary of findings tables).

  • Interpreting the SR results.

Qualifications

Although graduate programmes are far more common for statisticians than for information specialists, there is no unique way of becoming a statistician and the educational systems for applied statistics vary widely across countries (Reference Zapf, Huebner, Rauch and Kieser18). The ISCB drafted a reflection paper to discuss what constitutes a “qualified statistician” within the framework of drug approval (Reference Gerlinger, Edler and Friede20). According to this paper, the following requirements should be fulfilled:

  • Typical career paths are given by, but are not limited to, special university degree programmes such as Master of Science in Medical Biometry and certificates such as the certificate “Biometry in Medicine” of the German Region of IBS (Reference Feldmann26).

  • The academic qualification should be at least a master’s degree in statistics or an equivalent degree.

  • Several years of practical experience in the planning, analysis, and reporting of clinical trials, including methods of computational statistics and software, are required.

  • Continued education and training, for example, via attending relevant conferences and workshops.

Equivalent requirements should be fulfilled for statisticians involved in SRs to ensure their high quality. In our opinion, the following minimum requirements should be met to qualify as a statistician in an SR team:

  • A master’s degree in statistics or an equivalent qualification (e.g., epidemiology with further education in clinical biostatistics).

  • Several years of practical experience in the planning, analysis, and reporting of clinical or epidemiological trials.

  • Several years of practical experience in computational statistics and statistical software.

  • Practical experience in methods of evidence-based medicine, meta-analysis, and other forms of evidence synthesis.

  • Continued education and training in new methods for evidence synthesis.

  • An understanding of health care.

Key arguments

Methodological Challenges in SRs

Besides a massive increase in the amount of scientific literature that needs to identified and screened, there has been a strong expansion of statistical and information retrieval methods, which are also becoming more and more complex, thus making it difficult to keep up to date and impossible for a single researcher to cover (Reference Lefebvre, Glanville, Wieland, Coles and Weightman27). The first challenge in an SR emerges early in the planning phase, as it is difficult to judge how complex the research question could become and what hurdles could arise during the course of the project; methodological expertise is thus already essential at this early stage. The next challenge is the choice of the appropriate method for each case of use, including the consideration of resources. At this stage too, the involvement of information specialists and statisticians is essential and helps to minimize bias and produce valid robust results as well as to reduce research waste (Reference Waffenschmidt, van Amsterdam-Lunze and Gomez4).

Further Involvement of Information Specialists and Statisticians in SRs

After the establishment of the Cochrane Collaboration in the 1990s, the routine involvement of information specialists and statisticians was soon specified on the organizational level (e.g., introduction of the first “Trials Search Coordinators” in Cochrane). Similar requirements are currently being implemented within the planned harmonization of European HTA from 2025 onwards: As noted above, the European HTA network EUnetHTA requires both expertise in information retrieval and in statistical analysis (6). In our opinion, these requirements should also apply to other areas such as scientific journals (peer reviews of SRs), as well as assessment and reporting guidelines for SRs (inclusion of items in AMSTAR (Reference Shea, Reeves and Wells28) and PRISMA checklists (Reference Page, McKenzie and Bossuyt29;Reference Rethlefsen, Kirtley and Waffenschmidt30) specifying the methodological expertise required).

Coping with Limited Resources

In resource-limited settings, it will not always be possible to include methodological experts as an integral part of an SR team. However, other solutions have been developed to make use of their expertise. For example, a network of information specialists has been established within EUnetHTA (Joint Action 3) to provide input to individual assessments conducted by smaller HTA agencies (Reference Waffenschmidt, van Amsterdam-Lunze and Gomez4). A similar network of statisticians has also been established. In addition, many HTA agencies that do not directly employ methodological experts seek their advice on a freelance basis or use support services such as libraries with information management services. Online consultations can be provided to facilitate access to external expertise.

PhD students or other early stage researchers at universities usually have little experience in conducting SRs. Information specialists and statisticians should therefore be directly involved in researcher development, for example through workshops. In addition, advice on SRs should be available on an ad hoc basis. The implementation of these measures would avoid bias and the waste of resources, thus enabling the production of efficiently produced SRs of high quality.

Conclusion

SRs are becoming more and more complex to conduct and methodological experts such as information specialists and statisticians should routinely be involved right from the start of a project. They ensure the high methodological quality of an SR by applying up-to-date and appropriate methods, resulting in unbiased and thus reliable results. This increases the trustworthiness of SRs as the basis for informed health policy and clinical decision making.

Acknowledgments

The authors thank Natalie McGauran for editorial support.

Funding statement

This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.

Competing interest

The authors declare none.

References

O’Rourke, B, Oortwijn, W, Schuller, T, International Joint Task Group. The new definition of health technology assessment: A milestone in international collaboration. Int J Technol Assess Health Care. 2020;36:187-190.Google ScholarPubMed
Metzendorf, MI, Featherstone, RM. Ensuring quality as the basis of evidence synthesis: Leveraging information specialists’ knowledge, skills, and expertise. Cochrane Database Syst Rev. 2018;4:ED000125.Google ScholarPubMed
Uttley, L, Montgomery, P. The influence of the team in conducting a systematic review. Syst Rev. 2017;6:149.Google Scholar
Waffenschmidt, S, van Amsterdam-Lunze, M, Gomez, RI, et al. Information specialist collaboration in Europe: Collaborative methods, processes, and infrastructure through EUnetHTA. Int J Technol Assess Health Care. 2020;37:e20.CrossRefGoogle ScholarPubMed
European Parliament. Council of the European Union [Internet] Regulation of the European Parliament and of the Council on health technology assessment and amending Directive 2011/24/EU [cited 28.12.2021]. 2021. Available from: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32021R2282&from=IT.Google Scholar
EUnetHTA 21. [Internet] D5.3.1 – Procedural guidance for appointment of assessors and co-assessors for JCA/CA – guidance document [cited 15.08.2022]. 2022. Available from: https://d2yaq9q3r816qg.cloudfront.net/wp-content/uploads/2022/06/D5.3.1_Appointment_Assessors_CoAssessors-final-version-v1.0.pdf?x69613.Google Scholar
Cochrane Community. [Internet] Information Specialists Portal [cited 15.08.2022]. 2022. Available from: https://community.cochrane.org/organizational-info/resources/resources-groups/information-specialists-portal.Google Scholar
HTAi. [Internet] Interest Group Information Retrieval [cited 15.08.2022]. 2022. Available from: https://htai.org/interest-groups/information-retrieval/.Google Scholar
Littlewood, A, Fenton, C, Bridges, C, et al. [Internet] Cochrane information specialists’ handbook [cited 06.06.2019]. 2018. Available from: https://training.cochrane.org/resource/cochrane-information-specialists-handbook.Google Scholar
Waffenschmidt, S, Hausner, E. Collaborative working to improve searching. In: Levay, P, Craven, J, editors. Systematic searching: Practical ideas for improving results. London: Facet Publishing; 2019. p. 229-248.Google Scholar
McGowan, J, Sampson, M. Systematic reviews need systematic searchers. J Med Libr Assoc. 2005;93:7480.Google ScholarPubMed
Spencer, AJ, Eldredge, JD. Roles for librarians in systematic reviews: A scoping review. J Med Libr Assoc. 2018;106:4656.CrossRefGoogle ScholarPubMed
Booth, A, Beecroft, C. The SPECTRAL project: A training needs analysis for providers of clinical question answering services. Health Info Libr J. 2010;27:198207.CrossRefGoogle Scholar
Tran, A. In-house peer supported literature search training: A public health perspective. Health Info Libr J. 2017;34:258262.CrossRefGoogle ScholarPubMed
Wikipedia. [Internet] Master of Library and Information Science [cited 07.12.2022]. 2022. Available from: https://en.wikipedia.org/wiki/Master_of_Library_and_Information_Science.Google Scholar
Knüttel, H, Krause, E, Semmler-Schmetz, M, Reimann, I, Metzendorf, M-I. Health sciences libraries in Germany: New directions. Health Info Libr J. 2020;37:8388.CrossRefGoogle ScholarPubMed
Ellenberg, JH. Biostatistical collaboration in medical research. Biometrics. 1990;46:118; discussion 19-32.CrossRefGoogle ScholarPubMed
Zapf, A, Huebner, M, Rauch, G, Kieser, M. What makes a biostatistician? Stat Med. 2019;38:695701.CrossRefGoogle ScholarPubMed
Zapf, A, Rauch, G, Kieser, M. Why do you need a biostatistician? BMC Med Res Methodol. 2020;20:23.CrossRefGoogle Scholar
Gerlinger, C, Edler, L, Friede, T, et al. Considerations on what constitutes a ‘qualified statistician’ in regulatory guidelines. Stat Med. 2012;31:13031305.Google ScholarPubMed
Higgins, JPT, Li, T, Deeks, JJ. Choosing effect measures and computing estimates of effect. In: JPT, Higgins, Thomas, J, Chandler, J, Cumpston, M, Li, T, Page, MJ, Welch, VA, editors. Cochrane handbook for systematic reviews of interventions. Hoboken: Wiley-Blackwell; 2019. p. 143-176.CrossRefGoogle Scholar
Higgins, JPT, Savović, J, Page, MJ, Elbers, RG, Sterne, JAC. Assessing risk of bias in a randomized trial. In: Higgins, JPT, Thomas, J, Chandler, J, Cumpston, M, Li, T, Page, MJ, Welch, VA, editors. Cochrane handbook for systematic reviews of interventions. Hoboken: Wiley-Blackwell; 2019. p. 205228.CrossRefGoogle Scholar
Deeks, JJ, Higgins, JPT, Altman, DG. Analysing data and undertaking meta-analyses. In: Higgins, JPT, Thomas, J, Chandler, J, Cumpston, M, Li, T, Page, MJ, Welch, VA, editors. Cochrane handbook for systematic reviews of interventions. Hoboken: Wiley-Blackwell; 2019. p. 241284.CrossRefGoogle Scholar
Chaimani, A, Caldwell, D, Li, T. Undertaking network meta-analyses. In: Higgins, JPT, Thomas, J, Chandler, J, Cumpston, M, Li, T, Page, MJ, Welch, VA, editors. Cochrane handbook for systematic reviews of interventions. Hoboken: Wiley-Blackwell; 2019. p. 285320.Google Scholar
Schünemann, HJH, Higgins, JPT, Vist, GE, et al. Completing ‘Summary of findings’ tables and grading the certainty of the evidence. In: Higgins, JPT, Thomas, J, Chandler, J, Cumpston, M, Li, T, Page, MJ, Welch, VA, editors. Cochrane handbook for systematic reviews of interventions. Hoboken: Wiley-Blackwell; 2019. p. 375402.CrossRefGoogle Scholar
Feldmann, U. Qualification of the responsible trial statistician. Biom J. 2002;44:117118.3.0.CO;2-Q>CrossRefGoogle Scholar
Lefebvre, C, Glanville, J, Wieland, LS, Coles, B, Weightman, AL. Methodological developments in searching for studies for systematic reviews: Past, present and future? Syst Rev. 2013;2:78.CrossRefGoogle ScholarPubMed
Shea, BJ, Reeves, BC, Wells, G, et al. AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.CrossRefGoogle ScholarPubMed
Page, MJ, McKenzie, JE, Bossuyt, PM, et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.CrossRefGoogle ScholarPubMed
Rethlefsen, ML, Kirtley, S, Waffenschmidt, S, et al. PRISMA-S: An extension to the PRISMA statement for reporting literature searches in systematic reviews. Syst Rev. 2021;10:39.Google Scholar