The objective of a systematic review is to integrate the existing knowledge in a particular subject or, more specifically, concerning a scientific question. A systematic review is carried out under the following conditions:
(1) A review is conducted by a methodology which is previously set out and fully detailed in a protocol that is thorough, critical and designed to limit bias.
(2) There is an analysis and integration of the findings from the primary research studies regarding the health problem posed.
In the case where the study design included in the systematic review enables the combination and statistical treatment of the results, a meta-analysis or quantitative systematic review can be carried out.
The first reference to a systematic review was made by Sir George Biddell Airy (1801–1892)(Reference Airy1), the British mathematician and astronomer. His manual describes the method used to carry out a quantitative synthesis of the results of different studies, which Glass(Reference Glass2 )subsequently named “Meta-Analysis”. In the biomedical field, one of the first examples of meta-analysis was published in 1904 by Karl Pearson(Reference Pearson3), who concluded that the vaccine for Intestinal Fever was not effective enough to be recommended as a vaccination. Statistical methods were later developed for meta-analysis, and from the 1970 s onwards they were applied with greater frequency, with a large number of publications specifically by Glass(Reference Glass2, Reference Glass, McGaw and Smith4) and Stjernswärd(Reference Stjernswärd5, Reference Stjernswärd, Muenz and von Essen6).
The benefits of systematic reviews are derived from the pursuit of their specific objectives, namely the description of a specific event; the analysis of the impact of certain factors on an illness; the appraisal of diagnostic tests; the assessment of the treatment; the identification of special population groups or an evaluation of the characteristics of individual research(Reference Delgado Rodriguez7).
Reviews can be proposed with different starting points: for example, with the purpose of examining existing knowledge surrounding particular questions posed. In these cases, if an association is found, a hypothesis can be put forward to test the possible relationship between exposure and effect, its meaning and magnitude, and if the same effect is experienced by the different population subgroups. Furthermore, methodological problems or other flaws in the study of the association to be analysed may be detected.
For any initial analysis in a systematic review, it is very important to establish the protocol to be followed, as this will determine the criteria for the selection and inclusion of articles (study design, language, quality, etc), the strategies to avoid any possible bias as well as the format for presenting the results.
This study thus provides a methodological proposal for qualitative systematic reviews.
Systematic review methodology
Although there exist different approaches for scientific literature reviews, systematic reviews are considered to offer the most reliable method. The systematic review protocol guarantees that the review is carried out with the rigours demanded of all research and should contain the following sections:
– Formulation of the problem to be studied.
– Identification of the studies through a bibliographic search.
– Selection of the studies and a critical quality assessment.
– Data extraction and synthesis of results.
– Conclusions and Recommendations.
Defining the research question
This is the preliminary, fundamental step of all scientific literature systematic reviews, since adequate formulation of the research question will determine the correctness of the process(Reference Rothman, Greenland and Lash8), which will in turn generate the protocol.
The research question must be based on the following elements: specific population to be researched; evaluation of the intervention; a comparison using other techniques or options, if appropriate, and the results which will give a measure of the intervention studied(9, 10). These elements comprise the acronym PICO (Population, Intervention, Comparison and Outcomes).
Bibliographic searches and data sources
The identification and location of studies must be carried out via a strategic and systematic bibliographic search which identifies the appropriate studies. It must also be easily replicated and applicable to the majority of existing health science databases.
This search must take into account the elements of the review research question, as well as the most appropriate study designs(11). Consequently, an effective review search formula is developed, determining the suitable Descriptors and Qualifiers(12, 13)for each of the PICO elements, combining them with the appropriate Boolean operators. It is important to establish the criteria for inclusion in the review beforehand, in order to determine which articles should be extracted.
However, it is important to consider that the exhaustiveness of the search is more important than its precision (where more irrelevant articles are identified), although the ideal is to strike a balance between the degree of exhaustiveness (highly sensitive searches) and precision (specific searches).When deciding the scope of the bibliographic search formula, it is always preferable to use noise-generating words, which may result in retrieving irrelevant articles that can subsequently be discarded, than risk losing relevant articles that have not been retrieved by the search. In the event of the latter case, where highly defined searches are used, it is advisable to redefine the search formula.
Once the search is completed on the computerised databases (a systematic review should strive to be as broad as possible in the use of bibliographic sources), other information sources need to be consulted in order to minimise any possible publication bias. Furthermore, the bibliography included in the selected articles should be checked with a view to locating additional studies; “grey literature” should be retrieved, such as doctoral theses, conference proceedings, research projects, reports from institutions or from the pharmaceutical industry, and information should be requested from experts.
It is important for the search strategy to be clearly described in the protocol and subsequently in the publication of the systematic review, including the Descriptors, their combination, the period covered by the search, the types of articles being searched, and any other information that helps to accurately reproduce the search carried out.
Concerning documentary typology, it is important to highlight that the Cochrane Collaboration focuses on systematic reviews of randomised controlled trials as they tend to provide more reliable information than other sources of evidence on the differential effects of the various health care alternatives(14).
Selection and critical appraisal of studies
Strict compliance with this section will limit bias, facilitate the interpretation and discussion of the results as well as ensure the replication of the process. This section must establish the most suitable research design for tackling the question under review.
Studies will be selected according to the inclusion or exclusion criteria established by two independent reviewers. Whenever possible, the independent reviewers will not be aware of the authors, journals or results of the articles to be selected. It is useful to measure the correlation between the data obtained by the independent reviewers, and establish beforehand the minimum level of agreement that is required.
Experts in a specific area often have previously formed opinions that can bias their evaluations, both in terms of the relevance and the validity of the articles to be included. Therefore, although it is important that at least one of the reviewers is well versed on the issue of the review, it may also be beneficial to have a second reviewer who is not an expert in the field(14).
The fact that a scientific article has undergone a peer review does not necessarily guarantee the quality of the research(15). The authors' scientific competence and the prestige of the biomedical journal are important criteria, but not sufficient in themselves to guarantee the credibility of the research(Reference Gérvas and Pérez Fernández16).
Problems relating to replicability and bias can affect two stages of the review process: the final decision regarding study inclusion criteria and quality appraisal(15). A systematic review will minimise these problems and provide appropriate conclusions only if a thorough appraisal is carried out on the primary studies which form the basis of the review. Thus, before deciding to include an article, an appraisal of its methodological rigour must be carried out. A critical appraisal is aimed at discovering if the methods and, as a consequence, the results of a study are valid. This quality assessment should be carried out confidentially and by two independent reviewers. Qualitative and quantitative scales are useful for quality assessments as they will have a positive impact on the reliability of the conclusions. However, these methods introduce a certain inter and intra observer variability, therefore it is necessary to establish the minimum degree of agreement that needs to be reached between the reviewers. The following sources are cited to assist in quality evaluation:
– American Medical Association: JAMAevidence(17).
– CRD's guidance for undertaking reviews in health care(11).
– U.S. Preventive Services Task Force (USPSTF)(18).
– Scottish Intercollegiate Guidelines Network (SIGN)(19).
– EBM-Tools, Center for Evidence-Based Medicine (CEBM)(20).
– Consolidated Standards of Reporting Trials (CONSORT)(21).
– Jadad scoring or the Oxford quality scoring system(Reference Jadad, Moore and Carroll22).
– Evidence Base, National Institute for Health and Clinical Excellence (NICE)(23).
– Canadian Agency for Drugs and Technologies in Health (CADTH)(24).
– Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)(Reference Moher, Liberati and Tetzlaff25).
Data extraction and synthesis of results
The objective of this stage of the review is to compile a narrative summary of the individual studies as this will facilitate the general appraisal. Meta-analysis techniques will be used to combine the quantitative results of the individual studies.
Data extraction is the process through which information is obtained from the primary studies which have been selected from the bibliographic search(26). Data will be collected using an appropriate form which will have been previously included in the protocol. Subsequently, the data retrieved will be transferred to the evidence or review tables. Specifically, the information relating to the acronym PICO (Population, Intervention, Comparison and Outcomes) will be included, as well as the time variable.
On occasions, the lack of relevant information may make it necessary to contact the corresponding author of the retrieved articles, and if necessary request further information.
The descriptive presentation of the results should begin with a statement on the number of accepted articles, indicating those that were included or excluded and the rationale behind the decision. A flow chart of the entire process may assist in explaining this section with greater clarity.
The non quantitative synthesis of the results is set out in the review table to help readers to compare the evidence gathered on the subject. The accuracy and the synopsis of the information in the tables is one of the main steps involved in a systematic review.
Furthermore, together with the PICO variables, the variables related to design and methodological quality are also important in assessing the consistency of the articles included in the review.
Conclusions and recommendations
A systematic review is a synthesis of the best available evidence on a specific scientific question obtained using a methodology which minimises the margin of error. It could be said that the results speak for themselves.
Nevertheless, the final product is not a mere summary of what is known about an intervention, it also highlights the areas that require further research(10). In the discussion section, all reviews should deal with any important methodological limitations of the studies included and of the methods used in the review that may affect any decisions that need to be taken, or future research.
The conclusions are only justifiable when the process of data collection, analysis and integration is done thoroughly and systematically. The efforts involved in conducting a systematic review can be beneficial in identifying knowledge gaps in the subject and suggesting recommendations for future initiatives(15) based exclusively on the knowledge reviewed. A common error is to reach conclusions which go beyond the evidence analysed.
It is however appropriate to present the primary results of the review and its implications for practice, as well as to consider and formulate new questions that may arise from the revised research as regards application to different population groups and new interventions, for example.
Systematic reviews provide a good evaluation of the available evidence, but do not offer any guarantees. The need for prior knowledge, common sense and clinical judgement will always be necessary to interpret the results of this review.
Applying the methodology to a review of Omega-3 Fatty Acids in health and illness
For the production of this journal supplement, a systematic technique was proposed as the methodology for the review of scientific literature on Omega-3 Fatty Acids related to health or illness.
Data sources:
The data used in the different studies of this journal supplement were obtained by consulting and accessing various online health science databases. Consequently, the following sources can be recommended:
– Medlars Online International Literature (MEDLINE), via PubMed
– EMBASE
– Web of Knowledge, Institute for Scientific Information (ISI)
– The Cochrane Library Plus
– International Pharmaceutical Abstracts (IPA)
– Psychology Information (PsycINFO)
– Cumulative Index to Nursing and Allied Health Literature (CINAHL)
– Biological Abstract
– Food Science and Technology Abstracts (FSTA)
– Latin American and Caribbean Heath Sciences Literature (LILACS)
– Pan American Health Organization Library (PAHO)
– World Health Organization Library Information System (WHOLIS)
Data processing
The articles studied were those published in any country or by any institution or individual researcher, and also those published since the beginning of the indexing of each source used.
The differences regarding the selection of primary articles, due to the language of publication, the study design, and the methodological study are noted in each of the reviews conducted.
For document retrieval, the Medical Subject Headings (MeSH) developed by the U.S National Library of Medicine (NLM) were used wherever possible.
Likewise, subheadings were also used, whenever the formulated search strategy deemed it appropriate.
By studying the NLM Thesaurus, the appropriate MeSH (or Major) were obtained for each of the reviews. Similarly, boundaries, such as type of articles, species, and ages, were used.
Tags were included in the search strategy when there was no suitable MeSH available, or when they enabled a more exhaustive search formula.
Search formulae were initially developed for use in the MEDLINE database, via PubMed, by using Boolean connectors and subsequently adapting them to the aforementioned databases. In any case, these formulae can be systematically reproduced at any time in the respective databases.
Data collection
The format for data collection from the primary studies was agreed upon in the protocol and was designed to facilitate the selection of the necessary information. These tasks can be carried out by specific software programmes that are used in systematic reviews as they permit the organisation of the articles and the data that needs to be handled. One of these tools is the RevMan (Review Manager)(27) software used to prepare and maintain Cochrane reviews. It enables the following: preparation of the manuscript text; construction of the tables that show the characteristics of the studies; performance of a meta-analysis if appropriate and the graphical presentation of the results. Together with RevMan, Archie (the Cochrane Collaboration's central server for managing documents and contact details) forms the Cochrane Information Management System (IMS), which is designed to assist contributors to the Cochrane Collaboration.
Another computer application designed for collaborative online work is the Systematic Reviews Collaborative Tool (SysCollab)(28), specifically developed for this journal supplement, although soon a publicly available version will allow the simultaneous participation of the researchers involved in all stages of the review. It will enable the management and selection of the information from the articles, the establishment of the protocol, the generation of the forms to analyse the quality of the documents and real time and deferred communication between participants.
Furthermore, the text of the review can be drawn up as a collaborative effort, along with the generation of the tables and graphics.
Although consistent with the structure proposed by Cochrane, this software also incorporates the evidence based criteria of other medicine organisations. Regardless of the format or software used, the synthesis of the data in the review tables should include, in addition to the PICO acronym, all other variables that enable the evidence gathered from the primary studies to be compared.
Selection of articles
The final selection of the articles was made according to the inclusion or exclusion criteria defined in each of the reviews, although in all cases, the following issues were taken into consideration:
● Typology. The type of chosen document is indicated in each of the reviews.
● Study design. Selection depended on the characteristics of the question posed by each of the reviews, and those which included some degree of recommendation based on the levels of scientific evidence set out by the US Agency for Health Research and Quality(18), were included.
● Quality. Articles were excluded if they did not meet the minimum requirements as regards quality of design, in their Spanish version, in accordance with to the Critical Appraisal Skills Programme, (CASPe)(29).
● Control of bias. Studies which could not guarantee the control of bias (for example, co-morbidities that could distort the outcome of the study).
● When ad hoc criteria were used for certain reviews, they were included in the methodology and the possible limitations of ad hoc criteria were also mentioned in the discussion.
Furthermore, as a secondary search to guarantee exhaustiveness and reduce possible publication biases, the bibliographic list of the articles selected in the primary search was examined. The aim was to identify studies that were not detected in the electronic review. Likewise, the available “grey literature” was identified in the studies where possible.
A flow chart was recommended to clearly explain the rationale for the inclusion or exclusion of the articles.
Acknowledgements
Both authors have contributed equally to the development of this work, in accordance with the Vancouver Norms. We certify that there is no conflict of interest. This article received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
