---
title: "Systematic review"
chunk: 5/6
source: "https://en.wikipedia.org/wiki/Systematic_review"
category: "reference"
tags: "science, encyclopedia"
date_saved: "2026-05-05T04:26:30.659047+00:00"
instance: "kb-cron"
---

=== Out-dated or risk of bias ===
While systematic reviews are regarded as the strongest form of evidence, a 2003 review of 300 studies found that not all systematic reviews were equally reliable, and that their reporting can be improved by a universally agreed upon set of standards and guidelines. A further study by the same group found that of 100 systematic reviews monitored, 7% needed updating at the time of publication, another 4% within a year, and another 11% within 2 years; this figure was higher in rapidly changing fields of medicine, especially cardiovascular medicine. A 2003 study suggested that extending searches beyond major databases, perhaps into grey literature, would increase the effectiveness of reviews. A 2005 analysis of 377 Cochrane systematic reviews of healthcare interventions found that, after four years, 9% of the updated reviews resulted in a changed conclusion compared to the original versions. Among Cochrane systematic reviews on medical interventions published in 2010 and updated by 2017, very few (4%) of the updated reviews reported a change in conclusion for a primary outcome.

Some authors have highlighted problems with systematic reviews, particularly those conducted by Cochrane, noting that published reviews are often biased, out of date, and excessively long. Cochrane reviews have been criticized as not being sufficiently critical in the selection of trials and including too many of low quality. They proposed several solutions, including limiting studies in meta-analyses and reviews to registered clinical trials, requiring that original data be made available for statistical checking, paying greater attention to sample size estimates, and eliminating dependence on only published data. Some of these difficulties were noted as early as 1994:much poor research arises because researchers feel compelled for career reasons to carry out research that they are ill-equipped to perform, and nobody stops them.
Methodological limitations of meta-analysis have also been noted. Another concern is that the methods used to conduct a systematic review are sometimes changed once researchers see the available trials they are going to include. Some websites have described retractions of systematic reviews and published reports of studies included in published systematic reviews. Eligibility criteria that is arbitrary may affect the perceived quality of the review.

=== Limited reporting of data from human studies ===
The AllTrials campaign report that around half of clinical trials have never reported results and works to improve reporting. 'Positive' trials were twice as likely to be published as those with 'negative' results.
As of 2016, it is legal for-profit companies to conduct clinical trials and not publish the results. For example, in the past 10 years, 8.7 million patients have taken part in trials that have not published results. These factors mean that it is likely there is a significant publication bias, with only 'positive' or perceived favourable results being published. A recent systematic review of industry sponsorship and research outcomes concluded that "sponsorship of drug and device studies by the manufacturing company leads to more favorable efficacy results and conclusions than sponsorship by other sources" and that the existence of an industry bias that cannot be explained by standard 'risk of bias' assessments.

=== Poor compliance with review reporting guidelines ===
The rapid growth of systematic reviews in recent years has been accompanied by the attendant issue of poor compliance with guidelines, particularly in areas such as declaration of registered study protocols, funding source declaration, risk of bias data, issues resulting from data abstraction, and description of clear study objectives. A host of studies have identified weaknesses in the rigour and reproducibility of search strategies in systematic reviews. To remedy this issue, a new PRISMA guideline extension called PRISMA-S is being developed. Furthermore, tools and checklists for peer-reviewing search strategies have been created, such as the Peer Review of Electronic Search Strategies (PRESS) guidelines.
A key challenge for using systematic reviews in clinical practice and healthcare policy is assessing the quality of a given review. Consequently, a range of appraisal tools to evaluate systematic reviews have been designed. The two most popular measurement instruments and scoring tools for systematic review quality assessment are AMSTAR 2 (a measurement tool to assess the methodological quality of systematic reviews) and ROBIS (Risk Of Bias In Systematic reviews); however, these are not appropriate for all systematic review types. Some recent peer-reviewed articles have carried out comparisons between AMSTAR 2 and ROBIS tools.

== History ==
The first publication that is now recognized as equivalent to a modern systematic review was a 1753 paper by James Lind, which reviewed all of the previous publications about scurvy.  Systematic reviews appeared only sporadically until the 1980s, and became common after 2000.  More than 10,000 systematic reviews are published each year.

=== History in medicine ===
A 1904 British Medical Journal paper by Karl Pearson collated data from several studies in the UK, India and South Africa of typhoid inoculation. He used a meta-analytic approach to aggregate the outcomes of multiple clinical studies. In 1972, Archie Cochrane wrote: "It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials". Critical appraisal and synthesis of research findings in a systematic way emerged in 1975 under the term 'meta analysis'. Early syntheses were conducted in broad areas of public policy and social interventions, with systematic research synthesis applied to medicine and health. Inspired by his own personal experiences as a senior medical officer in prisoner of war camps, Archie Cochrane worked to improve the scientific method in medical evidence. His call for the increased use of randomised controlled trials and systematic reviews led to the creation of The Cochrane Collaboration, which was founded in 1993 and named after him, building on the work by Iain Chalmers and colleagues in the area of pregnancy and childbirth.

== See also ==
Critical appraisal
Further research is needed
Systematic searching
Horizon scanning
Literature review
Living review
Meta-analysis
Metascience
Peer review
Review journal
Generalized model aggregation (GMA)
Umbrella review