We’ve previously discussed how we evaluate a single study. For the questions we try to answer, though, it’s rarely sufficient to consult a single study; studies are specific to a particular time, place, and context, and to get a robust answer to a question like “Do insecticide-treated nets reduce child mortality?” one should conduct – or ideally, find – a thorough and unbiased survey of the available research. Doing so is important: we feel it is easy (and common) to form an inaccurate view based on a problematic survey of research.

This post discusses what we feel makes for a good literature review: a report that surveys the available research on a particular question. Our preferred way to answer a research question is to find an existing literature review with strong answers to these questions; when necessary, we conduct our own literature review with the same questions in mind.

Our key questions for a literature review

• What are the motivations of the literature reviewer? A biased survey of research can easily lead to a biased conclusion, if the reviewer is selective about which studies to include and which to focus on. We are generally highly wary of literature reviews commissioned by charities (for example, a 2005 survey of studies on microfinance commissioned by the Grameen Foundation) or advocacy groups. We prefer reviews that are done by parties with no obvious stake in coming to one recommendation or another, and with a stake in maintaining a reputation for neutrality (these can, in appropriate cases, include government agencies as well as independent groups such as the Cochrane Collaboration).
• How did the literature reviewer choose which studies to include? Since one of the ways a literature review can be distorted is through selective inclusion of studies, we take interest in the question of whether it has included all (and only) sufficiently high-quality studies that bear on the question of interest.In some cases, there are only a few high-quality studies available on the question of interest, such that the reviewer can discuss each study individually, and the reader can hold the reviewer accountable if s/he knows of another high-quality study that has been left out. However, for a topic like the impact of insecticide-treated nets on malaria, there may be many high-quality studies available. In these cases, we prefer literature reviews in which the reviewer is clear about his/her search protocol, ideally such that the search could be replicated by a reader.
• How thoroughly and consistently does the literature review discuss the strengths and weaknesses of each study? As we wrote previously, studies can vary a great deal in quality and importance. When we see a literature review simply asserting that a particular study supports a particular claim – without discussing the strengths and weaknesses of this study – we consider it a low-quality literature review and do not put weight on it. In our view, a good literature review is one that provides a maximally thorough, consistent, understandable summary of the strengths and weaknesses of each study it includes.
• Does the literature review include meta-analysis, attempting to quantitatively combine the results of several studies? In some cases it is possible to perform meta-analysis: combining the results from multiple studies to get a single “pooled” quantitative result. In other cases a literature review limits itself to summarizing the strengths and weaknesses of each study reviewed and giving a qualitative conclusion.

Strong and weak literature reviewsIn general, we feel that the Cochrane Collaboration performs strong literature reviews by the criteria above. Examples of its reviews include a review we discussed previously on deworming and a review on insecticide-treated nets to protect against malaria.

• The Cochrane Collaboration is an independent group that aims to base its brand on unbiased research, and does not take commercial funding.
• Cochrane reviews generally explicitly lay out their search strategy and selection criteria in their summaries.
• Cochrane reviews generally list all of the studies considered along with relatively in-depth discussions of their methodology, strengths and weaknesses (full text is required to see these).
• Cochrane reviews generally perform quantitative meta-analysis and include the conclusions of such analysis in their summaries.

An example of a more problematic literature review is King, Dickman and Tisch 2005, cited in our report on deworming. This review does well on some of our criteria: it is clear about its search and inclusion criteria (see Figure 1 on page 1562), and it performs quantified meta-analysis (see Table 1 on page 1565). However,

• It provides a list of all studies included, but unlike the Cochrane reviews we’ve seen, it does not provide any information for these studies (methodology, sample size, etc.) other than the reference.
• It does not discuss individual studies’ strengths and weaknesses at all.
• It does not make it possible for the reader to connect the study’s conclusions (in Table 5) to specific studies. (Figures 2-4 break down a few, but not all, of the study’s conclusions with lists of individual studies.) Since over 100 studies were included, we do not see a practical way for a reader to vet the literature review’s conclusions.
• There is also ambiguity in what the reported conclusions mean: for example, Table 5 does not specify whether it is examining the impact of deworming on the level or change of each listed outcome (i.e., impact on weight vs. impact on change in weight over time).

We have at times seen advocacy groups and/or foundations put out literature reviews that are far more flawed than the study discussed above. Though we generally don’t keep track of these, we provide one example, a paper entitled “What can we learn from playing interactive games?” A representative quote from this paper:

There is also evidence that game playing can improve cognitive processing skills such as visual discernment, which involves the ability to divide visual attention and allocate it to two or more simultaneous events (Greenfield et al., 1994b); parallel processing, the ability to engage in multiple cognitive tasks simultaneously (Gunter, 1998); and other forms of visual discrimi-nation including the ability to process cluttered visual scenes and rapid sequences of images (Riesenhuber, 2004). Experiments have also found improvements in eye-hand coordination after playing video games (Rosenberg et al., 2005).

The paper does not discuss selection, inclusion, strengths, or weaknesses of studies, or even their basic design and the nature/magnitude of their findings (for example, how is “parallel processing” measured?)

All else equal, we would prefer a world in which all literature reviews were more like Cochrane reviews than like the more problematic reviews discussed above. However, it’s worth noting that Cochrane reviews appear to be quite expensive, upwards of $100,000 each. Conducting a truly thorough and unbiased literature review is not necessarily easy or cheap, but we feel it is often necessary to get an accurate picture of what the research says on a given question.