Eight problems with literature reviews and how to fix them

Traditional approaches to reviewing literature may be susceptible to bias and result in incorrect decisions. This is of particular concern when reviews address policy- and practice-relevant questions. Systematic reviews have been introduced as a more rigorous approach to synthesizing evidence across studies; they rely on a suite of evidence-based methods aimed at maximizing rigour and minimizing susceptibility to bias. Despite the increasing popularity of systematic reviews in the environmental field, evidence synthesis methods continue to be poorly applied in practice, resulting in the publication of syntheses that are highly susceptible to bias. Recognizing the constraints that researchers can sometimes feel when attempting to plan, conduct and publish rigorous and comprehensive evidence syntheses, we aim here to identify major pitfalls in the conduct and reporting of systematic reviews, making use of recent examples from across the field. Adopting a ‘critical friend’ role in supporting would-be systematic reviews and avoiding individual responses to police use of the ‘systematic review’ label, we go on to identify methodological solutions to mitigate these pitfalls. We then highlight existing support available to avoid these issues and call on the entire community, including systematic review specialists, to work towards better evidence syntheses for better evidence and better decisions.

This is a preview of subscription content, access via your institution

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

cancel any time

Subscribe to this journal

Receive 12 digital issues and online access to articles

133,45 € per year

only 11,12 € per issue

Buy this article

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

Challenges and recommendations on the conduct of systematic reviews of observational epidemiologic studies in environmental and occupational health

Article 15 May 2020

Insights from a cross-sector review on how to conceptualise the quality of use of research evidence

Article Open access 09 June 2021

The past, present and future of Registered Reports

Article 15 November 2021

Change history

References

  1. Grant, M. J. & Booth, A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr. J.26, 91–108 (2009). PubMedGoogle Scholar
  2. Haddaway, N. R. & Macura, B. The role of reporting standards in producing robust literature reviews. Nat. Clim. Change8, 444–447 (2018). Google Scholar
  3. Pullin, A. S. & Knight, T. M. Science informing policy–a health warning for the environment. Environ. Evid.1, 15 (2012). Google Scholar
  4. Haddaway, N., Woodcock, P., Macura, B. & Collins, A. Making literature reviews more reliable through application of lessons from systematic reviews. Conserv. Biol.29, 1596–1605 (2015). CASPubMedGoogle Scholar
  5. Pullin, A., Frampton, G., Livoreil, B. & Petrokofsky, G. Guidelines and Standards for Evidence Synthesis in Environmental Management (Collaboration for Environmental Evidence, 2018).
  6. White, H. The twenty-first century experimenting society: the four waves of the evidence revolution. Palgrave Commun.5, 47 (2019). Google Scholar
  7. O’Leary, B. C. et al. The reliability of evidence review methodology in environmental science and conservation. Environ. Sci. Policy64, 75–82 (2016). Google Scholar
  8. Woodcock, P., Pullin, A. S. & Kaiser, M. J. Evaluating and improving the reliability of evidence syntheses in conservation and environmental science: a methodology. Biol. Conserv.176, 54–62 (2014). Google Scholar
  9. Campbell Systematic Reviews: Policies and Guidelines (Campbell Collaboration, 2014).
  10. Higgins, J. P. et al. Cochrane Handbook for Systematic Reviews of Interventions (John Wiley & Sons, 2019).
  11. Shea, B. J. et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ358, j4008 (2017). PubMedPubMed CentralGoogle Scholar
  12. Haddaway, N. R., Land, M. & Macura, B. “A little learning is a dangerous thing”: a call for better understanding of the term ‘systematic review’. Environ. Int.99, 356–360 (2017). PubMedGoogle Scholar
  13. Freeman, R. E. Strategic Management: A Stakeholder Approach (Cambridge Univ. Press, 2010).
  14. Haddaway, N. R. et al. A framework for stakeholder engagement during systematic reviews and maps in environmental management. Environ. Evid.6, 11 (2017). Google Scholar
  15. Land, M., Macura, B., Bernes, C. & Johansson, S. A five-step approach for stakeholder engagement in prioritisation and planning of environmental evidence syntheses. Environ. Evid.6, 25 (2017). Google Scholar
  16. Oliver, S. & Dickson, K. Policy-relevant systematic reviews to strengthen health systems: models and mechanisms to support their production. Evid. Policy12, 235–259 (2016). Google Scholar
  17. Savilaakso, S. et al. Systematic review of effects on biodiversity from oil palm production. Environ. Evid.3, 4 (2014). Google Scholar
  18. Savilaakso, S., Laumonier, Y., Guariguata, M. R. & Nasi, R. Does production of oil palm, soybean, or jatropha change biodiversity and ecosystem functions in tropical forests. Environ. Evid.2, 17 (2013). Google Scholar
  19. Haddaway, N. R. & Crowe, S. Experiences and lessons in stakeholder engagement in environmental evidence synthesis: a truly special series. Environ. Evid.7, 11 (2018). Google Scholar
  20. Sánchez-Bayo, F. & Wyckhuys, K. A. Worldwide decline of the entomofauna: a review of its drivers. Biol. Conserv.232, 8–27 (2019). Google Scholar
  21. Agarwala, M. & Ginsberg, J. R. Untangling outcomes of de jure and de facto community-based management of natural resources. Conserv. Biol.31, 1232–1246 (2017). PubMedGoogle Scholar
  22. Gurevitch, J., Curtis, P. S. & Jones, M. H. Meta-analysis in ecology. Adv. Ecol. Res.32, 199–247 (2001). CASGoogle Scholar
  23. Haddaway, N. R., Macura, B., Whaley, P. & Pullin, A. S. ROSES RepOrting standards for Systematic Evidence Syntheses: pro forma, flow-diagram and descriptive summary of the plan and conduct of environmental systematic reviews and systematic maps. Environ. Evid.7, 7 (2018). Google Scholar
  24. Lwasa, S. et al. A meta-analysis of urban and peri-urban agriculture and forestry in mediating climate change. Curr. Opin. Environ. Sustain.13, 68–73 (2015). Google Scholar
  25. Pacifici, M. et al. Species’ traits influenced their response to recent climate change. Nat. Clim. Change7, 205–208 (2017). Google Scholar
  26. Owen-Smith, N. Ramifying effects of the risk of predation on African multi-predator, multi-prey large-mammal assemblages and the conservation implications. Biol. Conserv.232, 51–58 (2019). Google Scholar
  27. Prugh, L. R. et al. Designing studies of predation risk for improved inference in carnivore-ungulate systems. Biol. Conserv.232, 194–207 (2019). Google Scholar
  28. Li, Y. et al. Effects of biochar application in forest ecosystems on soil properties and greenhouse gas emissions: a review. J. Soil Sediment.18, 546–563 (2018). CASGoogle Scholar
  29. Moher, D., Liberati, A., Tetzlaff, J. & Altman, D. G., The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med.6, e1000097 (2009). PubMedPubMed CentralGoogle Scholar
  30. Bernes, C. et al. What is the influence of a reduction of planktivorous and benthivorous fish on water quality in temperate eutrophic lakes? A systematic review. Environ. Evid.4, 7 (2015). Google Scholar
  31. McDonagh, M., Peterson, K., Raina, P., Chang, S. & Shekelle, P. Avoiding bias in selecting studies. Methods Guide for Effectiveness and Comparative Effectiveness Reviews [Internet] (Agency for Healthcare Research and Quality, 2013).
  32. Burivalova, Z., Hua, F., Koh, L. P., Garcia, C. & Putz, F. A critical comparison of conventional, certified, and community management of tropical forests for timber in terms of environmental, economic, and social variables. Conserv. Lett.10, 4–14 (2017). Google Scholar
  33. Min-Venditti, A. A., Moore, G. W. & Fleischman, F. What policies improve forest cover? A systematic review of research from Mesoamerica. Glob. Environ. Change47, 21–27 (2017). Google Scholar
  34. Bramer, W. M., Giustini, D. & Kramer, B. M. R. Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study. Syst. Rev.5, 39 (2016). PubMedPubMed CentralGoogle Scholar
  35. Bramer, W. M., Giustini, D., Kramer, B. M. R. & Anderson, P. F. The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews. Syst. Rev.2, 115 (2013). PubMedPubMed CentralGoogle Scholar
  36. Gusenbauer, M. & Haddaway, N. R. Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Res. Synth. Methods11, 181–217 (2020). PubMedPubMed CentralGoogle Scholar
  37. Livoreil, B. et al. Systematic searching for environmental evidence using multiple tools and sources. Environ. Evid.6, 23 (2017). Google Scholar
  38. Mlinarić, A., Horvat, M. & Šupak Smolčić, V. Dealing with the positive publication bias: why you should really publish your negative results. Biochem. Med.27, 447–452 (2017). Google Scholar
  39. Lin, L. & Chu, H. Quantifying publication bias in meta‐analysis. Biometrics74, 785–794 (2018). PubMedGoogle Scholar
  40. Haddaway, N. R. & Bayliss, H. R. Shades of grey: two forms of grey literature important for reviews in conservation. Biol. Conserv.191, 827–829 (2015). Google Scholar
  41. Viechtbauer, W. Conducting meta-analyses in R with the metafor package. J. Stat. Softw.36, 1–48 (2010). Google Scholar
  42. Bilotta, G. S., Milner, A. M. & Boyd, I. On the use of systematic reviews to inform environmental policies. Environ. Sci. Policy42, 67–77 (2014). Google Scholar
  43. Englund, G., Sarnelle, O. & Cooper, S. D. The importance of data‐selection criteria: meta‐analyses of stream predation experiments. Ecology80, 1132–1141 (1999). Google Scholar
  44. Burivalova, Z., Şekercioğlu, Ç. H. & Koh, L. P. Thresholds of logging intensity to maintain tropical forest biodiversity. Curr. Biol.24, 1893–1898 (2014). CASPubMedGoogle Scholar
  45. Bicknell, J. E., Struebig, M. J., Edwards, D. P. & Davies, Z. G. Improved timber harvest techniques maintain biodiversity in tropical forests. Curr. Biol.24, R1119–R1120 (2014). CASPubMedGoogle Scholar
  46. Damette, O. & Delacote, P. Unsustainable timber harvesting, deforestation and the role of certification. Ecol. Econ.70, 1211–1219 (2011). Google Scholar
  47. Blomley, T. et al. Seeing the wood for the trees: an assessment of the impact of participatory forest management on forest condition in Tanzania. Oryx42, 380–391 (2008). Google Scholar
  48. Haddaway, N. R. et al. How does tillage intensity affect soil organic carbon? A systematic review. Environ. Evid.6, 30 (2017). Google Scholar
  49. Higgins, J. P. et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ343, d5928 (2011). PubMedPubMed CentralGoogle Scholar
  50. Stewart, G. Meta-analysis in applied ecology. Biol. Lett.6, 78–81 (2010). PubMedGoogle Scholar
  51. Koricheva, J. & Gurevitch, J. Uses and misuses of meta‐analysis in plant ecology. J. Ecol.102, 828–844 (2014). Google Scholar
  52. Vetter, D., Ruecker, G. & Storch, I. Meta‐analysis: a need for well‐defined usage in ecology and conservation biology. Ecosphere4, 1–24 (2013). Google Scholar
  53. Stewart, G. B. & Schmid, C. H. Lessons from meta-analysis in ecology and evolution: the need for trans-disciplinary evidence synthesis methodologies. Res. Synth. Methods6, 109–110 (2015). PubMedGoogle Scholar
  54. Macura, B. et al. Systematic reviews of qualitative evidence for environmental policy and management: an overview of different methodological options. Environ. Evid.8, 24 (2019). Google Scholar
  55. Koricheva, J. & Gurevitch, J. in Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J. et al.) Ch. 1 (Princeton Scholarship Online, 2013).
  56. Britt, M., Haworth, S. E., Johnson, J. B., Martchenko, D. & Shafer, A. B. The importance of non-academic coauthors in bridging the conservation genetics gap. Biol. Conserv.218, 118–123 (2018). Google Scholar
  57. Graham, L., Gaulton, R., Gerard, F. & Staley, J. T. The influence of hedgerow structural condition on wildlife habitat provision in farmed landscapes. Biol. Conserv.220, 122–131 (2018). Google Scholar
  58. Delaquis, E., de Haan, S. & Wyckhuys, K. A. On-farm diversity offsets environmental pressures in tropical agro-ecosystems: a synthetic review for cassava-based systems. Agric. Ecosyst. Environ.251, 226–235 (2018). Google Scholar
  59. Popay, J. et al. Guidance on the Conduct of Narrative Synthesis in Systematic Reviews: A Product from the ESRC Methods Programme Version 1 (Lancaster Univ., 2006).
  60. Pullin, A. S. et al. Human well-being impacts of terrestrial protected areas. Environ. Evid.2, 19 (2013). Google Scholar
  61. Waffenschmidt, S., Knelangen, M., Sieben, W., Bühn, S. & Pieper, D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med. Res. Methodol.19, 132 (2019). PubMedPubMed CentralGoogle Scholar
  62. Rallo, A. & García-Arberas, L. Differences in abiotic water conditions between fluvial reaches and crayfish fauna in some northern rivers of the Iberian Peninsula. Aquat. Living Resour.15, 119–128 (2002). Google Scholar
  63. Glasziou, P. & Chalmers, I. Research waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers. BMJ363, k4645 (2018). Google Scholar
  64. Haddaway, N. R. Open Synthesis: on the need for evidence synthesis to embrace Open Science. Environ. Evid.7, 26 (2018). Google Scholar

Acknowledgements

We thank C. Shortall from Rothamstead Research for useful discussions on the topic.

Author information

Authors and Affiliations

  1. Mercator Research Institute on Climate Change and Global Commons, Berlin, Germany Neal R. Haddaway
  2. Stockholm Environment Institute, Stockholm, Sweden Neal R. Haddaway & Biljana Macura
  3. Africa Centre for Evidence, University of Johannesburg, Johannesburg, South Africa Neal R. Haddaway
  4. College of Medicine and Health, Exeter University, Exeter, UK Alison Bethel
  5. Department of Zoology, University of Cambridge, Cambridge, UK Lynn V. Dicks
  6. School of Biological Sciences, University of East Anglia, Norwich, UK Lynn V. Dicks
  7. Department of Biological Sciences, Royal Holloway University of London, Egham, UK Julia Koricheva
  8. Department of Zoology, University of Oxford, Oxford, UK Gillian Petrokofsky
  9. Collaboration for Environmental Evidence, UK Centre, School of Natural Sciences, Bangor University, Bangor, UK Andrew S. Pullin
  10. Liljus ltd, London, UK Sini Savilaakso
  11. Department of Forest Sciences, University of Helsinki, Helsinki, Finland Sini Savilaakso
  12. Evidence Synthesis Lab, School of Natural and Environmental Sciences, University of Newcastle, Newcastle-upon-Tyne, UK Gavin B. Stewart
  1. Neal R. Haddaway