Skip to main content Accessibility help
×
Home

DOES SIZE MATTER? A CRITICAL REVIEW OF META-ANALYSIS IN AGRONOMY

  • TIMOTHY J. KRUPNIK (a1), JENS A. ANDERSSON (a2), LEONARD RUSINAMHODZI (a3), MARC CORBEELS (a3) (a4), CAROL SHENNAN (a5) and BRUNO GÉRARD (a6)...

Summary

Intended to test broad hypotheses and arrive at unifying conclusions, meta-analysis is the process of extracting, assembling, and analyzing large quantities of data from multiple publications to increase statistical power and uncover explanatory patterns. This paper describes the ways in which meta-analysis has been applied to support claims and counter-claims regarding two topics widely debated in agricultural research, namely organic agriculture (OA) and conservation agriculture (CA). We describe the origins of debate for each topic and assess prominent meta-analyses considering data-selection criteria, research question framing, and the interpretation and extrapolation of meta-analytical results. Meta-analyses of OA and CA are also examined in the context of the political economy of development-oriented agricultural research. Does size matter? We suggest that it does, although somewhat ironically. While meta-analysis aims to pool all relevant studies and generate comprehensive databases from which broad insights can be drawn, our case studies suggest that the organization of many meta-analyses may affect the generalizability and usefulness of research results. The politicized nature of debates over OA and CA also appear to affect the divergent ways in which meta-analytical results may be interpreted and extrapolated in struggles over the legitimacy of both practices. Rather than resolving scientific contestation, these factors appear to contribute to the ongoing debate. Meta-analysis is nonetheless becoming increasingly popular with agricultural researchers attracted by the power for the statistical inference offered by large datasets. This paper consequently offers three suggestions for how scientists and readers of scientific literature can more carefully evaluate meta-analyses. First, the ways in which papers and data are collected should be critically assessed. Second, the justification of research questions, framing of farming systems, and the scales at which research results are extrapolated and discussed should be carefully evaluated. Third, when applied to strongly politicized topics situated in an arena of scientific debate, as is the case with OA and CA, more conservative interpretations of meta-analytical results that recognize the socially and politically embedded nature of agricultural research is are needed.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      DOES SIZE MATTER? A CRITICAL REVIEW OF META-ANALYSIS IN AGRONOMY
      Available formats
      ×

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      DOES SIZE MATTER? A CRITICAL REVIEW OF META-ANALYSIS IN AGRONOMY
      Available formats
      ×

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      DOES SIZE MATTER? A CRITICAL REVIEW OF META-ANALYSIS IN AGRONOMY
      Available formats
      ×

Copyright

This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.

Corresponding author

§§Corresponding author. Email: t.krupnik@cgiar.org

References

Hide All
Andersson, J. A. (2014). Comment on “Evaluating conservation agriculture for small-scale farmers in sub-Saharan Africa and South Asia”. Agriculture, Ecosystems and Environment 196:2123.
Andersson, J. A. and D'Souza, S. (2014). From adoption claims to understanding farmers and contexts: A literature review of Conservation Agriculture (CA) adoption among smallholder farmers in southern Africa. Agriculture, Ecosystems and Environment 187:116132.
Andersson, J. A. and Sumberg, J. (2015). Knowledge politics in development-oriented agronomy. In Agronomy for Development: The Politics of Knowledge in Agricultural Research, 113 (Ed , J. Sumberg, ). London: Routledge.
Avery, A. (2007). ‘Organic abundance’ report: Fatally flawed. Renewable Agriculture and Food Systems 22:321329.
Badgley, C., Moghtader, J., Quintero, E., Zakem, E. and Chappell, M. J. (2007a). Organic agriculture and the global food supply. Renewable Agriculture and Food Systems 22:86108.
Badgley, C. and Perfecto, I. (2007). Can organic agriculture feed the world? Renewable Agriculture and Food Systems 22:8085.
Badgley, C., Perfecto, I., Chappell, M. J. and Samulon, A. (2007b). Strengthening the case for organic agriculture: Response to Alex Avery. Renewable Agriculture and Food Systems 22:323327.
Bardwell, L. V. (1991). Problem-Framing: A perspective on environmental problem-solving. Environmental Management 5:603612.
Beus, C. E. and Dunlap, R. E. (2010). Conventional versus alternative agriculture: The paradigmatic roots of the debate. Rural Sociology 55:590616.
Borenstein, M., Hedges, L. V., Higgens, J. P. T. and Rothstein, H. R. (2009). Introduction to Meta-Analysis. Hoboken: Wiley.
Brandt, K., Srednicka-Tober, D., Baranski, M., Sanderson, R., Leifert, C. and Seal, C. (2013). Methods for comparing data across differently designed agronomic studies: examples of different meta-analysis methods used to compare relative composition of plant foods grown using organic or conventional production methods and a protocol for a systematic review. Journal of Agricultural and Food Chemistry 61:71737180.
Brouder, S. M. and Gomez-Macpherson, H. (2014). The impact of conservation agriculture on smallholder agricultural yields: A scoping review of the evidence. Agriculture, Ecosystems and Environment 187:1132.
Buffet, H. (2015). You can listen to the academics, or you can listen to the farmers. The Howard G. Buffett Foundation. Online: http://harvestingthepotential.org/brownrevolution/assets/doc_01.pdf. Accessed 4 January, 2019.
Cassman, K. G. (2007). Editorial response by Kenneth Cassman: can organic agriculture feed the world—science to the rescue?. Renewable Agriculture and Food Systems 22:8384.
Cohnstaedt, L. W. and Poland, J. (2017). Review articles: The black-market of scientific currency. Annals of the Entomological Society of America 110:90–90.
Connor, D. J. (2013). Organically grown crops do not a cropping system make and nor can organic agriculture nearly feed the world. Field Crops Research 144:145147.
Connor, D. (2008). Organic agriculture cannot feed the world. Field Crops Research 106:187190.
Corbeels, M., de Graaff, J., Ndah, T. H., Penot, E., Baudron, F., Naudin, K., Andrieu, N., Chirat, G., Schuler, J., Nyagumbo, I., Rusinamhodzi, L., Traore, K., Mzoba, H. D. and Adolwa, I. S. (2014a). Understanding the impact and adoption of conservation agriculture in Africa: A multi-scale analysis. Agriculture, Ecosystems and Environment 187:155170.
Corbeels, M., Sakyi, R. K., Kühne, R. F. and Whitbread, A. (2014b). Meta-analysis of crop responses to conservation agriculture in sub-Saharan Africa. In CCAFS Report No. 12., 20 Copenhagen: CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS).
de Ponti, T., Rijk, B. and van Ittersum, M. K. (2012). The crop yield gap between organic and conventional agriculture. Agricultural Systems 108:19.
de Roo, N., Andersson, J. A. and Krupnik, T. J. (2019). On-farm trials for development impact? the organisation of research and the scaling of agricultural technologies. Experimental Agriculture. 52 (2):163184.
Derpsch, R., Franzluebbers, A. J., Duiker, S. W., Reicosky, D. C., Koeller, K., Friedrich, T., Sturny, W. G., , J. C. M. and Weiss, K. (2014). Why do we need to standardize no-tillage research? Soil and Tillage Research 137:1622.
Dobermann, A. (2012). Getting back to the field. Nature 485:176177.
Doré, T., Makowski, D., Malézieux, E., Munier-Jolan, N. M. T. and Tittonell, P. (2011). Facing up to the paradigm of ecological intensification in agronomy: Revisiting methods, concepts and knowledge. European Journal of Agronomy 34:197210.
Eisler, I. (1990). Meta-analysis: Magic wand or exploratory tool? comment on Markus et al. Journal of Family Therapy 12:223228.
Ekboir, J. (2001). Developing no-till packages for small-scale farmers. In 2001 CIMMYT World Wheat Overview and Outlook. International Maize and Wheat Improvement Center. Mexico, DF, pp. 1–37.
Elsevier (2009). Research output in developing countries reveals 194% increase in five years. Online: https://www.elsevier.com/about/press-releases/corporate-social-responsibility/research-output-in-developing-countries-reveals-194-increase-in-five-years. Accessed 4 January 2019.
Englund, G., Sarnelle, O. and Cooper, S. D. (1999). The importance of data-selection criteria: Meta-analyses of stream predation experiments. Ecology 80:11321141.
Erenstein, O. (2002). Crop residue mulching in tropical and semi-tropical countries: An evaluation of residue availability and other technological implications. Soil Tillage Research 67:115133.
Erenstein, O. and Laxmi, V. (2008). Zero tillage impacts in India's rice-wheat systems: A review. Soil Tillage Research 100:114.
Eysenck, H. J. (1978). An exercise in mega-silliness. American Psychologist 33:517.
FAO (2017). Conservation agriculture. Online: http://www.fao.org/ag/ca/. Accessed 4 January 2019.
Fisher, M. (2015). Moving science through: Meta-analysis. CSA News 60:46.
Fitz-Gibbon, C. T. (1984). Meta-analysis: An explication. British Educational Research Journal 10:135144.
Friedrich, T., Pisante, M., Sturny, W. G., Corsi, S. and Buffet, H. (2015). Comment on Pittelkow et al article by Friedrich et al. Nature. Online: https://www.nature.com/articles/nature13809. Accessed 4 January 2019.
Garg, A. X., Hackam, D. and Tonelli, M. (2008). Systematic review and meta-analysis: when one study is just not enough. Clinical journal of the American Society of Nephrology 3:253260.
Giller, K. E., Andersson, J. A., Corbeels, M., Kirkegaard, J., Mortensen, D., Erenstein, O. and Vanlauwe, B. (2015). Beyond conservation agriculture. Frontiers in Plant Science 6:114.
Giller, K. E., Andersson, J. A., Sumberg, J. and Thompson, J. (2017). A golden age for agronomy? In Agronomy for Development: The Politics of Knowledge in Agricultural Research, 150160 (Ed Sumberg, J.). London: Routledge.
Giller, K. E., Witter, E., Corbeels, M. and Tittonell, P. (2009). Conservation agriculture and smallholder farming in Africa: The heretics’ view. Field Crops Research 114:2334.
Goffman, E. (1974). Frame Analysis: An Essay on the Organization of Experience. New York: Harper & Row.
Gooding, D., Pinch, T. and Schaffer, S. (1989). The Uses of Experiments: Studies in the Natural Sciences. Cambridge: Cambridge University Press.
Goulding, K., Trewavas, A. J. and Giller, K. (2011). Feeding the world- a contribution to the debate. World Agriculture 2:3238.
Gurevitch, J., Koricheva, J., Nakagawa, S. and Stewart, G. (2018). Meta-analysis and the science of research synthesis. Nature 555:175182.
Haidich, A. B. (2010). Meta-analysis in medical research. Hippokratika 14:2937.
Hedges, L. V., Gurevitch, J. and Curtis, P. S. (1999). The meta-analysis of response ratios in experimental ecology. Ecology 80:11501156.
Hobbs, P. R., Sayre, K. and Gupta, R. (2008). The role of conservation agriculture in sustainable agriculture. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 363:543555.
Hossard, L., Archer, D. W., Bertrand, M., Colnenne-David, C., Debaeke, P., Ernfors, M., Jeuffroy, M.-H., Munier-Jolain, N., Nilsson, C., Sanford, G. R., Snapp, S. S., Jensen, E. S. and Makowski, D. (2016). A meta-analysis of maize and wheat yields in low-input vs. conventional and organic systems. Agronomy Journal 108:11551167.
Huang, M., Zhou, X., Cao, F., Xia, B. and Zou, Y. (2015). No-tillage effect on rice yield in China: A meta-analysis. Field Crops Research 183:126137.
International Federation of Organic Agricultural Movements (2015). Definition of organic agriculture. Online: https://www.ifoam.bio/en/organic-landmarks/definition-organic-agriculture. Accessed 4 January 2019.
Ioannidis, J. P. A. (2016). The Mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. The Milbank Quarterly 94:485514.
Keil, A., D'souza, A. and McDonald, A. (2015). Zero-tillage as a pathway for sustainable wheat intensification in the Eastern Indo-Gangetic Plains: Does it work in farmers’ fields?. Food Security 7:9831001.
Khun, N. J. and Hu, Y. (2015). Comment on Pittelkow et al article by Yaxian Hu et al. Nature 2016. Online: https://www.nature.com/articles/nature13809. Accessed 4 January 2019.
Kirchmann, H., Kätterer, T., Bergström, L., Börjesson, G. and Bolinder, M. A. (2016). Flaws and criteria for design and evaluation of comparative organic and conventional cropping systems. Field Crops Research 186:99106.
Knapp, S. and van der Heijden, M. G. A. (2018). A global meta-analysis of yield stability in organic and conservation agriculture. Nature Communications 9:112.
Koricheva, J., Gurevitch, J. and Gómez-Aparicio, L. (2014). Uses and misuses of meta-analysis in plant ecology. Journal of Ecology 102:828844.
Kravchenko, A. N., Snapp, S. S. and Robertson, G. P. (2017). Field-scale experiments reveal persistent yield gaps in low-input and organic cropping systems. Proceedings of the National Academy of Sciences of the United States of America 114:926931.
Krupnik, T. J., Shennan, C., Settle, W. H., Demont, M., Ndiaye, A. B. and Rodenburg, J. (2012). Improving irrigated rice production in the Senegal River valley through experiential learning and innovation. Agricultural Systems 109:101112.
Leeuwis, C., Schut, M. and Klerkx, L. (2017). Systems research in the CGIAR as an arena of struggle: Competing discourses on the embedding of research in development. In The Politics of Knowledge in Agricultural Research, 5978 (Ed Sumberg, J.). London: Routledge.
Leifeld, J. (2016). Current approaches neglect possible agricultural cutback under large-scale organic farming. A comment to Ponisio et al. Proceedings of the Royal Society B: Biological Sciences 283 (1824): 20151623.
Longo, D. L. and Drazen, J. M. (2016). Data sharing. North England Journal of Medicine 374:276277.
Lundy, M. E., Pittelkow, C. M., Linquist, B. A., Liang, X., van Groenigen, K. J., Lee, J., Six, J., Venterea, R. T. and van Kessel, C. (2015). Nitrogen fertilization reduces yield declines following no-till adoption. Field Crops Research. 183:204210.
Maat, H. (2011). The history and future of agricultural experiments. NJAS - Wageningen Journal of Life Sciences. 57:187195.
Magdoff, F. (2007). Ecological agriculture: Principles, practices, and constraints. Renewable Agriculture and Food Systems 22:109117.
Meyfroidt, P., Carlson, K. M., Fagan, M. E., Gutiérrez-Vélez, V. H., Macedo, M. N., Curran, L. M., DeFries, R. S., Dyer, G. A., Gibbs, H. K., Lambin, E. F., Morton, D. C. and Robiglio, V. (2014). Multiple pathways of commodity crop expansion in tropical forest landscapes. Environmental Research Letters 9:074012.
Murtaugh, P. A. (2002). Journal quality, effect size, and publication bias in meta-analysis. Ecology 83:11621166.
Philibert, A., Loyce, C. and Makowski, D. (2012). Assessment of the quality of meta-analysis in agronomy. Agriculture, Ecosystems and Environment 148:7282.
Pittelkow, C. M., Liang, X., Linquist, B. A., van Groenigen, K. J., Lee, J., Lundy, M. E., van Gestel, N., Six, J., Venterea, R. T. and van Kessel, C. (2015a). Productivity limits and potentials of the principles of conservation agriculture. Nature 517:365368.
Pittelkow, C. M., Linquist, B. A., Lundy, M. E., Liang, X., van Groenigen, K. J., Lee, J., van Gestel, N., Six, J., Venterea, R. T. and van Kessel, C. (2015b). When does no-till yield more? A global meta-analysis. Field Crops Research 183:156168.
Ponisio, L. C. and Kremen, C. (2016). System-level approach needed to evaluate the transition to more sustainable agriculture. Proceedings of the Royal Society B: Biological Sciences 283 (1824):20152913.
Ponisio, L. C., M'Gonigle, L. K., Mace, K. C., Palomino, J., de Valpine, P. and Kremen, C. (2015). Diversification practices reduce organic to conventional yield gap. Proceedings of the Royal Society B: Biological Sciences 282:20141396.
Pretty, J. and Hine, R. E. (2001). Reducing food poverty with sustainable agriculture. In SAFE-World Research Project, February 2001. U.K.: University of Essex.
Reganold, J. P. (2012). The fruits of organic farming. Nature 485:176177.
Rigby, D. and Caceres, D. (2001). Organic farming and the sustainability of agricultural systems. Agricultural Systems 68:2140.
Rosenthal, L. and Schisterman, E. (2010). Meta-analysis: Drawing conclusions when study results vary. Methods in Molecular Biology 594:427434.
Rusinamhodzi, L. (2015). Crop rotations and residue management in conservation agriculture. In Conservation Agriculture, 2137 (Eds Farooq, M. and Siddique, K. H. M.). Switzerland: Springer.
Rusinamhodzi, L., Corbeels, M., van Wijk, M., Rufino, M., Nyamangara, J. and Giller, K. (2011). A meta-analysis of long-term effects of conservation agriculture on maize grain yield under rain-fed conditions. Agronomy for Sustainable Development 31:657673.
Seufert, V., Ramankutty, N. and Foley, J. A. (2012). Comparing the yields of organic and conventional agriculture. Nature 485:229232.
Shennan, C., Krupnik, T. J., Baird, G., Cohen, H., Forbush, K., Lovell, R. J. and Olimpi, E. M. (2017). Organic and conventional agriculture: A useful framing?. Annual Review of Environment and Resources 42:317346.
Sileshi, G., Akinnifesi, F. K., Ajayi, O. C. and Place, F. (2008). Meta-analysis of maize yield response to woody and herbaceous legumes in sub-Saharan Africa. Plant and Soil 307:119.
So, H.-B., Reicosky, D. C., Derpsch, R., Kassam, A., Goddard, T. and Buffet, H. (2015). Comment on Pittelkow et al article by Hwat-Bing So et al. Nature. Online: https://www.nature.com/articles/nature13809. Accessed 4 January 2019.
Steward, P. R., Dougill, A. J., Thierfelder, C., Pittelkow, C. M., Stringer, L. C., Kaudzala, M. and Shackelford, G. E. (2017). The adaptive capacity of maize-based conservation agriculture systems to climate stress in tropical and subtropical environments: A meta-regression of yields. Agriculture, Ecosystems and Environment 251:194202.
Steward, P. R., Dougill, A. J., Thierfelder, C., Pittelkow, C. M., Stringer, L. C., Kudzala, M. and Shackelford, G. E. (2018). The adaptive capacity of maize-based conservation agriculture systems to climate stress in tropical and subtropical environments: A meta-regression of yields. Agriculture, Ecosystems and Environment 251:194202.
Sumberg, J., Thompson, J. A. and Woodhouse, P. (2014). Contested agronomy: Agricultural research in a changing world. In Contested Agronomy: Agricultural Research in a Changing World, 121 (Eds Sumberg, J. and Thompson, J. A.). New York: Routledge.
Sumberg, J., Thompson, J. and Woodhouse, P. (2012). Why agronomy in the developing world has become contentious. Agriculture and Human Values 30:7183.
Valbuena, D., Erenstein, O., Homann-Kee Tui, S., Abdoulaye, T., Claessens, L., Duncan, A. J., Gérard, B., Rufino, M. C., Teufel, N., van Rooyen, A. and van Wijk, M. T. (2012). Conservation agriculture in mixed crop–livestock systems: Scoping crop residue trade-offs in Sub-Saharan Africa and South Asia. Field Crops Research 132:175184.
Van den Putte, A., Govers, G., Diels, J., Gillijns, K. and Demuzere, M. (2010). Assessing the effect of soil tillage on crop growth: A meta-regression analysis on European crop yields under conservation agriculture. European Journal of Agronomy 33:231241.
Whittaker, R. J. (2010). Meta-analyses and mega-mistakes: calling time on meta-analysis of the species richness-productivity relationship. Ecology 9:25222533.
Zheng, C., Jiang, Y., Chen, C., Sun, Y., Feng, J., Deng, A., Song, Z. and Zhang, W. (2014). The impacts of conservation agriculture on crop yield in China depend on specific practices, crops and cropping regions. The Crop Journal 2:289296.
Žukauskas, P., Vveinhardt, J. and Andriukaitienė, R. (2018). Philosophy and paradigm of scientific research. Management Culture and Corporate Social Responsibility. doi: 10.5772/intechopen.70628.

DOES SIZE MATTER? A CRITICAL REVIEW OF META-ANALYSIS IN AGRONOMY

  • TIMOTHY J. KRUPNIK (a1), JENS A. ANDERSSON (a2), LEONARD RUSINAMHODZI (a3), MARC CORBEELS (a3) (a4), CAROL SHENNAN (a5) and BRUNO GÉRARD (a6)...

Metrics

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed