Hostname: page-component-76fb5796d-r6qrq Total loading time: 0 Render date: 2024-04-29T09:11:04.345Z Has data issue: false hasContentIssue false
  • English
  • Français

MAKING THE MOST OF IMPERFECT DATA: A CRITICAL EVALUATION OF STANDARD INFORMATION COLLECTED IN FARM HOUSEHOLD SURVEYS

Published online by Cambridge University Press:  18 December 2018

SIMON FRAVAL Open the ORCID record for SIMON FRAVAL [Opens in a new window] ,
JAMES HAMMOND ,
JANNIKE WICHERN ,
SIMON J. OOSTING ,
IMKE J. M. DE BOER ,
NILS TEUFEL ,
MATS LANNERSTAD ,
KATHARINA WAHA ,
TIM PAGELLA  and
TODD S. ROSENSTOCK
...Show all authors
SIMON FRAVAL*
Affiliation:
International Livestock Research Institute (ILRI), P.O. Box 30709, Nairobi 00100, Kenya Animal Production Systems group, Wageningen University & Research, P.O. Box 338, 6700 AH, Wageningen, The Netherlands
JAMES HAMMOND
Affiliation:
International Livestock Research Institute (ILRI), P.O. Box 30709, Nairobi 00100, Kenya International Centre for Research on Agroforestry (ICRAF), Nairobi 00100, Kenya
JANNIKE WICHERN
Affiliation:
Plant Production Systems group, Wageningen University & Research, P.O. Box 430, 6700 AK, Wageningen, The Netherlands
SIMON J. OOSTING
Affiliation:
Animal Production Systems group, Wageningen University & Research, P.O. Box 338, 6700 AH, Wageningen, The Netherlands
IMKE J. M. DE BOER
Affiliation:
Animal Production Systems group, Wageningen University & Research, P.O. Box 338, 6700 AH, Wageningen, The Netherlands
NILS TEUFEL
Affiliation:
International Livestock Research Institute (ILRI), P.O. Box 30709, Nairobi 00100, Kenya
MATS LANNERSTAD
Affiliation:
Independent Consultant, 115 23 Stockholm, Sweden
KATHARINA WAHA
Affiliation:
Agriculture and Food, Commonwealth Scientific and Industrial Research Organisation (CSIRO), 306 Carmody Road, St Lucia, QLD 4067, Australia
TIM PAGELLA
Affiliation:
School of Environment, Natural Resources and Geography, Bangor University, Deiniol Road, Bangor, Gwynedd LL57 2UW, UK
TODD S. ROSENSTOCK
Affiliation:
International Centre for Research on Agroforestry (ICRAF), Nairobi 00100, Kenya
KEN E. GILLER
Affiliation:
Plant Production Systems group, Wageningen University & Research, P.O. Box 430, 6700 AK, Wageningen, The Netherlands
MARIO HERRERO
Affiliation:
Agriculture and Food, Commonwealth Scientific and Industrial Research Organisation (CSIRO), 306 Carmody Road, St Lucia, QLD 4067, Australia
DAVID HARRIS
Affiliation:
International Centre for Research on Agroforestry (ICRAF), Nairobi 00100, Kenya
MARK T. VAN WIJK
Affiliation:
International Livestock Research Institute (ILRI), P.O. Box 30709, Nairobi 00100, Kenya
Get access Rights & Permissions [Opens in a new window]

Summary

Household surveys are one of the most commonly used tools for generating insight into rural communities. Despite their prevalence, few studies comprehensively evaluate the quality of data derived from farm household surveys. We critically evaluated a series of standard reported values and indicators that are captured in multiple farm household surveys, and then quantified their credibility, consistency and, thus, their reliability. Surprisingly, even variables which might be considered ‘easy to estimate’ had instances of non-credible observations. In addition, measurements of maize yields and land owned were found to be less reliable than other stationary variables. This lack of reliability has implications for monitoring food security status, poverty status and the land productivity of households. Despite this rather bleak picture, our analysis also shows that if the same farm households are followed over time, the sample sizes needed to detect substantial changes are in the order of hundreds of surveys, and not in the thousands. Our research highlights the value of targeted and systematised household surveys and the importance of ongoing efforts to improve data quality. Improvements must be based on the foundations of robust survey design, transparency of experimental design and effective training. The quality and usability of such data can be further enhanced by improving coordination between agencies, incorporating mixed modes of data collection and continuing systematic validation programmes.

Type
Research Article
Information
Experimental Agriculture , Volume 55 , Special Issue 2: Themed Issue: Contested Agronomy , April 2019 , pp. 230 - 250
Copyright
Copyright © Cambridge University Press 2018 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

Alwin, D. F. (2007). Margins of Error: A Study of Reliability in Survey Measurement. New Jersey: John Wiley & Sons, Inc. http://doi.org/10.1002/9780470146316.Google Scholar
Beegle, K., Carletto, C. and Himelein, K. (2012). Reliability of recall in agricultural data. Journal of Development Economics 98 (1):34–41. http://doi.org/10.1016/j.jdeveco.2011.09.005.Google Scholar
Carletto, C., Gourley, S., Murry, S. and Zezza, A. (2016). Cheaper, Faster, and More Than Good Enough is GPS the New Gold Standard in Land Area Measurement? (No. 7759). Washington, DC: World Bank.Google Scholar
Carletto, C., Savastano, S., & Zezza, A. (2011). Fact or Artefact: The Impact of Measurement Errors on the Farm Size - Productivity Relationship (No. 5908). Washington, DC: World Bank. Retrieved from http://documents.worldbank.org/curated/en/2011/12/15545065/fact-or-artefact-impact-measurement-errors-farm-size-productivity-relationshipGoogle Scholar
Carletto, C., Zezza, A. and Banerjee, R. (2013). Towards better measurement of household food security: Harmonizing indicators and the role of household surveys. Global Food Security. https://doi.org/10.1016/j.gfs.2012.11.006Google Scholar
Central Intelligence Agency (CIA). (2016a). Death rate: Country Comparison to the World. Retrieved March 20, 2017 from https://www.cia.gov/library/publications/the-world-factbook/fields/2066.html#tzGoogle Scholar
Central Intelligence Agency (CIA). (2016b). Urbanisation: Country Comparison to the World. Retrieved March 20, 2017 from https://www.cia.gov/library/publications/the-world-factbook/fields/2212.html#tzGoogle Scholar
Champely, S. (2016). PWR: Basic Functions for Power Analysis. Retrieved from https://cran.r-project.org/package=pwrGoogle Scholar
Christiaensen, L. (2017). Agriculture in Africa – Telling myths from facts: A synthesis. Food Policy 67:1–11. https://doi.org/10.1016/j.foodpol.2017.02.002Google Scholar
de Nicola, F. and Giné, X. (2014). How accurate are recall data? Evidence from coastal India. Journal of Development Economics 106:52–65. http://doi.org/10.1016/j.jdeveco.2013.08.008.Google Scholar
Deininger, K., Carletto, C., Savastano, S. and Muwonge, J. (2011). Can Diaries Help Improve Agricultural Production Statistics? Evidence from Uganda (No. 5717). Washington, DC: World Bank.Google Scholar
Evans, B. (1995). On the difference between reliability of measurement and precision of survey instruments. The Canadian Journal of Program Evaluation 10 (2):17–32.Google Scholar
Fecso, R. (2011). A review of errors of direct observation in crop yield surveys. In Measurement Errors in Surveys, 327–346 (Eds Biemer, P. P., Groves, R. M., Lyberg, L. E., Mathiowetz, N. A. and Sudman, S.). Hoboken: John Wiley & Sons, Inc. http://doi.org/10.1002/9781118150382.ch17Google Scholar
Finn, A. and Ranchhod, V. (2017). Genuine fakes: The prevalence and implications of data fabrication in a large South African survey. The World Bank Economic Review 31 (1):129–157. https://doi.org/10.1093/wber/lhv054Google Scholar
Fisher, M., Reimer, J. J. and Carr, E. R. (2010). Who should be interviewed in surveys of household income?. World Development 38 (7):966–973. http://doi.org/10.1016/j.worlddev.2009.11.024.Google Scholar
Food and Agricultural Organisation of the United Nations (FAO). (2001). Human energy requirements: Report of a joint FAO/WHO/UNU expert consultation. FAO Food and Nutrition Technical Report Series, 0:96. https://doi.org/9251052123.Google Scholar
Food and Agricultural Organisation of the United Nations (FAO). (2017a). Statistical Programme of Work 2016–2017. http://www.fao.org/3/a-br622e.pdf.Google Scholar
Food and Agricultural Organisation of the United Nations (FAO). (2017b). Food Price Monitoring and Analysis. Retrieved 1 Mar 2017 from http://www.fao.org/giews/food-prices/tool/public/index.html#/home.Google Scholar
Fraval, S., Hammond, J., Lannerstad, M., Oosting, S. J., Sayula, G., Teufel, N., Silvestri, S., Poole, E. J., Herrero, M. and van Wijk, M. T. (2018). Livelihoods and food security in an urban linked, high potential region of Tanzania: Changes over a three year period. Agricultural Systems 160 (January 2017):87–95. https://doi.org/10.1016/j.agsy.2017.10.013Google Scholar
Frelat, R., Lopez-Ridaura, S., Giller, K. E., Herrero, M., Douxchamps, S., Djurfeldt, A. A., Erenstein, O., Henderson, B., Kassie, M., Paul, B. K., Rigolot, C., Ritzema, R. S., Rodriguez, D., van Asten, P. J. A. and van Wijk, M. T. (2016). Drivers of household food availability in sub-Saharan Africa based on big data from small farms. Proceedings of the National Academy of Sciences USA 113:1518384112. doi:10.1073/pnas.1518384112Google Scholar
Gebrechorkos, S. H., Hülsmann, S. and Bernhofer, C. (2018). Changes in temperature and precipitation extremes in Ethiopia, Kenya, and Tanzania. International Journal of Climatology 1–13. https://doi.org/10.1002/joc.5777.Google Scholar
Gibson, J., Beegle, K., De Weerdt, J. and Friedman, J. (2015). What does variation in survey design reveal about the nature of measurement errors in household consumption?. Oxford Bulletin of Economics and Statistics 77 (3):466–474. http://doi.org/10.1111/obes.12066.Google Scholar
Giller, K. E., Tittonell, P., Rufino, M. C., van Wijk, M. T., Zingore, S., Mapfumo, P., Adjei-Nsiahe, S., Herrero, M., Chikowod, R., Corbeels, M., Rowe, E. C., Baijukya, F., Mwijage, A., Smith, J., Yeboah, E., van der Burg, W. J., Sanogo, O. M., Misiko, M., de Ridder, N., Karanjaf, S., Kaizzi, C., K'ungu, J., Mwale, M., Nwaga, D., Pacini, C. and Vanlauwe, B. (2011). Communicating complexity: Integrated assessment of trade-offs concerning soil fertility management within African farming systems to support innovation and development. Agricultural Systems 104 (2):191–203. http://doi.org/10.1016/j.agsy.2010.07.002.Google Scholar
Global Yield Gap and Water Productivity Atlas. (GYGA, n.d). Retrieved 2017 from www.yieldgap.org.Google Scholar
Gollin, D. (2006). Impacts of International Research on Intertemporal Yield Stability in Wheat and Maize: An Economic Assessment. Mexico: CIMMYT.Google Scholar
Hammond, J., Fraval, S., van Etten, J., Suchini, J. G., Mercado, L., Pagella, T., Frelat, R., Lannerstad, M., Douxchamps, S., Teufel, N., Valbuena, D. and van Wijk, M. T. (2017). The Rural Household Multi-Indicator Survey (RHOMIS) for rapid characterisation of households to inform climate smart agriculture interventions: Description and applications in East Africa and Central America. Agricultural Systems 151:225–233. doi: 10.1016/j.agsy.2016.05.003.Google Scholar
International Fund for Agricultural Development (IFAD). (2016). Rural Development Report 2016. Retrieved from https://www.ifad.org/documents/30600024/30604583/RDR_WEB.pdf/c734d0c4-fbb1-4507-9b4b-6c432c6f38c3.Google Scholar
Jayne, T. S., Chamberlin, J., Traub, L., Sitko, N., Muyanga, M., Yeboah, F. K., Anseeuw, W., Chapoto, A., Wineman, A., Nkonde, C. and Kachule, R. (2016). Africa's changing farm size distribution patterns: The rise of medium-scale farms. Agricultural Economics 47:197–214. doi: 10.1111/agec.12308Google Scholar
Jerven, M. and Johnston, D. (2015). Statistical tragedy in Africa? Evaluating the data base for African economic development. The Journal of Development Studies 51 (2):111–115. https://doi.org/10.1080/00220388.2014.968141.Google Scholar
Juster, F. T., Cao, H., Couper, M., Hill, D., Hurd, M. D., Lupton, J., Perry, M. and Smith, J. P. (2007). Enhancing the Quality of Data on the Measurement of Income and Wealth (No. 151). Ann Arbor.Google Scholar
Kalkuhl, M., Braun, J. Von and Torero, M. (2016). Food Price Volatility and its Implications for Food Security and Policy. Springer Open. https://doi.org/10.1007/978-3-319-28201-5.Google Scholar
Kanyongo, G. Y., Brooks, G. P., Kyei-Blankison, L. and Gocmen, G. (2007). Reliability and statistical power: How measurement fallibility affects power and required sample sizes for several parametric and nonparametric statistics. Journal of Modern Applied Statistical Methods 6 (1):81–90.Google Scholar
Kilic, T., Carletto, C., Zezza, A. and Savastano, S. (2013). Missing (Ness) in Action: Selectivity Bias in GPS-Based Land Area Measurements (No. 6490). Washington, DC: World Bank. http://doi.org/10.1016/j.worlddev.2016.11.018.Google Scholar
Kilic, T. and Sohnesen, T. P. (2015). Same Question But Different Answer: Experimental Evidence on Questionnaire Design's Impact on Poverty Measured by Proxies. Review of Income and Wealth. Washington, DC. https://doi.org/10.1111/roiw.12343.Google Scholar
Leeuw, E. D. De. (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics 21 (2):233–255.Google Scholar
Little, T. D. and Rhemtulla, M. (2013). Planned missing data designs for developmental researchers. Child Development Perspectives 7 (4):199–204. https://doi.org/10.1111/cdep.12043Google Scholar
Mathiowetz, N. A., Brown, C. and Bound, J. (2001). Measurement Error in Surveys of the Low-Income Population. Studies of Welfare Populations: Data Collection and Research Issues. (Vol. 1). Washington, DC: The National Academies Press.Google Scholar
Moore, J. C., Stinson, L. L. and Welniak, E. J. J. (2000). Income measurement error in surveys: A review. Journal of Official Statistics 16 (4):31–361. Retrieved from http://www.jos.nu/Articles/abstract.asp?article=164331.Google Scholar
Neri, A. and Ranalli, M. G. (2012). To Misreport or not to Report? The Measurement of Household Financial Wealth (No. 870). October. Rome: Banca D'Italia. http://doi.org/10.1162/JEEA.2008.6.6.1109.Google Scholar
Organisation for Economic Co-operation and Development (OECD), & Food and Agricultural Organization of the United Nations (FAO). (2017). OECD-FAO Agricultural Outlook 2017-;2026.Google Scholar
Organisation for Economic Co-operation and Development (OECD). (2009). Methods to Monitor and Evaluate the Impacts of Agricultural Policies on Rural Development. Paris: OECD.Google Scholar
Pica-ciamarra, U., Morgan, N. and Baker, D. (2012). Core Livestock Data and Indicators: Results of a Stakeholder Survey. Rome: FAO.Google Scholar
Reardon, T., Crawford, E. and Kelly, V. (1994). Links between nonfarm income and farm investment in African households: Adding the capital market perspective. American Journal of Agricultural Economics 76 (5):1172–1176.Google Scholar
Revelle, W. (2017). Psych: Procedures for Personality and Psychological Research, Northwestern University, Evanston, Illinois, USA, https://CRAN.R-project.org/package=psychVersion=1.7.8.Google Scholar
Rosenstock, T. S., Lamanna, C., Chesterman, S., Hammond, J., Kadiyala, S., Luedeling, E., Shepherd, K., Derenzi, B. and Wijk, M. T. Van. (2017). When less is more: Innovations for tracking progress toward global targets. Current Opinion in Environmental Sustainability 26–27:54–61. http://doi.org/10.1016/j.cosust.2017.02.010.Google Scholar
Rufino, M. C., Quiros, C., Boureima, M., Desta, S., Douxchamps, S., Herrero, M., Kiplimo, J., Lamissa, D., Mango, J., Moussa, A. S., Naab, J., Ndour, Y., Sayula, G., Silvestri, S., Singh, D., Teufel, N. and Wanyama, I. (2013). Developing Generic Tools for Characterizing Agricultural Systems for Climate and Global Change Studies (IMPACTlite – Phase 2). Nairobi: ILRI.Google Scholar
Shrout, P. E. and Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability.1. Shrout PE, Fleiss JL: Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin 86 (2):420–428. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/18839484.Google Scholar
Swindale, A. and Bilinsky, P. (2006). Household Dietary Diversity Score (HDDS) for Measurement of Household Food Access: Indicator Guide (v.2). Washington, DC: FHI 360/FANTA.Google Scholar
Thornton, P. K. and Herrero, M. (2015). livestock farming systems in sub-Saharan Africa. Nature Publishing Group 5 (9):830–836. https://doi.org/10.1038/nclimate2754.Google Scholar
Uganda Bureau of Statistics (UBOS). (n.d.). The Uganda national panel survey 2009/10: Basic Information Document. Kampala.Google Scholar
Uganda Bureau of Statistics (UBOS). (2002). 2002 Uganda Population and Housing Census Analytical Report. Distribution. Kampala. Retrieved from http://www.ubos.org/onlinefiles/uploads/ubos/pdfdocuments/2002CensusPopnSizeGrowthAnalyticalReport.pdf.Google Scholar
Uganda Bureau of Statistics (UBOS). (2007). Uganda national household survey 2005/2006. Kampala. Retrieved August 15, 2016 from http://www.ubos.org/onlinefiles/uploads/ubos/statistical_abstracts/Statistical Abstract 2014.pdf.Google Scholar
United Nations Department of Economic and Social Affairs (UN). (2005). Household Sample Surveys in Developing and Transition Countries. Studies in Methods (Vol. F). Retrieved from http://unstats.un.org/unsd/hhsurveys/.Google Scholar
United Nations Framework Convention on Climate Change (UNFCCC). (2012). Standard for sampling and surveys for CDM project activities and programme activities. Bonn.Google Scholar
Weisberg, H. (2005). The Total Survey Error Approach. Chicago, IL: The University of Chicago Press.Google Scholar
World Bank. (2017). Living Standards Measurement Survey. Retrieved 15 Jan 2017 from www.worldbank.org/lsms.Google Scholar
World Bank. (n.d) Living Standards Measurement Study-Integrated Surveys on Agriculture. Retrieved September 15, 2018 from http://surveys.worldbank.org/lsms/programs/integrated-surveys-agriculture-ISA.Google Scholar
Zezza, A., Federighi, G., Adamou, K. and Hiernaux, P. (2014). Milking the Data: Measuring Income from Milk Production in Extensive Livestock Systems Experimental Evidence from Niger (No. 7114). Rome.Google Scholar
Supplementary material: File

Fraval et al. supplementary material

Tables S1-S3

Download Fraval et al. supplementary material(File)
File 26.3 KB