Skip to main content Accessibility help
×
Home

MAKING THE MOST OF IMPERFECT DATA: A CRITICAL EVALUATION OF STANDARD INFORMATION COLLECTED IN FARM HOUSEHOLD SURVEYS

  • SIMON FRAVAL (a1) (a2), JAMES HAMMOND (a1) (a3), JANNIKE WICHERN (a4), SIMON J. OOSTING (a2), IMKE J. M. DE BOER (a2), NILS TEUFEL (a1), MATS LANNERSTAD (a5), KATHARINA WAHA (a6), TIM PAGELLA (a7), TODD S. ROSENSTOCK (a3), KEN E. GILLER (a4), MARIO HERRERO (a6), DAVID HARRIS (a3) and MARK T. VAN WIJK (a1)...

Summary

Household surveys are one of the most commonly used tools for generating insight into rural communities. Despite their prevalence, few studies comprehensively evaluate the quality of data derived from farm household surveys. We critically evaluated a series of standard reported values and indicators that are captured in multiple farm household surveys, and then quantified their credibility, consistency and, thus, their reliability. Surprisingly, even variables which might be considered ‘easy to estimate’ had instances of non-credible observations. In addition, measurements of maize yields and land owned were found to be less reliable than other stationary variables. This lack of reliability has implications for monitoring food security status, poverty status and the land productivity of households. Despite this rather bleak picture, our analysis also shows that if the same farm households are followed over time, the sample sizes needed to detect substantial changes are in the order of hundreds of surveys, and not in the thousands. Our research highlights the value of targeted and systematised household surveys and the importance of ongoing efforts to improve data quality. Improvements must be based on the foundations of robust survey design, transparency of experimental design and effective training. The quality and usability of such data can be further enhanced by improving coordination between agencies, incorporating mixed modes of data collection and continuing systematic validation programmes.

Copyright

Corresponding author

¶¶Corresponding author. Email: simon.fraval@wur.nl

References

Hide All
Alwin, D. F. (2007). Margins of Error: A Study of Reliability in Survey Measurement. New Jersey: John Wiley & Sons, Inc. http://doi.org/10.1002/9780470146316.
Beegle, K., Carletto, C. and Himelein, K. (2012). Reliability of recall in agricultural data. Journal of Development Economics 98 (1):3441. http://doi.org/10.1016/j.jdeveco.2011.09.005.
Carletto, C., Gourley, S., Murry, S. and Zezza, A. (2016). Cheaper, Faster, and More Than Good Enough is GPS the New Gold Standard in Land Area Measurement? (No. 7759). Washington, DC: World Bank.
Carletto, C., Savastano, S., & Zezza, A. (2011). Fact or Artefact: The Impact of Measurement Errors on the Farm Size - Productivity Relationship (No. 5908). Washington, DC: World Bank. Retrieved from http://documents.worldbank.org/curated/en/2011/12/15545065/fact-or-artefact-impact-measurement-errors-farm-size-productivity-relationship
Carletto, C., Zezza, A. and Banerjee, R. (2013). Towards better measurement of household food security: Harmonizing indicators and the role of household surveys. Global Food Security. https://doi.org/10.1016/j.gfs.2012.11.006
Central Intelligence Agency (CIA). (2016a). Death rate: Country Comparison to the World. Retrieved March 20, 2017 from https://www.cia.gov/library/publications/the-world-factbook/fields/2066.html#tz
Central Intelligence Agency (CIA). (2016b). Urbanisation: Country Comparison to the World. Retrieved March 20, 2017 from https://www.cia.gov/library/publications/the-world-factbook/fields/2212.html#tz
Champely, S. (2016). PWR: Basic Functions for Power Analysis. Retrieved from https://cran.r-project.org/package=pwr
Christiaensen, L. (2017). Agriculture in Africa – Telling myths from facts: A synthesis. Food Policy 67:111. https://doi.org/10.1016/j.foodpol.2017.02.002
de Nicola, F. and Giné, X. (2014). How accurate are recall data? Evidence from coastal India. Journal of Development Economics 106:5265. http://doi.org/10.1016/j.jdeveco.2013.08.008.
Deininger, K., Carletto, C., Savastano, S. and Muwonge, J. (2011). Can Diaries Help Improve Agricultural Production Statistics? Evidence from Uganda (No. 5717). Washington, DC: World Bank.
Evans, B. (1995). On the difference between reliability of measurement and precision of survey instruments. The Canadian Journal of Program Evaluation 10 (2):1732.
Fecso, R. (2011). A review of errors of direct observation in crop yield surveys. In Measurement Errors in Surveys, 327346 (Eds Biemer, P. P., Groves, R. M., Lyberg, L. E., Mathiowetz, N. A. and Sudman, S.). Hoboken: John Wiley & Sons, Inc. http://doi.org/10.1002/9781118150382.ch17
Finn, A. and Ranchhod, V. (2017). Genuine fakes: The prevalence and implications of data fabrication in a large South African survey. The World Bank Economic Review 31 (1):129157. https://doi.org/10.1093/wber/lhv054
Fisher, M., Reimer, J. J. and Carr, E. R. (2010). Who should be interviewed in surveys of household income?. World Development 38 (7):966973. http://doi.org/10.1016/j.worlddev.2009.11.024.
Food and Agricultural Organisation of the United Nations (FAO). (2001). Human energy requirements: Report of a joint FAO/WHO/UNU expert consultation. FAO Food and Nutrition Technical Report Series, 0:96. https://doi.org/9251052123.
Food and Agricultural Organisation of the United Nations (FAO). (2017a). Statistical Programme of Work 2016–2017. http://www.fao.org/3/a-br622e.pdf.
Food and Agricultural Organisation of the United Nations (FAO). (2017b). Food Price Monitoring and Analysis. Retrieved 1 Mar 2017 from http://www.fao.org/giews/food-prices/tool/public/index.html#/home.
Fraval, S., Hammond, J., Lannerstad, M., Oosting, S. J., Sayula, G., Teufel, N., Silvestri, S., Poole, E. J., Herrero, M. and van Wijk, M. T. (2018). Livelihoods and food security in an urban linked, high potential region of Tanzania: Changes over a three year period. Agricultural Systems 160 (January 2017):8795. https://doi.org/10.1016/j.agsy.2017.10.013
Frelat, R., Lopez-Ridaura, S., Giller, K. E., Herrero, M., Douxchamps, S., Djurfeldt, A. A., Erenstein, O., Henderson, B., Kassie, M., Paul, B. K., Rigolot, C., Ritzema, R. S., Rodriguez, D., van Asten, P. J. A. and van Wijk, M. T. (2016). Drivers of household food availability in sub-Saharan Africa based on big data from small farms. Proceedings of the National Academy of Sciences USA 113:1518384112. doi:10.1073/pnas.1518384112
Gebrechorkos, S. H., Hülsmann, S. and Bernhofer, C. (2018). Changes in temperature and precipitation extremes in Ethiopia, Kenya, and Tanzania. International Journal of Climatology 113. https://doi.org/10.1002/joc.5777.
Gibson, J., Beegle, K., De Weerdt, J. and Friedman, J. (2015). What does variation in survey design reveal about the nature of measurement errors in household consumption?. Oxford Bulletin of Economics and Statistics 77 (3):466474. http://doi.org/10.1111/obes.12066.
Giller, K. E., Tittonell, P., Rufino, M. C., van Wijk, M. T., Zingore, S., Mapfumo, P., Adjei-Nsiahe, S., Herrero, M., Chikowod, R., Corbeels, M., Rowe, E. C., Baijukya, F., Mwijage, A., Smith, J., Yeboah, E., van der Burg, W. J., Sanogo, O. M., Misiko, M., de Ridder, N., Karanjaf, S., Kaizzi, C., K'ungu, J., Mwale, M., Nwaga, D., Pacini, C. and Vanlauwe, B. (2011). Communicating complexity: Integrated assessment of trade-offs concerning soil fertility management within African farming systems to support innovation and development. Agricultural Systems 104 (2):191203. http://doi.org/10.1016/j.agsy.2010.07.002.
Global Yield Gap and Water Productivity Atlas. (GYGA, n.d). Retrieved 2017 from www.yieldgap.org.
Gollin, D. (2006). Impacts of International Research on Intertemporal Yield Stability in Wheat and Maize: An Economic Assessment. Mexico: CIMMYT.
Hammond, J., Fraval, S., van Etten, J., Suchini, J. G., Mercado, L., Pagella, T., Frelat, R., Lannerstad, M., Douxchamps, S., Teufel, N., Valbuena, D. and van Wijk, M. T. (2017). The Rural Household Multi-Indicator Survey (RHOMIS) for rapid characterisation of households to inform climate smart agriculture interventions: Description and applications in East Africa and Central America. Agricultural Systems 151:225233. doi: 10.1016/j.agsy.2016.05.003.
International Fund for Agricultural Development (IFAD). (2016). Rural Development Report 2016. Retrieved from https://www.ifad.org/documents/30600024/30604583/RDR_WEB.pdf/c734d0c4-fbb1-4507-9b4b-6c432c6f38c3.
Jayne, T. S., Chamberlin, J., Traub, L., Sitko, N., Muyanga, M., Yeboah, F. K., Anseeuw, W., Chapoto, A., Wineman, A., Nkonde, C. and Kachule, R. (2016). Africa's changing farm size distribution patterns: The rise of medium-scale farms. Agricultural Economics 47:197214. doi: 10.1111/agec.12308
Jerven, M. and Johnston, D. (2015). Statistical tragedy in Africa? Evaluating the data base for African economic development. The Journal of Development Studies 51 (2):111115. https://doi.org/10.1080/00220388.2014.968141.
Juster, F. T., Cao, H., Couper, M., Hill, D., Hurd, M. D., Lupton, J., Perry, M. and Smith, J. P. (2007). Enhancing the Quality of Data on the Measurement of Income and Wealth (No. 151). Ann Arbor.
Kalkuhl, M., Braun, J. Von and Torero, M. (2016). Food Price Volatility and its Implications for Food Security and Policy. Springer Open. https://doi.org/10.1007/978-3-319-28201-5.
Kanyongo, G. Y., Brooks, G. P., Kyei-Blankison, L. and Gocmen, G. (2007). Reliability and statistical power: How measurement fallibility affects power and required sample sizes for several parametric and nonparametric statistics. Journal of Modern Applied Statistical Methods 6 (1):8190.
Kilic, T., Carletto, C., Zezza, A. and Savastano, S. (2013). Missing (Ness) in Action: Selectivity Bias in GPS-Based Land Area Measurements (No. 6490). Washington, DC: World Bank. http://doi.org/10.1016/j.worlddev.2016.11.018.
Kilic, T. and Sohnesen, T. P. (2015). Same Question But Different Answer: Experimental Evidence on Questionnaire Design's Impact on Poverty Measured by Proxies. Review of Income and Wealth. Washington, DC. https://doi.org/10.1111/roiw.12343.
Leeuw, E. D. De. (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics 21 (2):233255.
Little, T. D. and Rhemtulla, M. (2013). Planned missing data designs for developmental researchers. Child Development Perspectives 7 (4):199204. https://doi.org/10.1111/cdep.12043
Mathiowetz, N. A., Brown, C. and Bound, J. (2001). Measurement Error in Surveys of the Low-Income Population. Studies of Welfare Populations: Data Collection and Research Issues. (Vol. 1). Washington, DC: The National Academies Press.
Moore, J. C., Stinson, L. L. and Welniak, E. J. J. (2000). Income measurement error in surveys: A review. Journal of Official Statistics 16 (4):31361. Retrieved from http://www.jos.nu/Articles/abstract.asp?article=164331.
Neri, A. and Ranalli, M. G. (2012). To Misreport or not to Report? The Measurement of Household Financial Wealth (No. 870). October. Rome: Banca D'Italia. http://doi.org/10.1162/JEEA.2008.6.6.1109.
Organisation for Economic Co-operation and Development (OECD), & Food and Agricultural Organization of the United Nations (FAO). (2017). OECD-FAO Agricultural Outlook 2017-;2026.
Organisation for Economic Co-operation and Development (OECD). (2009). Methods to Monitor and Evaluate the Impacts of Agricultural Policies on Rural Development. Paris: OECD.
Pica-ciamarra, U., Morgan, N. and Baker, D. (2012). Core Livestock Data and Indicators: Results of a Stakeholder Survey. Rome: FAO.
Reardon, T., Crawford, E. and Kelly, V. (1994). Links between nonfarm income and farm investment in African households: Adding the capital market perspective. American Journal of Agricultural Economics 76 (5):11721176.
Revelle, W. (2017). Psych: Procedures for Personality and Psychological Research, Northwestern University, Evanston, Illinois, USA, https://CRAN.R-project.org/package=psychVersion=1.7.8.
Rosenstock, T. S., Lamanna, C., Chesterman, S., Hammond, J., Kadiyala, S., Luedeling, E., Shepherd, K., Derenzi, B. and Wijk, M. T. Van. (2017). When less is more: Innovations for tracking progress toward global targets. Current Opinion in Environmental Sustainability 26–27:5461. http://doi.org/10.1016/j.cosust.2017.02.010.
Rufino, M. C., Quiros, C., Boureima, M., Desta, S., Douxchamps, S., Herrero, M., Kiplimo, J., Lamissa, D., Mango, J., Moussa, A. S., Naab, J., Ndour, Y., Sayula, G., Silvestri, S., Singh, D., Teufel, N. and Wanyama, I. (2013). Developing Generic Tools for Characterizing Agricultural Systems for Climate and Global Change Studies (IMPACTlite – Phase 2). Nairobi: ILRI.
Shrout, P. E. and Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability.1. Shrout PE, Fleiss JL: Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin 86 (2):420428. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/18839484.
Swindale, A. and Bilinsky, P. (2006). Household Dietary Diversity Score (HDDS) for Measurement of Household Food Access: Indicator Guide (v.2). Washington, DC: FHI 360/FANTA.
Thornton, P. K. and Herrero, M. (2015). livestock farming systems in sub-Saharan Africa. Nature Publishing Group 5 (9):830836. https://doi.org/10.1038/nclimate2754.
Uganda Bureau of Statistics (UBOS). (n.d.). The Uganda national panel survey 2009/10: Basic Information Document. Kampala.
Uganda Bureau of Statistics (UBOS). (2002). 2002 Uganda Population and Housing Census Analytical Report. Distribution. Kampala. Retrieved from http://www.ubos.org/onlinefiles/uploads/ubos/pdfdocuments/2002CensusPopnSizeGrowthAnalyticalReport.pdf.
Uganda Bureau of Statistics (UBOS). (2007). Uganda national household survey 2005/2006. Kampala. Retrieved August 15, 2016 from http://www.ubos.org/onlinefiles/uploads/ubos/statistical_abstracts/Statistical Abstract 2014.pdf.
United Nations Department of Economic and Social Affairs (UN). (2005). Household Sample Surveys in Developing and Transition Countries. Studies in Methods (Vol. F). Retrieved from http://unstats.un.org/unsd/hhsurveys/.
United Nations Framework Convention on Climate Change (UNFCCC). (2012). Standard for sampling and surveys for CDM project activities and programme activities. Bonn.
Weisberg, H. (2005). The Total Survey Error Approach. Chicago, IL: The University of Chicago Press.
World Bank. (2017). Living Standards Measurement Survey. Retrieved 15 Jan 2017 from www.worldbank.org/lsms.
World Bank. (n.d) Living Standards Measurement Study-Integrated Surveys on Agriculture. Retrieved September 15, 2018 from http://surveys.worldbank.org/lsms/programs/integrated-surveys-agriculture-ISA.
Zezza, A., Federighi, G., Adamou, K. and Hiernaux, P. (2014). Milking the Data: Measuring Income from Milk Production in Extensive Livestock Systems Experimental Evidence from Niger (No. 7114). Rome.
Type Description Title
WORD
Supplementary materials

Fraval et al. supplementary material
Tables S1-S3

 Word (26 KB)
26 KB

Metrics

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed