Skip to main content
×
×
Home

Using pre- and post-survey instruments in interventions: determining the random response benchmark and its implications for measuring effectiveness

  • George C Davis (a1) (a2), Ranju Baral (a3), Thomas Strayer (a4) and Elena L Serrano (a1)
Abstract
Objective

The present communication demonstrates that even if individuals are answering a pre/post survey at random, the percentage of individuals showing improvement from the pre- to the post-survey can be surprisingly high. Some simple formulas and tables are presented that will allow analysts to quickly determine the expected percentage of individuals showing improvement if participants just answered the survey at random. This benchmark percentage, in turn, defines the appropriate null hypothesis for testing if the actual percentage observed is greater than the expected random answering percentage.

Design

The analysis is demonstrated by testing if actual improvement in a component of the US Department of Agriculture’s (USDA) Expanded Food and Nutrition Education Program is significantly different from random answering improvement.

Setting

USA.

Subjects

From 2011 to 2014, 364320 adults completed a standardized pre- and post-survey administered by the USDA.

Results

For each year, the statement that the actual number of improvements is less than the expected number if the questions were just answered at random cannot be rejected. This does not mean that the pre-/post-test survey instrument is flawed, only that the data are being inappropriately evaluated.

Conclusions

Knowing the percentage of individuals showing improvement on a pre/post survey instrument when questions are randomly answered is an important benchmark number to determine in order to draw valid inferences about nutrition interventions. The results presented here should help analysts in determining this benchmark number for some common survey structures and avoid drawing faulty inferences about the effectiveness of an intervention.

Copyright
Corresponding author
* Corresponding author: Email georgedavis@vt.edu
References
Hide All
1. Andreyeva, T, Middleton, AE, Long, MW et al. (2011) Food retailer practices, attitudes, and beliefs about the supply of healthy foods. Public Health Nutr 14, 10241031.
2. Song, HJ, Gittelsohn, J, Kim, M et al. (2009) A corner store intervention in a low-income urban community is associated with increased availability and sales of some healthy foods. Public Health Nutr 12, 20602067.
3. Martínez-Donate, AP, Riggall, AJ, Meinen, AM et al. (2015) Evaluation of a pilot healthy eating intervention in restaurants and food stores of a rural community: a randomized community trial. BMC Public Health 15, 136.
4. Escaron, AL, Martinez-Donate, AP, Riggall, AJ et al. (2016) Developing and implementing ‘Waupaca Eating Smart’: a restaurant and supermarket intervention to promote healthy eating through changes in the food environment. Health Promot Pract 17, 265277.
5. Lee, RM, Rothstein, JD, Gergen, J et al. (2015) Process evaluation of a comprehensive supermarket intervention in a low-income Baltimore community. Health Promot Pract 16, 849858.
6. Baral, R, Davis, GC, Blake, S et al. (2013) Using national data to estimate average cost effectiveness of EFNEP outcomes by state/territory. J Nutr Educ Behav 45, 183187.
7. Ott, RL & Longnecker, M (2001) An Introduction to Statistical Methods and Analysis, 5th ed. Pacific Grove, CA: Duxbury.
8. US Department of Agriculture (2016) Expanded Food and Nutrition Education Program (EFNEP). http://www.nifa.usda.gov/program/expanded-food-and-nutrition-education-program-efnep (accessed September 2016).
9. US Department of Agriculture (2016) Expanded Food and Nutrition Education Program. National Data. Years 2010–2014. http://www.nifa.usda.gov/efnep-national-data-reports (accessed September 2016).
10. Murphy, SP, Kaiser, LL, Townsend, MS et al. (2001) Evaluation of validity of items for a food behavior checklist. J Am Diet Assoc 101, 751761.
11. George, GC, Milani, TJ, Hanss-Nuss, H et al. (2004) Development and validation of a semi-quantitative food frequency questionnaire for young adult women in the southwestern United States. Nutr Res 24, 2943.
12. Lim, SS, Gold, A, Gaillard, PR et al. (2015) Validation of 2 brief fruit and vegetable assessment instruments among third-grade students. J Nutr Educ Behav 47, 446451.
13. Bond, TG & Fox, CM (2007) Applying the Rasch Model: Fundamental Measurement in the Human Sciences, 2nd ed. Mahwah, NJ: LEA Publishers.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Public Health Nutrition
  • ISSN: 1368-9800
  • EISSN: 1475-2727
  • URL: /core/journals/public-health-nutrition
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Keywords:

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 5
Total number of PDF views: 75 *
Loading metrics...

Abstract views

Total abstract views: 439 *
Loading metrics...

* Views captured on Cambridge Core between 21st December 2017 - 22nd May 2018. This data will be updated every 24 hours.