Surveys that measure household food security using the US Household Food Security Survey Module are typically conducted as telephone interviews or as in-person, face-to-face interviews. It is important to know whether data collected using these two methods are directly comparable, or whether there is a mode-of-interview effect that might bias comparisons between data collected using different methods. For example, the Current Population Survey Food Security Supplement (CPS-FSS), on which the US Department of Agriculture (USDA) bases its national food security statistics, is conducted using both methods, but about two-thirds of households are interviewed by telephone. If a community or a state conducts a survey using in-person interviews and wants to compare their results with national statistics, researchers need to know whether the interview mode affects responses.
This analysis compares food security survey data from telephone and in-person interviews in the CPS-FSS to determine whether interview mode (telephone vs. in-person) affects measurement of food security.
Methods
Data source
Data are from the CPS-FSS from April 2001 and December 2001, 2002, 2003 and 2004. The Current Population Survey, to which the CPS-FSS is a supplement, is the nation’s primary source of labour force, employment and earnings data. It is a nationally representative survey of about 60 000 households conducted monthly by the Census Bureau for the Bureau of Labor Statistics. The sample is a probability sample from a complete address list of US households maintained and updated by the Census Bureau. In December, after completing the labour force interview, households are administered the Food Security Supplement, which includes questions about households’ ability to access enough food for their needs. About 15% of households that complete the labour force interview are unable or unwilling to complete the supplement. Sample weights of households that complete the supplement are adjusted by the Census Bureau to match state- and national-level population controls so that statistics based on the supplement weights represent the civilian, non-institutionalised, national population. USDA uses the CPS-FSS data as the basis of its annual reports on the food security of the nation’s householdsReference Nord, Andrews and Carlson1, and for research on food security measurement and factors affecting households’ food security.
Each household selected for the CPS is interviewed in four consecutive months and then in the same four months in the following year. Most interviews are conducted in-person in the household’s first month in the survey and in their first re-entry month in the second year, and by telephone in the other six months. A new group of households enters the sample each month; in any given month about one-quarter of the sample is in month-in-sample groups that are normally interviewed by telephone.
Interview mode does not depend entirely on month-in-sample, however. Some households prefer to be interviewed by telephone in months 1 and 5; some households do not have telephones, or prefer to be interviewed in-person in other months. This pattern is not random with respect to food security. Generally, more affluent households are disproportionately interviewed by telephone.
Food security measurement
Household food security – access by all household members at all times to enough food for active healthy living – is measured in the CPS-FSS by responses to a series of questions about food-related behaviours, experiences and conditions that are known to characterise households having difficulty meeting their food needsReference Hamilton, Cook, Thompson, Buron, Frongillo and Olson2, Reference Bickel, Nord, Price, Hamilton and Cook3. The questions cover a wide range of severity of food access problems. Each question specifies lack of money or other resources to obtain food as the reason for the condition or behaviour. The series includes 10 questions about food conditions at the household level and among adults in the household and, if there are children present in the household, an additional eight questions about children’s food conditions. Based on the number of food-insecure conditions reported (a raw-score count of dichotomous responses), households are classified as either food-secure (0–2 food-insecure conditions) or food-insecure (3 or more food-insecure conditions). Food-insecure households are further classified as having low food security (3–5 food-insecure conditions) or very low food security (6+ food-insecure conditions).Footnote * Households with children are classified as having very low food security if they report eight or more food-insecure conditions, including conditions among both adults and children.
Classification based on raw score is statistically justified provided response data fit certain assumptions specified by the Rasch measurement model. Statistical methods based on these assumptions were used to assess whether response patterns differed significantly between interview modes.
Comparing response patterns – the Rasch measurement model
An essential characteristic of the food security scale is that the items comprising it vary across a wide range of severity of food insecurity. The precise severity level of each item (the ‘item calibration’ or ‘item score’) is estimated empirically from the overall household response pattern to the scale items. Generally, more severe items are less frequently affirmed than less severe items, and a household that affirms an item of mid-range severity is likely to have also affirmed all less severe items. Similarly, a household that denies an item at mid range is likely to deny all more severe items. These response patterns are not universal, but they are predominant. The Rasch measurement modelReference Fischer and Molenaar4, Reference Bond and Fox5 formalises this item severity-ordering concept and provides a basis for standard statistical methods to estimate each item’s severity and assess the extent to which the response patterns observed in a dataset are consistent with the severity-order concept.
To assess whether response patterns were affected by survey administration mode, data from the two interview modes were fitted separately to the Rasch model. Overall model fit, item severity scores and item-fit statistics were compared between the two modes.Footnote † Overall model fit was assessed by comparing the dispersion (measured by standard deviation) of item scores between modes. Item dispersion is a measure of how consistently item responses adhere to the expected severity ordering. Lower dispersion indicates less consistently ordered responses (i.e. lower item discrimination), which can result from poorer understanding of questions, less carefully considered responses, or less precision by interviewers in recording responses. The severities of items relative to one another were compared between the two modes to identify any items or groups of items that were more likely to be affirmed, given responses to other items, in one mode compared with the other.
Item-infit statistics were compared to assess whether any specific item was understood less consistently in one interview mode compared with the other. Item infit measures the misfit of each item compared with expectations given overall model fit. A value of 1 indicates that the extent of out-of-order responses to the item is at the average for all items in the scale. Values above 1 indicate a disproportionate share of out-of-order responses and, therefore, lower discrimination of the item. Infit values in the range of 0.8 to 1.2 are generally considered suitable for a Rasch-model-based measureReference Linacre and Wright6.Footnote *
All models were estimated using unconditional (joint) maximum likelihood methods implemented in ERSRasch. ERSRasch is a set of SAS programs developed by USDA’s Economic Research Service to facilitate estimation of Rasch-based measurement models for food security data. The estimation algorithm follows that used in WinstepsReference Linacre7 and returns identical item scores and item-fit statistics when applied to the same data.
Assessing effects of interview mode on prevalence estimates
Even if the Household Food Security Scale measures essentially the same phenomenon in the two survey administration modes, it is possible that a general response bias could exist that affects all items and could bias estimated food insecurity prevalence rates between the two modes. Assessing whether such a bias exists in the CPS-FSS data is complicated by the non-random assignment of households to interview mode. Households interviewed in-person had considerably higher prevalence rates of food insecurity and very low food security than those interviewed by telephone (odds ratios were approximately 1.5), but also had lower income and education and less favourable labour force characteristics. To isolate any effect of administration mode, we estimated multivariate logistic regression models of food insecurity and very low food security on interview mode, with an extensive set of control variables for income, employment, and other household characteristics that could be associated with interview mode and food security. Control variables were derived from data in the labour force section of the survey and from the supplement as follows:
• Household structure was entered as a set of mutually exclusive dummy variables with the reference category, married couple with one or more children, omitted.
• Age of oldest child was entered as a mutually exclusive set of dummy variables. The reference category consisted of households with no child or with the oldest child aged 0–4 years. Households with vs. without children were fully identified by the household structure variables, so the odds ratios for these variables are comparisons to households with a child aged 0–4 years and no older child.
• Income/poverty is the ratio of annual household income to the federal poverty line for that household. (Annual household income is the CPS control card income – total annual income of all household members reported at entry into the survey and updated one year later at re-entry – assessed at the centre of the categorical range.) Poverty lines were based on the Census Bureau’s poverty thresholds for the year of the survey. A set of dummy variables identifying income/poverty ranges was also included to control more completely for non-linearity in the association between income and food security.
• Labour force participation and employment of all adults in each household were described by two series of dummy variables. First, each person in the household aged 18 years and over (or minor reference person or spouse of reference person) was placed in one of the following standard labour force categories: (1) employed full-time; (2) not in labour force, retired; (3) employed, part-time for non-economic reasons; (4) employed, part-time for economic reasons; (5) unemployed (looking for work or laid off); (6) not in labour force, disabled; (7) not in labour force for reasons other than retirement or disability. The person in the lowest-numbered category was considered the ‘primary earner’, and his/her classification determined the values of the primary earner dummy variables. The labour force categories of other adults in the household were represented by a second set of dummy variables describing whether any other adult (other than the primary earner) was in each category.
Analysis samples
The multi-year analysis sample (after omitting 13% of households for which interview mode was not identified) consisted of 198728 households: 65749 were interviewed in-person and 132979 by telephone. The samples used to assess item response patterns, which omitted households that affirmed all items or denied all items, consisted of 14055 households interviewed in-person and 21413 households interviewed by telephone. These are very large samples for scale-analysis purposes and provide precise estimates of item calibrations. The sample used for the logistic regression analyses included households that affirmed or denied all items, but omitted households lacking income information (an essential control variable), and consisted of 171 642 households.
All analyses were conducted using adjusted household supplement weights. The adjustment consisted of a ratio adjustment, constant within each survey, calculated to result in a mean weight of 1.0 for all households in the analysis sample within that survey.
Results
There was no substantial effect of interview mode on the relative severity of items or on the consistency of response patterns (Table 1 and Fig. 1). Although differences between interview modes were statistically significant for several items, the differences were substantively small. Furthermore, there were no systematic differences between adult-referenced and child-referenced items, or between less severe and more severe items. The average item discrimination (measured by the standard deviation of items when item scores were estimated on a logistic metric) differed by only 0.2% between the two interview modes, indicating that the ordering of responses was equally consistent in the two modes.
Comparison of item scores between telephone and in-person interviews
Data source: Current Population Survey Food Security Supplements, April 2001 and December 2001, 2002, 2003 and 2004.
*Difference is statistically significant with 90% confidence.
†In-person scores were adjusted by a linear transformation to equate the mean and standard deviation of all item scores to those of the telephone-interview scale. The effect of the transformation was negligible, however, since the mean is arbitrary and the discrimination coefficients were essentially identical, differing by only 0.2%.
Comparison of item scores between telephone and in-person interviews. Data source: Current Population Survey Food Security Supplements, April 2001 and December 2001, 2002, 2003 and 2004
Item-infit statistics were all within conventionally acceptable limits (between 0.8 and 1.2) for both modes (Table 2). There were no apparent systematic differences associated with child vs. adult items or with more vs. less severe items. The greatest differences were the lower discrimination (i.e. higher infit, indicating less consistent ordering of responses relative to those of other items) for the most severe child item, children did not eat for whole day, when administered by telephone compared with administration in-person, and the opposite association for the item, children were hungry. (Infits for this latter item were, however, below unity for both modes, indicating more consistent responses than for the average of all items.)
Comparison of item-infit statistics between telephone and in-person interviews
Data source: Current Population Survey Food Security Supplements, April 2001 and December 2001, 2002, 2003 and 2004.
The prevalence of food insecurity was considerably higher among households interviewed in-person (13.99%) than among those interviewed by telephone (9.46%) – an odds ratio of 1.56. The corresponding statistics for very low food security were 4.63% for those interviewed in-person and 2.87% for those interviewed by telephone – an odds ratio of 1.64. However, almost all of these differences were accounted for by identifiable differences in the socio-economic characteristics of the households in the two samples (Table 3). With controls, the odds ratio of food insecurity based on in-person vs. telephone interview was reduced from 1.56 to 1.036 (P = 0.088) and that for very low food security was reduced from 1.64 to 1.073 (P = 0.029). It is not certain whether these adjusted odds ratios reflect true mode effects (small upward biases for in-person vs. telephone interviews) or whether they reflect inadequate controls for differences between the samples, in which case the bias could be negligible or opposite. In either case, it seems unlikely that the mode effect is very large.
Logistic regression models of low food security and very low food security on mode of interview with controls for household socio-economic characteristics (n = 171642)
CPS – Current Population Survey; MSA – metropolitan statistical area.
Data source: Current Population Survey Food Security Supplements, April 2001 and December 2001, 2002, 2003 and 2004.
Statistics are based on household supplement weights adjusted so that the weighted number of households is equal to the unweighted number of cases in each year. The models also included dummy variables (coefficients not shown) for the year of survey and for the month-in-sample.
Discussion
In the context of the CPS-FSS, there is no evidence of a substantial effect of interview mode on the measurement of food security. Response patterns to the indicators that comprise the food security scale were, for practical purposes, invariant between modes. The prevalences of food insecurity and of very low food security estimated from the two interview modes differed by only small proportions once differences in the socio-economic characteristics of households interviewed in the two modes were accounted for. The remaining differences implied an upward bias for households interviewed in-person compared with those interviewed by telephone amounting to about 4% for food insecurity and 7% for very low food security. However, it is likely that some of this – perhaps all of it – is due to incomplete control for differences in the socio-economic characteristics of the two groups. The true bias could be in either direction, but it appears unlikely that it is very large in either case.
The extent to which these results can be generalised to other surveys may depend to some extent on survey characteristics other than interview mode. The Census Bureau has very high name recognition and most respondents have a high level of confidence in the agency. In a survey conducted by an agency or institution that is unfamiliar to respondents, or for which respondents’ respect and confidence is lower or less certain, the interview mode might have a larger effect. However, for surveys that are carefully conducted by agencies or institutions that have respondents’ confidence and trust, the results of this research suggest that the interview mode will have little or no effect on the measurement of food security.
Acknowledgements
Both authors are employed by the US Department of Agriculture and conducted this study as part of their official responsibilities. There are no conflicts of interest. M.N. conceived the study, designed and conducted the analysis, and wrote a description of the results. H.H. reviewed, revised and completed the manuscript, and prepared it for submission.The views expressed here are those of the authors, and may not be attributed to the Economic Research Service, the Food and Nutrition Service, or the US Department of Agriculture.
