Setting

We conducted a retrospective longitudinal cohort study conducted at Atrium Health Wake Forest Baptist between January 2017 and June 2021. This study was approved by the Wake Forest University Health Sciences Institutional Review Board (IRB00071608).

Study population

We included all patients in one academic pediatric primary care clinic with patients who were screened at least once for FI between January 2017 and August 2021. Encounters were included if FI was screened for and documented in the EHR (N = 25,214) and were excluded if FI was not screened for or documented in the EHR (N = 3,296). The clinic is in an urban neighborhood and serves a predominantly racial and ethnic minority population of which more than 90% are covered by Medicaid insurance. The medical system is also part of a tertiary care hospital, has multiple board-certified pediatric subspecialties, a pediatric emergency department, and serves the majority of children across western North Carolina.

Measurement of FI

The clinic screens all patients for FI during routine clinical care, including well-child and acute visits. This screening occurs on paper in English or Spanish and is verbally asked through a certified interpreter if the parent/caregiver speaks another language or is unable to read. Household FI is screened using two validated questions, “In the past year, did you worry that your food would run out before you got money or Food Stamps to buy more?” and “In the past year, did the food you bought just not last and you didn’t have money to get more?” with dichotomous response options (yes or no) [Reference Hager, Quigg and Black14]. An affirmative response to either question is considered a positive screen. These questions are recommended by the American Academy of Pediatrics [15]. At the time of a clinic visit, the provider discusses the screening results with the family and enters the results into discrete data fields within the EHR. When a patient has a positive screen, the provider asks the family if they would like a bag of food from the clinic’s food pharmacy, referral to a food navigator, connection with federal nutrition programs, and/or a list of community resources.

Demographics

Patient’s age, sex, race, and ethnicity at the time of the visit were extracted from the EHR. Race and ethnicity, which were self-reported by parents/caregivers and recorded in the EHR, were combined into one variable and categorized as non-Hispanic White, non-Hispanic Black, Hispanic, or other (includes American Indian, Alaska Native, Asian, Native Hawaiian, Other Pacific Islander, Other, Patient Refused, Unknown).

Other Unmet Health-Related Social Needs Screening

As part of routine clinical care, the clinic also screens for other unmet health-related social needs over the prior 12 months including intimate partner violence, risk of homelessness, transportation difficulties, legal needs, caregiver coping concerns, and caregiver substance use. These needs are screened alongside FI. All screening results and resources provided are documented within the EHR (Supplementary Table 1).

Chronic conditions and healthcare Utilization

All problem lists and encounter diagnoses from visits occurring at the primary care clinic and any department across the medical center since 2012, when our EHR was adopted, were included. Diagnoses data included in the problem list or encounter diagnoses based on the respective International Classification of Diseases 10th Revision (ICD-10), were extracted, including attention-deficit hyperactivity disorder (ADHD), failure to gain weight, hypertension, asthma, eczema, prematurity, depression, and anxiety. Emergency department visits and hospitalizations were also extracted.

Anthropometrics

Children’s height and weight are recorded as standard of care at each clinic visit, and body mass index (BMI) percentiles were calculated from the encounter at which FI was measured. Underweight was defined as BMI of <5th percentile. Healthy weight was defined as BMI between ≥ 5 to<85th percentile. Overweight and obesity were defined as BMI of ≥ 85 to <95th, and ≥ 95th percentile, respectively.

Neighborhood information

We used the US Census Bureau’s 2010 American Community Survey (ACS) 5-year estimates to identify neighborhood information. The ACS includes neighborhood-level information at both the zip code level and the census block level including income, poverty, employment, and home ownership estimates [16]. Zip code predictors included total population count, number of housing units, median age, percentage 25 and older who had a high school education or less, median income, percentage of households with incomes under 100 percent of the federal poverty level in last 12 months, percentage of households where the person who lives in the household owns the house, percentage 16 years of age and older who work and commute 60 minutes or more to work in an area, percentage 16 years of age and older who are unemployed, percent minority, percentage of the area considered to be urban, and percentage of the area considered to be rural. Census block predictors included the same variables as zip code but measured at the smaller geographic level of census block. The home address of all patients in the health system is automatically geocoded by an automated system through the Wake Forest Clinical and Translational Science Institute’s Translational Data Warehouse. We merged geocode data from the EHR with zip code level and census block level data from the ACS.

Statistical analysis

The analysis cohort was created by merging the above data sources (demographics, anthropometrics, social needs, chronic conditions, and neighborhood information) by encounter number. The analysis cohort was randomly split at the patient level into 80% training set and 20% testing/validation set. Each patient could contribute more than one record to the analysis cohort. Various combinations of predictor variables and modeling approaches were used to train and validate prediction models. Demographic predictor variables included age in years, female sex, Black race, Hispanic ethnicity, other race and ethnicity (defined as non-black, non-Hispanic, and/or nonwhite), and BMI percentile. Twelve-month “look back” indicator variables included previous FI, intimate partner violence, risk of homelessness, transportation difficulties, legal needs, caregiver coping concerns, caregiver substance use, overweight/obesity, ADHD, failure to gain weight, hypertension, asthma, eczema, prematurity, depression, anxiety, hospitalizations, and emergency department visits. The look-back variables were initially coded as absent and were recoded as present if they appeared within the look-back window. The look-back window was expanded for the same set of variables to 18 and 24 months. Six different iterations of each logistic regression, random forest, and gradient-boosted model, respectively, were developed. The modeling approaches were defined by ACS census block variables with a 12, 18, or 24-month clinical data look-back period, or ACS zip code variables using 12, 18, or 24-month clinical data look-back period. All models included demographics.

Modeling approaches included logistic regression, random forest, and gradient-boosted machine. Default values of hyperparameters were used for random forest (500 trees, number of variables to split at each node equal to the rounded down square root of the number of predictors, minimum node size of 1) and gradient boosted machine (learning rate of 0.3, maximum tree depth of 6) [Reference Breiman17–Reference Chen, Benesty and Khotilovich20]. No variable selection was performed, as all variables considered are readily available in the EHR. All models were built on the training set using 10-fold cross-validation to estimate out-of-sample performance for the metrics of accuracy, area under the curve (C-statistic), precision-recall area under the curve, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). In the 10-fold cross-validation, each patient’s observations were uniquely mapped to only one of the folds to prevent overly optimistic estimates of performance. Final models from the training set were then evaluated on the test set. Test set evaluations included measures of discrimination and calibration. Model discrimination was measured using the C-statistic and compared between models using DeLong’s test [Reference DeLong, DeLong and Clarke-Pearson21,Reference Robin, Turck and Hainard22]. Model calibration was measured using the Hosmer–Lemeshow slope statistic [Reference Hosmer and Lemesbow23]. The Hosmer–Lemeshow statistic breaks the data into deciles of predicted FI and examines the relationship by decile between average predicted FI and observed rate of FI. A slope of 1 for a line connecting the deciles indicates a well-calibrated model.

Missing data were addressed in two ways. First, a single imputation was employed to fill in all missing data, using multiple imputations by chained equations [Reference van Buuren and Groothuis-Oudshoorn24]. As a sensitivity analysis, missing data were coded as a category [Reference Josse, Scornet and Varoquaux25]. Continuous variables were summarized using tertiles, with a fourth level added to reflect missing data. Categorical variables included an extra level to reflect missing, e.g., sex was coded as male, female, or missing. Additional sensitivity analyses included using only data from before the onset of the COVID-19 pandemic (visits occurring before 3/11/2020) and removing previous FI as a predictor from the model.

All analyses were conducted using R version 4.3.2 [26]. Machine learning prediction models were trained and validated using the R package “mlr3” [Reference Lang, Binder and Richter27]. All models were built and described using the principles outlined in the Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement [Reference Collins, Reitsma, Altman and Moons28] (Supplementary Table 2).