Plant-based diets (PBD) have been found to be environmentally sustainable and beneficial for health. Observational research showed that higher plant-based diet quality improves health-related quality of life (HRQoL) in adult women, however this is unclear for older adults. This association may be due to anti-inflammatory properties of PBD. Older adults, prone to chronic inflammation, may therefore profit from PBD. We investigated the relation between PBD and HRQoL in older adults of both sexes and tested whether the effects are associated with circulating high-sensitivity C-reactive protein (hsCRP) levels. We used data of the population-based Lifelines Cohort Study (n = 6,635, mean age = 65.2 years) and a subsample in which hsCRP was measured (n = 2,251, mean age = 65.2 years). We applied a plant-based diet index measuring adherence to a healthful (hPDI) and an unhealthful (uPDI) plant-based diet based on food frequency questionnaires. The RAND-36 questionnaire was applied as measure of HRQoL, from which we derived physical and mental HRQoL. Older adults with the highest adherence to a hPDI had respectively 15% and 12% greater odds for high physical quality of life and mental quality of life. Meanwhile, higher adherence to uPDI was associated with respectively 16% and 13% lower odds for high physical and mental quality of life. An additive but no interactive effect of hsCRP on the association between PBD and HRQoL has been observed. Adherence to a healthful plant-based diet and circulating levels of inflammation are independently associated with physical and mental HRQoL. Mechanisms other than inflammation through which PBD could influence HRQoL may be explored in further research.