There is very strong in vitro and observational data showing a protective association between dairy and calcium intake and colorectal cancer. Vitamin D status and possibly genetics also seem to influence this link.
A few RCTs suggest calcium + vit. D supplements help prevent intestinal polyp recurrence. We lack RCT evidence that this in turn stops colon cancer, and if so it is more likely to work in at-risk rather than general populations.