Wondering if changing my regular dairy and cheese for low fat versions would be enough.

Probly not.

I started getting a lot of Vitamin D (midday sun in the summer, no sunscreen)

Then I did intermediate fasting. Only eting diner.