"The relationship between saturated fat and cholesterol to weight gain and cardiovascular disease is a large and well-documented one," Sroufe says. "That's easily addressed by reducing dairy."
Subsequent weight loss and improved cardiovascular health, he says, are often accompanied by other benefits.
"One of the first things you see for men is restored sexual function," he says. "That's a big one. Any time you reduce your cholesterol and fat intake, you're going to see myriad benefits."
The trick is replacing dairy with nutrient-dense foods that are also lower in saturated fat and calories.
The critical nutrients to make room for are calcium, potassium, protein and vitamin D, Frechman says. One approach is to stock up on supplements and start checking labels to see what products are fortified. Orange juice, cereals, bread products and milk alternatives are frequently fortified with calcium and vitamin D.
A healthier approach, she says, is to look for foods that contain such nutrients. Beans, grains, most vegetables and some fruits are also rich sources.
For potassium, Frechman recommends bananas, avocados, potatoes, tomatoes, oranges, kiwis, papayas and mangoes.
Calcium-rich foods include dark green, leafy veggies (spinach, kale, bok choy, mustard greens, broccoli); almonds; beans (edamame, kidney beans); sardines; and canned salmon, particularly with the bones mashed into the meat.
For vitamin D, salmon and other fish are also good sources.
As for protein: "Any type of nut butter," Frechman says. "Beans, eggs, tofu, meat, fish, peas, spinach. You'd be surprised how much protein vegetables have. Fruit has hardly any."
A dairy-free diet doesn't have to feel restrictive, Sroufe says.
"It's not just about giving something up," he says. "It's about increasing variety in our diets."
And possibly feeling better along the way.
"Since milk offers us no guarantee of bone health, and it really wasn't intended for us in the first place, and you're looking at a payload of fat, hormones and calories," Keon says, "then it's well worth taking a look at."
But what is dairy, exactly?
Dairy is any food product derived from cow's milk. Aside from the milk itself, dairy items include butter, most cheeses, yogurt, whey protein and ice cream, to name a few. Read the ingredients to be sure your food products do not contain milk products if you're trying to avoid dairy.