It’s mostly an artefact from how we measure it.

Chocolate demand is rising, which means cocoa demand is rising. It grows best in really warm places, where there’s plenty of readily available mountainous land that’s currently covered by old growth forest. They farm cocoa very unsustainably, by illegally logging the land, growing beans for 5 to 8 years, and moving on (following the illegal loggers)

That land use change drives climate impact of chocolate. Milk drives the rest.