Yes I just thought about the same.

The idea was that cooling stuff doesnt involve manually moving it.

But entropy is the thing here. Moving energy against the path of natural entropy.