This is the most obvious answer but given OP’s strong feelings on the matter, they presumably already thought this would be the best course of action and would already be doing it were it so easy. They didn’t say it directly but I think the problem they’re having is that their colleagues are not only themselves choosing to waste their time in this manner but because they are, they are expecting it of OP and it sounds like they’re having some kind of conflict about it that they’ve brought up with OP directly. I’m guessing some of them have complained to OP or indicated to them that they’re “supposed” to be ready by 12.30 even though they’re wrong.

This makes it difficult because in that scenario OP can’t simply plough on doing their job to the letter of the contract and just ignore the others, they have to choose to actively fight it out with them or just relent and start arriving early. It sounds like what they’re having the dilemma about is if it sounds reasonable to have that fight, given that although it feels important to OP, they recognise that it’s more about the principle of the matter than the objective scale of the problem. Seems like they recognise that 5 minutes of their time wouldn’t be that big a deal as their colleagues have concluded, making it seem petty or disruptive to have that fight and all the awkwardness that will ensue just for those 5 minutes, but the principle that they should have to give up any extra time at all, regardless how long, just because their workmates are doing it is troubling OP.

Depending on how whiny and motivated his colleagues are it might be advisable to pick their battles and let this go, but if they think they’re in a position to stand their ground and not have to suffer too many consequences and loss of standing with their workmates then of course they should, and even encourage others to do the same. I don’t think we have enough of an idea what their workplace is like to know which one of those situations they’re in.