The ability to physically harm someone is only relevant if a) one chooses to believe someone will harm or then, or might or b) someone expresses that they will harm the other person, or might.

That is to say, “a person you don’t trust”. Implicitly you trust everyone in your society, more or less. Or in your “city”, if you will.

Just because you don’t like something doesn’t mean it doesn’t exist. Even if a power dynamic is completely equal, it still exists. It’s just balanced.

The ability to commit violence is at the very core of our civilization. Literally. There’s a reason we still call them POLICE officers. They’re the only people “in the city” (“polis” as in Akropolis, Annapolis, Marioupolis, etc) allowed to physically harm other people, and that is what makes them literally powerful.

Your reasoning only applies when you’re already in the context of “we’re not allowed to do violence or each other or the police will come and threaten me with violence unless I obey them”.

en.wikipedia.org/wiki/Monopoly_on_violence