So, you're basically asking why we have a judicial system.

If you look at world history pretty much every human society comes up with a code of what is okay and what isn't, and once they get to a certain size they start standardizing punishments as well.

The reason for this is because it works a lot better than ad hoc justice which is too reliant on variables like who was there that day.