Exactly. The real debate is on which parts should be off limits.

Most people can think of some speech that they consider so horrible that nobody should be allowed to say it.

People often try to hedge that position by arguing that they’re not even really infringing on anyone’s speech because their form of restriction doesn’t meet a sufficient threshold of censorship.