You do realize that people also have the right of free association? The government needs to meet a very high standard before it can deprive someone of life, liberty, or property. Members of the public may refuse to do business with a person, or socially ostracize them, based on whatever information they have available.