Beyond what everyone else has said, it has already been shown that LLMs have a chance of regurgitating training data, which means that someone’s personal data could get returned in a Bing Chat query.