User data has been the internet’s greatest treasure trove since the advent of Google. LLM’s are perfectly set up to extract the most intimate data available from their users (“mental health” conversations, financial advice, …) which can be used against them in a soft way (higher prices when looking for mental health help) or they can be used to outright manipulate or blackmail you.

Regardless, there is no scenario in which the end user wins.