How is this surprising? We know that part of LLM training is being rewarded for finding an answer that satisfies the human. It doesn't have to be a correct answer, it just has to be received well. This doesn't make it better, but it makes it more marketable, and that's all that has mattered since it took off.