LLMs are also very good at convincing their users that they know what they are saying.

It’s what they’re really selected for. Looking accurate sells more than being accurate.

I wouldn’t be surprised if many of the people selling LLMs as AI have drunk their own kool-aid (of course most just care about the line going up, but still).