Chatgpt flat out hallucinates quite frequently in my experience. It never says “I don’t know / that is impossible / no one knows” to queries that simply don’t have an answer. Instead, it opts to give a plausible-sounding but completely made-up answer.

A good AI system wouldn’t do this. It would be honest, and give no results when the information simply doesn’t exist. However, that is quite hard to do for LLMs as they are essentially glorified next-word predictors. The cost metric isn’t on accuracy of information, it’s on plausible-sounding conversation.