Yeah. The dunning kruger effect is a real problem here.

I saw a meme saying something like, gen AI is a real expert in everything but completely clueless about my area of specialisation.

As in… it generates plausible answers that seem great but they’re just terrible answers.

I’m a consultant I’m in a legal adjacent field. 20 years deep. I’ve been using a model from hugging face over the last few months.

It can save me time by generating a lot of boiler plate with references et cetera. However it very regularly overlooks critically important components. If I didnt know about these things then I wouldn’t know it was missing from the answer.

So really, it cant help you be more knowledgeable, it can only support you at your existing level.

Additionally, for complex / very specific questions, it’s just a confidently incorrect failure. It sucks that it cant tell you how confident it is with a given answer.