Even if it’s not entirely faked, you can instruct an AI to give you wrong answers to your questions. So unless you can see the entire conversation history, you can’t make any conclusions about a single response.

I imagine if somebody started a thread here that asked the same question, but said, “Wrong answers only,” people would find a lot of evidence that humans aren’t capable of figuring this out, either.