This is why I said I wasn’t sure how AI works behind the scenes. But I do know that logic isn’t difficult. Just to not fuck around between us. I have a CS background. Only saying this because I think you may have it as well and we can save some time.
It makes sense to me that logic is something AI can parse easily. Logic in my mind is very easy something that can tokenize text. Wouldn’t the difficulty be if the AI has the right context.
antonim@lemmy.dbzer0.com 3 weeks ago
Logic requires abstracting the argumentative form from the literal linguistic content and then generalising it, just how like math is done properly when you work with numbers and not just with sentences such as “two apples and three apples is five apples” (such abstraction in practice allows far more powerful and widely applicable operations than dealing with individual linguistic expressions; if you’ve ever solved very complex truth trees you’ll know how they allow streamlining and solutions that would be practically impossible to do if you had only the ordinary linguistic expression of the same problem). Logic doesn’t operate with textual tokens but with logical propositions and operators. “Difficulty” is not a meaningful term here, a tool is either technically capable of doing something (more or less successfully) or it isn’t.
That LLMs aren’t capable of this sort of precision and abstraction is shown by the OP link as well as the simple fact that chatbots used to be extremely bad at math (which is now probably patched up by adding a proper math module, rather than relying on the base LLM - my assumption, at least).
As for trying more examples of looking for logical fallacies, I tried out three different types of text. Since you say context is important, it’s best to take only the beginning of a text. One text I tried is the opening of the Wikipedia article on “history”, which ChatGPT described like this: “The passage you’ve provided is an informative and largely neutral overview of the academic discipline of history. It doesn’t make any strong arguments or persuasive claims, which are typically where logical fallacies appear.” It then went on to nitpick about some details “for the sake of thorough analysis”, but basically had no real complaints. Then I tried out the opening paragraph of Moby-Dick. That’s a fictional text so it would be reasonable to reject analysing its logical solidity, as GPT already did with the WP article, but it still tried to wring out some “criticism” that occasionally shows how it misunderstands the text (just as it misunderstood a part of my comment above). Finally, I asked it to find the fallacies in the first four paragraphs of Descartes’ Meditations on First Philosophy, which resulted in a criticism that was based on less logically rigid principles than the original text (accusing Descartes of the “slippery slope fallacy”).
I’ll post the full replies below.