Memorizing everything is impressive for a human.
It’s less impressive for a computer.
Comment on ChatGPT outperforms undergrads in intro-level courses, falls short later
festus@lemmy.ca 3 weeks ago
Not at all surprising. ChatGPT ‘knows’ a course’s content insofar as it’s memorized the textbook and all the exam questions. Once you start asking it questions it’s never seen before (more likely for advanced topics that don’t have a billion study guides and tutorials for) it falls short, even for basic questions that’d just require a bit of additional logic.
Mind you, memorizing everything is impressive and can get you a degree, but when tasked with a new problem never seen before ChatGPT is completely inadequate.
conciselyverbose@sh.itjust.works 3 weeks ago
kromem@lemmy.world 3 weeks ago
This is incorrect as was shown last year with the Skill-Mix research:
Furthermore, simple probability calculations indicate that GPT-4’s reasonable performance on k=5 is suggestive of going beyond “stochastic parrot” behavior (Bender et al., 2021), i.e., it combines skills in ways that it had not seen during training.
TheFriar@lemm.ee 3 weeks ago
Right? Can students use the internet on this test? Because the LLMs have the entire internet to search for the answers, and I guarantee you those textbooks and exam questions are online and searchable.
Vorticity@lemmy.world 3 weeks ago
I wonder how undergrads would do on the same exams given unlimited time and internet access but with LLMs blocked. That’s essentially what the LLMs have.
technocrit@lemmy.dbzer0.com 3 weeks ago
The LLMs blocked themselves?
Vorticity@lemmy.world 3 weeks ago
I don’t think they really query one another. Maybe they do though?