Comment

Comment on LLMs develop their own understanding of reality as their language abilities improve

wizardbeard@lemmy.dbzer0.com ⁨2⁩ ⁨months⁩ ago

Wow, that headline is significantly overstated. This has nothing to do with “understanding” or “reality”. Like holy shit this is an absurd leap from what they actually state in the abstract. Thanks OP for including it in the “preview” here.

Given a number of “starting states” and “ending states” of 2D grid based “environments”, an LLM is able to return increasing accurate “between states” as it is trained.

Neat! Not an understanding of reality. At the biggest stretch it is an “understanding” of an incredibly constrained problem space.

source

Sort:hotnew top

Hackworth@lemmy.world ⁨2⁩ ⁨months⁩ ago
I did some source digging to hopefully best address your observations. Science journalism (even when internal and likely done in concert with the authors) is fundamentally a game of telephone. But looking at the source papers:

They say it in an incredibly formal way, but they do seem to come to the conclusion that the LLM develops understanding. The paper makes that case within an incredibly narrow context, but it does include:

We anticipate that this technique may be generally applicable to a broad range of semantic probing experiments. We argue that the observed semantic content cannot be fully attributed to a retrieval-like process, and instead requires the LM to perform some degree of generalization over the semantics. More broadly, we see programs and their precise formal semantics as a promising direction for working toward a deeper understanding of the behavior of LMs, such as whether or how LMs acquire and use semantic representations of the underlying domain more generally.

With it now clear that the generalized case is not shown: the specific type of understanding that they have shown is non-trivial.

Conclusion: This paper presents empirical evidence that LMs of code can acquire the formal semantics of programs from next token prediction.

A foundational topic in the theory of programming languages, formal semantics (Winskel, 1993) is the study of how to formally specify the meaning of programs.

From Winskel: The Formal Semantics of Programming Languages provides the basic mathematical techniques necessary for those who are beginning a study of the semantics and logics of programming languages. These techniques will allow students to invent, formalize, and justify rules with which to reason about a variety of programming languages.

Also notable but unrelated: Jin and Rinard’s paper was supported, in part, by grants from the U.S. Defense Advanced Research Projects Agency (DARPA).

source
- MagicShel@programming.dev ⁨2⁩ ⁨months⁩ ago
  I mostly get what you’re saying, though I don’t have the requisite understanding to follow formal proofs, but if there is one thing I do know for certain, it’s that “understanding” is anthropomorphizing and shorthand for something that is very much not understanding in a human context at all.
  
  I get that it can be hard to find the right words to explain a some of these emergent phenomena, but I think it’s misleading to use words that make AI appear to have a thought process akin to anything we could understand as such—at least in settings where folks might not understand the shorthand as such.
  
  And maybe everyone here is aware of that, but it makes me uneasy, hence this comment to hopefully make that point.
  
  source
  - Hackworth@lemmy.world ⁨2⁩ ⁨months⁩ ago
    The paper is kind of saying that as well. I added a quote to the post to help set the context a bit more. As I understand it, they’ve shown that an LLM contains a model of its “world” (training data) and that this model becomes a more meaningful map of that “world” the
    
    source
  - technocrit@lemmy.dbzer0.com ⁨2⁩ ⁨months⁩ ago
    As someone who understands formal proofs, it’s completely misleading to conflate formalism with student pedagogy,
    
    source
    Hackworth@lemmy.world ⁨2⁩ ⁨months⁩ ago
    Genuine question: What evidence would make it seem likely to you that an AI “understands”? These papers are coming at an unyielding rate, so these conversations (regardless of the specifics) will continue. Do you have a test or threshold in mind?
    
    source