Comment

Comment on Meta’s star AI scientist Yann LeCun plans to leave for own startup

nymnympseudonym@piefed.social ⁨3⁩ ⁨weeks⁩ ago

Animal brains have pliable neuron networks and synapses to build and persist new relationships between things. LLMs do not. This is why they can’t have novel or spontaneous ideation

This Nobel prize winner seems to disagree with you.

Neural nets do indeed learn new relationships. Maybe you are thinking of the fact that most architectures require training to be a separate process from interacting; that is not the case for all architectures.

source

Sort:hotnew top

just_another_person@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
From your own linked paper:

To design a neural long-term memory module, we need a model that can encode the abstraction of the past history into its parameters. An example of this can be LLMs that are shown to be memorizing their training data [98, 96, 61]. Therefore, a simple idea is to train a neural network and expect it to memorize its training data. Memorization, however, has almost always been known as an undesirable phenomena in neural networks as it limits the model generalization [7], causes privacy concerns [98], and so results in poor performance at test time. Moreover, the memorization of the training data might not be helpful at test time, in which the data might be out-of-distribution. We argue that, we need an online meta-model that learns how to memorize/forget the data at test time. In this setup, the model is learning a function that is capable of memorization, but it is not overfitting to the training data, resulting in a better generalization at test time.

Literally what I just said. This is specifically addressing the problem I mentioned, and goes on further to exacting specificity on why it does not exist in production tools for the general public (it’ll never make money, and it’s slow, honestly). In fact, there is a minor argument later on that developing a separate supporting system negates even referring to the outcome as an LLM, and the supported referenced papers linked at the bottom dig even deeper into the exact thing I mentioned on the limitations of said models used in this way.

source