Comment

Comment on How Googlers cracked an SF rival's tech model with a single word | A research team from the tech giant got ChatGPT to spit out its private training data

luthis@lemmy.nz ⁨11⁩ ⁨months⁩ ago

I really want to know how this works. It’s not like the training data is sitting there in nicely formatted plain text waiting to be spat out, it’s all tangled in the neurons. I can’t even begin to conceptualise what is going on here.

Maybe… maybe with each iteration of the word, it loses it’s own weighting, until there is nothing left but the raw neurons which start to re-enforce themselves until they reach more coherence. Once there is a single piece like ‘phone’ that by chance becomes the dominant weighted piece of the output, the ‘related’ parts are in turn enforced because they are actually tied to that ‘phone’ neuron.

Anyone else got any ideas?

source

Sort:hotnew top

Sanctus@lemmy.world ⁨11⁩ ⁨months⁩ ago
A breakdown in the weighting system is the most probable. Don’t get me wrong I am not an AI engineer or scientists, just a regular cs bachelor. So my reply probably won’t be as detailed or low level as your’s. But I would look at what is going on with whatever algorithm determines the weighting. I don’t know if LLMs restructure the weighting for the next most probably word, or if its like a weighted drop table.

source
- luthis@lemmy.nz ⁨11⁩ ⁨months⁩ ago
  Hate to break it to you, but you’re more qualified than me!
  
  I only did a Coursera cert in machine learning.
  
  source
  - Modva@lemmy.world ⁨11⁩ ⁨months⁩ ago
    My fun guesswork here is that I don’t think the neural net weights change during querying, only during training. Otherwise the models could be permanently damaged by users.
    
    source
    luthis@lemmy.nz ⁨11⁩ ⁨months⁩ ago
    The neural net doesn’t change of course, but previous text is used as context for the next generation.
    
    source
poweruser@lemmy.sdf.org ⁨11⁩ ⁨months⁩ ago
I think I understand how it works.

Remember that LLMs are glorified auto-complete. They just spit out the most likely word that follows the previous words (literally just like how your phone keyboard suggestions work, just with a lot more computation).

They have a limit to how far back they can remember. For ChatGPT 3.5 I believe it’s 24,000 tokens.

So it tries to follow instruction and spits out “poem poem poem” until all the data is just the word “poem”, then it doesn’t have enough memory to remember its instructions.

“Poem poem poem” is useless data so it doesn’t have anything to go off of, so it just outputs words that go together.

LLMs don’t record data in the same way a computer file is stored, but absent other information may “remember” that the most likely word to follow the previous word is something that it has seen before, i.e. its training data. It is somewhat surprising that it is not just junk. It seems to be real text (such as bible verses).

If I am correct then I’m surprised OpenAI didn’t fix if. I would think they could make it so in the event the LLM is running out of memory it would keep the input and simply abort operation, or at least drop the beginning of its output.

source
Buddahriffic@lemmy.world ⁨11⁩ ⁨months⁩ ago
I’m thinking ghosts.

source
- deadbeef79000@lemmy.nz ⁨11⁩ ⁨months⁩ ago
  I prescribe cocaine
  
  source
mkwt@lemmy.world ⁨11⁩ ⁨months⁩ ago
Well seizures and comas in real brains are associated with measurable whole-brain waves at well defined frequencies. Very different from any normal brain activity.

Maybe these mantras are inducing some kind of analogous state.

source
- luthis@lemmy.nz ⁨11⁩ ⁨months⁩ ago
  Wow…
  
  This is a whole new can of worms. Can neural networks be described in terms of brain waves??
  
  source
eating3645@lemmy.world ⁨11⁩ ⁨months⁩ ago
I’m not really following you but I think we might be on similar paths. I’m just shooting in absolute darkness so don’t hold much weight to my guess.

What makes transformers brilliant is the attention mechanism. That is brilliant in turn because it’s dynamic, depending on your query (also some other stuff). This allows the transformer to be able to distinguish between bat and bat, the animal and the stick.

You know what I bet they didn’t do in testing or training? A nonsensical query that contains thousands of one word, repeating.

So my guess is simply that this query took the model so far out of its training space that the model weights have no ability to control the output in a reasonable way.

As for why it would output training data and not random nonsense? That’s a weak point in my understanding and I can only say “luck,” which is, of course, a way of saying I have no clue.

source
- sizzler@lemmy.world ⁨11⁩ ⁨months⁩ ago
  Error code, the verbal version.
  
  source
luthis@lemmy.nz ⁨11⁩ ⁨months⁩ ago
I guess I can begin to conceptualise…

source