There’s some nuance.
Using LLMs to augment data, especially for fine tuning (not training the base model), is a sound method. The Deepseek paper using, for instance, generated reasoning traces is famous for it.
Another is using LLMs to generate logprobs of text, and train not just on the text itself but on the *probability a frontier LLM sees in every ‘word.’ This is called distillation, though there’s some variation and complication.
But yes, the “dumb” way, aka putting data into a text box and asking an LLM to correct it, is dumb and dumber, because:
-
You introduce some combination of sampling errors and repetition/overused word issues, depending on the sampling settings
-
You possibly pollute your dataset with “filler”
-
In Musks specific proposition, it doesn’t even fill knowledge gaps the old Grok has.
In other words, Musk has no idea WTF he’s talking about.
hansolo@lemmy.today 9 months ago
Musk probably heard about “synthetic data” training, which is where you use machine learning to create thousands of things that are typical-enough to be good training data. Microsoft uses it to take documents users upload to Office365, train the ML model, and then use that ML output to train an LLM so they can technically say “no, your data any used to train an LLM.” Because it trained the thing that trained the LLM.
However, you can’t do that with LLM output and stuff like… History. WTF evidence and documents are the basis for the crap he wants to add? The hallucinations will just compound because who’s going to cross-check this other than Grok anyway?