Comment on Bill Gates feels Generative AI has plateaued, says GPT-5 will not be any better
KISSmyOS@lemmy.world 11 months agoAI trained on generative data
This has already been shown to degrade the output very quickly.
I think the wall that generative AI is hitting is the lack of more training data. All the web has been scraped to get it to where it is today, more and more content on the web is itself generated by AI and therefore not only useless but harmful if used as training data.
fruitycoder@sh.itjust.works 11 months ago
Orca 2 is an example of an opensource model that was built to better collect and build on synthetic data: microsoft.com/…/orca-progressive-learning-from-co…
The case I think being made is that building LFMs on the Internet gets you closer to an average internet users level of our put, using reinforcement learning you can further curate the outputs, then finally using these techniques you can generate even tighter high quality models.
It’s interesting stuff for sure.