Or they just have too much money.
Comment on Mark Zuckerberg indicates Meta is spending billions of dollars on Nvidia AI chips
31337@sh.itjust.works 11 months agoThe equivalent of 600k H100s seems pretty extreme though. IDK how many OpenAI has access to, but it’s estimated they “only” used 25k to train GPT4. OpenAI has, in the past, claimed the diminishing returns on just scaling their model past GPT4s size probably isn’t worth it. So, maybe Meta is planning on experimenting with new ANN architectures, or planning on mass deployment of models?
ExLisper@linux.community 11 months ago
FaceDeer@kbin.social 11 months ago
Which will be solved by them spending it.
NotMyOldRedditName@lemmy.world 11 months ago
Would that be diminishing returns on quality, or training speed?
If I could tweak a model and test it in an hour vs 4 hours, that could really speed up development time?
31337@sh.itjust.works 11 months ago
Quality. Yeah, using the extra compute to increase speed of development iterations would be a benefit. They could train a bunch of models in parallel and either pick the best model to use or use them all as an ensemble or something.
My guess is that the main reason for all the GPUs is they’re going to offer hosting and training infrastructure for everyone. That would align with the strategy of releasing models as “open” then trying to entice people into their cloud ecosystem. Or, maybe they really are trying to achieve AGI as they state in the article. I don’t really know of any ML architectures that would allow for AGI though (besides the theoretical, incomputable AIXI).
TropicalDingdong@lemmy.world 11 months ago
Might be a bit of a tell that they think they have something.
qupada@kbin.social 11 months ago
The estimated training time for GPT-4 is 90 days though.
Assuming you could scale that linearly with the amount of hardware, you'd get it down to about 3.5 days. From four times a year to twice a week.
If you're scrambling to get ahead of the competition, being able to iterate that quickly could very much be worth the money.