That’s the thing, companies (especially startups) have seen the price difference and many have elected to buy up consumer-grade cards.
Comment on AI Is Starting to Look Like the Dot Com Bubble
Mossheart@lemmy.ca 1 year agoI figured the gear they were using was orders of magnitude heftier than those cards. Stuff like the h100 cards that go for the price of a loaded SUV.
hark@lemmy.world 1 year ago
Baylahoo@sh.itjust.works 1 year ago
I’m not an expert just parroting info from Jayz2cents (YouTuber), but the big AI groups are using $10,000 cards for their stuff. Individuals or smaller companies are taking/going to take what’s left with GPUs to do their own development. This could mean another GPU shortage like the mining shortage andi would assume another bust would result in a flooded used market when it happens. Could be wrong, but he’s been correct pretty consistently with his predictions of other computer related stuff. Although, 10K is a little bit less than your fully loaded SUV example.
headmetwall@lemmy.dbzer0.com 1 year ago
They are, but training models is hard and inference (actually using them) is (relatively) cheap. If you make a a GPT-3 size model you don’t always need the full H100 with 80+ gb to run it when things like quantization show that you can get 99% of its performance at >1/4 the size.
Thus NVIDIA selling this at 3k as an ‘AI’ card, even though it wont be as fast.