GPU with a ton of vran is what you need, BUT
An alternate solution is something like a Mac mini with an m series chip and 16gb of unified memory. The neural cores on apple silicon are actually pretty impressive and since they use unified memory the models would have access to whatever the system has.
I only mention it because a Mac mini might be cheaper than GPU with tons of vram by a couple hundred bucks.
And it will sip power comparatively.
mozz@mbin.grits.dev 4 months ago
You’re the only one talking sense and you are sitting here with your 2 upvotes
The AI company business model is 100% unsustainable. It’s hard to say when they will get sick of hemorrhaging money by giving away this stuff more or less for free, but it might be soon. That’s totally separate from any legal issues that might come up. If you care about this stuff, learning about doing it locally and having a self hosted solution in place might not be a bad idea.
But upgrading anything aside from your GPU+VRAM is a pure and unfettered waste of money in that endeavor.