Thanks for the info. Yeah, I was wondering what kind of hardware you’d need to host LLMs locally with decent performance and your post clarifies that. I doubt many people would have the kind of hardware required.
Comment on Chatgpt shared link searchable
BreadstickNinja@lemmy.world 1 day agoYes, Ollama or a range of other backends (Ooba, Kobold, etc.) can run LLMs locally. Huggingface has a huge number of models suited to different tasks like coding, storywriting, general purpose, and so on. If you run both the backend and frontend locally, then no one monetizes your data.
The part I’d argue that the previous poster is glazing over a little bit is performance. Unless you have an enterprise-grade GPU cluster sitting in your basement, you’re going to make compromises on speed and/or quality relative to the giant models that run on commercial services.
puck@lemmy.world 1 day ago
tal@lemmy.today 1 day ago
It’s also going to cost more, because you almost certainly are only going to be using your hardware a tiny fraction of the time.
BreadstickNinja@lemmy.world 1 day ago
Possibly, yes. There are models that will run on consumer-grade GPUs that you might already have or might have purchased anyway, where you might say there’s no incremental cost. But the issue is that the performance will be limited. The models are forgetful and prone to getting stuck in loops of repeated phrases.
So if instead you custom-build a workstation with two 5090s or a Pro 6000 or something that pushes you up to the 100 GB VRAM tier, then absolutely, just as you said you’ll be spending thousands of dollars that probably won’t pay back relative to renting cloud GPU time.