Submitted 5 weeks ago by Brylant@discuss.online to selfhosted@lemmy.world
Why do people host LLMs at home when processing the same amount of data from the internet to train their LLM will never be even a little bit as efficient as sending a paid prompt to some high quality official model?
inb4 privacy concerns or a proof of concept this is out of discussion, I want someone to prove his LLM can be as insightful and accurate as paid one. I don’t care about anything else than quality of generated answers
Privacy would be the main concern. Every single one of your words, documents, pictures will probably end up in some large database over at OpenAI. I don't like that at all. And as a company for example, it might be against the law to share some information about clients with third parties.
Then you don't get any of the freedoms we got with Free Software. It's a service you rely on with very little opportunities to customize, or look inside and tinker. There is little control for the user whatsoever.
And since "the internet is for porn"... We also have a noteworthy community doing those kinds of things. And well... go ahead and ask the big services to generate a lewd story. Most of them even refuse to write a murder mystery story for me, instead they'll lecture me on how it is not ethical to murder someone. So there are use-cases where paid inference doesn't do it.
Why do people host LLMs at home when processing the same amount of data from the internet to train their LLM will never be even a little bit as efficient as sending a paid prompt to some high quality official model?
inb4 privacy concerns or a proof of concept this is out of discussion, I want someone to prove his LLM can be as insightful and accurate as paid one. I don’t care about anything else than quality of generated answers
If you ask other people for their reasoning and opinions, it doesn’t really make any sense to put something “out of the discussion”, does it? :P
But no, if you have no qualms about sharing your innermost feelings, sexual preference or illegal plans with those that have an explicit desire to exploit that information then there is little reason to attempt something as complicated and wasteful as self-hosting your own LLMs.
want someone to prove his LLM can be as insightful and accurate as paid one.
The full DeepSeek model is available for download, and should generate about the same quality answers as the official one, with the bonus of less censorship. I pretty trivially got it to talk about the Tiananmen Square, and they can’t even ban me for it.
That said, that’s rarely the point. It’s usually because you can, a cost saving measure, sometimes you plainly just don’t need a good model, sometimes you want privacy, sometimes you need privacy at the cost of quality.
If your business is shoving customer reviews into a model, you really don’t need the best model for it to tell you how angry the customer is.
Personally I just do it for fun and because I can. Sometimes you just do things for no other reason than because you can.
Poor internet connection/no internet at all, network latency too high for their needs, specific fine tuned LLM ?
Off course, main reason is privacy. My company host its own GPT4 chatbot, and forbid us to use public ones. But I suppose there are other legit use case to host its own LLM.
I want someone to prove his LLM can be as insightful and accurate as paid one.
I mean, you can train a model that’s domain-specific that some commercial provider doesn’t have a proprietary model to address. A model can only store so much information, and you can choose to weight that information towards training on what’s important to you. Or providers may just not offer a model in the field that you want to deal with at all.
But I don’t think that, for random individual user who just wants a general-purpose chatbot, he’s going to get better performance out of something self-hosted. Probably it’ll cost more for the hardware, since the hardware isn’t likely to be saturated and probably will be shared.
I think that the top reason for wanting to run an LLM model locally is the one you ruled out: privacy. You aren’t leaking information to someone’s computers.
Some other possible benefits:
Because you can guarantee access to the computational hardware. If my Internet connection goes down, neither does whatever I’m doing with the LLM.
Latency isn’t a factor, either from the network or shared computational systems. Right now, I don’t have anything that has much by way of real-time constraints, but I’m confident that applications will exist.
Afaik most people who seriously want to have a local LLM start of off a pre-trained one from the internet…?
You can control the electricity source (renewable) at home and cool your system without wasting drinking water.
Because they want to. You can get your proofs yourself.
Because you don’t train your self-hosted LLM.
As a result you only pay for the electricity of computing your tokens (your request), this can be especially reasonable if the same machine also does local game streaming and or transcoding, and thus already has the requirements to host a LLM.
If you don’t have rather unreasonable means, your local LLM is just very much more limited in parameters (size), and will not be as good as other, much larger models.
Privacy, Ethics and personal interest usually are the largest drivers from what I can tell.
I want someone to prove his LLM can be as insightful and accurate as paid one.
Are you using LLMs as search engines?
Bold.
I use Gemma, LLama 3.2, and Deepseek to either fix formatting, summarize documentation to give me commands for Linux software, and write simple code structure for me to refine into working code.
Sure it takes longer to generate than a cloud compute would, but 1) privacy obviously, 2) this feels better environmentally. I actually don’t know if that’s true, but it objectively touches less computers for such simple tasks. It would be wasteful of infrastructure to do it over the web.
Probably best asking this in lemmy.world/c/localllama@sh.itjust.works
What you ask for is impossible to prove, let alone test, without a rigorous definition of “insightful” or “accurate”.
Heh, you shouldn’t be paying for LLMs. Gemini 2.5 Pro is free, and so are a bunch of great API models.
I have technical reasons for running local models (instance cached responses, constrained grammar, logprob output, fine tuning), and I can help you set that up if you want, but TBH I am not going into a long technical proof of why that’s advantageous unless you really want to try this all yourself.
100% you don’t have to train a thing ollama uses open availability models. They many are decent, the best use a lot of ram/vram.
I want someone to prove his LLM can be as insightful and accurate as paid one.
Why would someone do that for you when they’re happy using their local LLM?
if you wish to get roasted with additional specifics, see !localllama@sh.itjust.works
dgriffith@aussie.zone 5 weeks ago
“Why do people do X, when in my opinion if you disregard the two top reasons for doing X, it’s pointless? Prove to me that it would be better!?”
Brylant@discuss.online 5 weeks ago
I’m not saying in advance that it’s pointless, it’s just your far-fetched preconception. I want to know if there are any further arguments for hosting LLMs besides the two I ruled out in advance for being too obvious. Furthermore, I think that since you moved so quickly to mockery, there are no other reasons than privacy and tinkering
TheJesusaurus@sh.itjust.works 5 weeks ago
Why would you need a reason other than privacy? Or for that matter any reason you want or nobe at all.
papertowels@mander.xyz 5 weeks ago
The self hosted community is probably the worst place you could ask about something while ruling out “privacy and tinkering”. Those words might as well be the motto here.