unspeakablehorror@thelemmy.club 1 month ago
Off with their heads! GO self-hosted, go local… toss the rest in the trash can before this crap gets a foothold and fully enshitifies
Going local is taxing on your hardware that is extremely expensive to replace. Hell, it could soon become almost impossible to replace. I genuinely don‘t recommend it.
Even if you HAVE to use LLMs for some reason, there are free alternatives right now that let Silicon Valley bleed money and they‘re quickly running out of it.
Cancelling any paid subscription probably hurts them more than anything else.
If LLM is tied to making you productive, going local is about controlling the means of production.
You aren’t supposed to run it on machine you work on anyway, do a server and send requests.
It’s not really taxing on your hardware unless you load and unload huge models all day or if your cooling is insufficient.
GO self-hosted,
So yours and another comment I saw today got me to dust off an old docker container I was playing with a few months ago to run deepseek-r1:8b on my server’s Intel A750 GPU with 8gb of VRAM. Not exactly top-of-the-line, but not bad.
I knew it would be slow and not as good as ChatGPT or whatever which I guess I can live with. I did ask it to write some example Rust code today which I hadn’t even thought to try and it worked.
But I also asked it to describe the characters in a popular TV show, and it got a ton of details wrong.
8b is the highest number of parameters I can run on my card. How do you propose someone in my situation run an LLM locally? Can you suggest some better models?
You are playing with ancient stuff that wasn’t even good at release. Try these:
A 4b model performing like a 30b model: huggingface.co/Nanbeige/Nanbeige4.1-3B
Google open source version of Gemini: huggingface.co/google/gemma-3-4b-it
Any suggestions on how to get these to gguf format? I found a GitHub project that claims to convert, but wondering if there’s a more direct way.
Well, not off to a great start.
To be clear, I think getting an LLM to run locally at all is super cool, but saying “go self hosted” sort of gloms over the fact that getting a local LLM to do anything close to what ChatGPT can do is a very expensive hobby.
I agree, it is a very expensive hobby, and it gets decent in the range 30-80b. However, the model you are using should not perform that bad, it seems that you might be hitting a config issue. Would you mind sharing the cli command you use to run it?
Honestly you pretty much don’t. Llama are insanely expensive to run as most of the model improvements will come from simply growing the model. It’s not realistic to run LLMs locally and compete with the hosted ones, it pretty much requires the economics of scale. Even if you invest in a 5090 you’re going to be behind the purpose made GPUs with 80GB VRAM.
Maybe it could work for some use cases but I rather just don’t use AI.
It goes down to number of vram / unified ram you have. There is no magic to make 8b perform like top tier subscription based LLMs (likely in 500b+ range, wouldn’t be surprised if trillions).
If you can get to 32b / 80b models, that’s where magic starts to happen.
GO self-hosted, go local
Last I checked it costed ~$6000 to run a high end LLM at terrible speeds and that was before the RAM price hike. And at the rate things are changing it might be obsolete in a few year. And I don’t have that much money either.
I’m going to stick with the free OpenCode Zen models for now and maybe switch to OpenCode Black or Synthetic or whatever when they stop being available and use the open weight models there.
I would, if I found even a remotely good usecase for LLMs. Would be useful for contextual search on a bunch of API documentation and books on algorithms, but I don’t want a sychophantic “copilot” or “assistant”, that does job so bad I would be fired for, all while being called ableist slurs and getting blacklisted from the industry.
I only have a need for RAG and it works well, wouldn’t say I need w copilot either. Everyone probably has some use case for this stuff but it’s gonna be different for everyone, the basic llm conversation model it’s really just a stepped up interactive Google
mushroommunk@lemmy.today 1 month ago
LLMs are already shit. Going local is still burning the world just to run a glorified text production machine
suspicious_hyperlink@lemmy.today 1 month ago
Having just finished getting an entire front end for my website, I disagree. A few years ago I would offshore this job to some third-world country devs. Now, AI can do the same thing, for cents, without having to wait for a few days for the initial results and another day or two for each revision needed
mushroommunk@lemmy.today 1 month ago
The fact you see nothing wrong with anything you said really speaks volumes to the inhumanity inherent with using “AI”.
suspicious_hyperlink@lemmy.today 1 month ago
Please enlighten me. I am working on systems solving real-world issues, and now I can ship my solutions faster, with lower costs. Sounds like a win-win for everyone involved except for the offshore employees that have to look for new gigs now
TherapyGary@lemmy.dbzer0.com 1 month ago
1000013638
angstylittlecatboy@reddthat.com 1 month ago
Do local LLMs really consume much more than a task like, playing a video game?