Does anyone run their own AI service?
If so,
Which one?
What hardware?
Do you have a way to access it from outside or from a phone?
And, if you’re open to sharing, what do you do with it?
I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don’t absolutely need specialized hardware to run and perform well enough for many purposes.
I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I’m not comfortable using them to help me with web searches and the such, because there’s no way that all this information won’t be going into a profile. This makes me want to have my own “helper” service.
Nighed@feddit.uk
Natanox@discuss.tchncs.de
curbstickle@anarchist.nexus 8 hours ago
As a service? No, not really.
I use llms for specific tasks, usually small models, for things like identification and tagging (immich, paperless), translation or transcription (whisper), etc. I have a bunch of larger models on a Mac mini and my work laptop for various testing purposes, and have used them to make summaries and other such things.
They aren’t great for search IMO, and search summaries can be quite bad.
I do find them useful to interrogate an unknown rs232 protocol or similar, though, which is where larger models come in handy.
Corporate LLMs are generally garbage to me not only because they are just data farming, but also because they really aren’t good at everything. They might claim it, but the reality is a tailored model is often the better approach.
Using openwebui as a front end for llama.cpp is a decent strategy, combined with sub-agent routing (so generic model hands off to tailored models then returns the right info). This can be useful for some scenarios.
A caveat on coding - IMO, they are mostly bad at it. If you are using it to generate a small part of something that needs to be repeated they are handy, but at scale they aren’t great, meaning for the entire project (or even an entire plugin or module, depending on complexity). Maintainability becomes a serious issue. While they have improved somewhat, there are limits to what they can do.
Macs with M series chips are good options for a hardware base since the memory is shared, and IMO is one of the most cost effective options over a dgpu with a ton of memory. A dgpu with a decent bit of memory (say 8-12GB) is a solid choice for a model thats larger (27b with 3 or 4b active as a mixture-of-experts model) especially if you have an iGPU for regular graphics tasks. Since its all shared on the M series macs, just get something that fits entirely in memory there.