Why wouldn’t you trust a local model? I mean sure they are less intelligent but if you dont expose it publicly you should be good to go
Comment on Is anyone self-hosting their own AI?
lka1988@lemmy.dbzer0.com 1 day ago
I wouldn’t mind running a model at home, mostly to help document my setup. But I just don’t trust it. So no.
andrewoidxyz@lemmy.world 1 day ago
lka1988@lemmy.dbzer0.com 1 day ago
I don’t trust it to be accurate; I’d have to reference the sources it pulled from. Extra work for little payoff.
andrewoidxyz@lemmy.world 23 hours ago
Got it, makes sense. You’d probably have to do some benchmarking first. Depending on how large the source catalog is.
mal3oon@lemmy.world 14 hours ago
It depends on your hardware. But right now the “cheapest” hardware you can buy for a usable setup, is the strix halo (128GB), and run gufo with qwen 3.8 flash next (Q4XL). It is sonnet 5 level. Very good speed (both prefill 1000+ and decode 50+), with up to 256k context.
I have it set up with omp + searxng. Sometimes with hermes. Really good! It changes the game for local setup.
lka1988@lemmy.dbzer0.com 13 hours ago
googles
Jesus christ
mal3oon@lemmy.world 11 hours ago
Just get the cheapest OEM, Bosgame M5. I know “cheapest” is in quotes. But again, it is the cheapest 128GB unified vram device you can get right now. I got mine at pre-orders, so yes, more than a year and a half ago, prices were a steal. Also, great desktop (mini) PC overall and very efficient, even if you don’t want to do AI.
lka1988@lemmy.dbzer0.com 6 hours ago
I have like 150GB of DDR3 spread over multiple 2/4/8/16GB sticks, I wonder if I can get that working for something like this 🤔