Comment on Is anyone self-hosting their own AI?
twilightwolf90@lemmy.world 3 days ago
A quick glance shows that I’m in the minority. I grabbed a Bosgames M5 with 128GB UMA RAM when I discovered openclaw (at the time claudebot). I’m running Lemonade, which is a wrapper for llama.cpp, sd-cpp, flux, and others. I’m happy to have paid a “mere” $2400 compared to what similar systems cost now, and now I’m toying with fine tuning my own models. Though, I primarily run Hermes Agent paired with Qwen3.6 35b A3B (using a heretic ablitieration for 0 refusals).
mal3oon@lemmy.world 15 hours ago
Try gufo with qwen 3.8 flash next. Much much better than 3.6-35B.
twilightwolf90@lemmy.world 13 hours ago
How well does it handle concurrency? I could see this as a vast improvement on llamacpp, but I was already speccing out vLLM.