You can use Vulkan fairly easily as long as you have 8G vram
I for one would enjoy triggering your unskippable cutscenes in setting up local CPU based AI if it can work on Linux with an older amd card.
Don’t have funds for anything fancy, but would be interesting in playing around with it. Been wanting to get something like that setup for home assistant.
ag10n@lemmy.world 1 month ago
brucethemoose@lemmy.world 1 month ago
The key is which one, and how though.
For the really sparse models, you might be better off trying ik_llama.cpp, especially if you are targeting a ‘small’ quant.
Passerby6497@lemmy.world 5 weeks ago
Only got 4G vram, unfortunately
SabinStargem@lemmy.today 1 month ago
If you just want an easy way to setup AI on Windows or Linux, KoboldCPP is my recommendation for your backend. It supports the GGUF format, which allows you to use both RAM and VRAM simultaneously. It won’t be the fastest thing, but it is easy enough to setup, with a bundled GUI for prep and actual usage. Through the IP address it gives, you can hook the backend into a frontend of choice.
brucethemoose@lemmy.world 1 month ago
Plenty of folks do AMD. A popular ‘homelab’ setup is 32GB AMD MI50s. Even Intel is fine these days!
But what’s your setup, precisely? CPU, RAM, and GPU.
afk_strats@lemmy.world 1 month ago
I have a MI50/7900xtx gaming/ai setup at homr which in i use for learning and to test out different models. Happy to answer questions
Passerby6497@lemmy.world 1 month ago
Looks like I’m running an AMD Ryzen 5 2600 CPU, AMD Radeon RX 570 GPU, and 32GB RAM
brucethemoose@lemmy.world 1 month ago
Mmmmm… I would wait a few days, and try a GGUF quantization of Kimi Linear once its better supported: huggingface.co/…/Kimi-Linear-48B-A3B-Instruct
Otherwise you can mess with Qwen 3 VL now, in the native llama.cpp UI: huggingface.co/…/Qwen3-VL-30B-A3B-Instruct-UD-Q4_…
If you’re interested, I can work out an optimal launch command. But to be blunt, with that setup, you’re kinda better off using free LLM APIs with a local chat UI.
Passerby6497@lemmy.world 5 weeks ago
Thanks for the info. I would like to run locally if possible, but I’m not opposed to using API and just limiting what I surface.