Comment on Selfhosted & AI
scrubbles@poptalk.scrubbles.tech@poptalk.scrubbles.tech 3 weeks ago I’ll have to play around with mine then, because I’ve had not great luck with it, or at least very disappointing. The CPU offloading is fairly slow, but maybe I should try tweaking more
brucethemoose@lemmy.world 3 weeks ago
Be sure to try the ik_llama.cpp fork. Basically, it specializes in MoE CPU offloading on Nvidia cards, and more efficient quantization types than mainline llama.cpp:
github.com/ikawrakow/ik_llama.cpp/
And see this repo for specific 3090 configs: github.com/noonghunna/club-3090
Honestly I should just write up my general setup in this community too.