Comment on Selfhosted & AI
scrubbles@poptalk.scrubbles.tech 6 days agoI’ll have to play around with mine then, because I’ve had not great luck with it, or at least very disappointing. The CPU offloading is fairly slow, but maybe I should try tweaking more
brucethemoose@lemmy.world 6 days ago
Be sure to try the ik_llama.cpp fork. Basically, it specializes in MoE CPU offloading on Nvidia cards, and more efficient quantization types than mainline llama.cpp:
github.com/ikawrakow/ik_llama.cpp/
And see this repo for specific 3090 configs: github.com/noonghunna/club-3090
Honestly I should just write up my general setup in this community too.