Comment on playing around with desktop ai
brucethemoose@lemmy.world 4 weeks ago
First, I would suggest using Deepseek 14B at least. The jump from 7B to 14B is huge, especially with the “bad” defaults ollama uses.
FYI you can utilize vram much more efficiently than ollama using exllama or base llama.cpp with a good quantization.
Secondly, Deepseek is good, but I would suggest trying “anti-slop” models that are trained to write better prose. My go to is EVA-Gutenberg since it’s trained on a base model.
PixelPilgrim@lemmings.world 4 weeks ago
I’ll have to see if my setup can handle 14b. I havent found eBay Gutenberg but I found llama 3 guttenberg.
I’ll have to check out exllama, but I only have 1 gig of vram