Comment on Smaug-72B-v0.1: The New Open-Source LLM Roaring to the Top of the Leaderboard
miss_brainfarts@lemmy.blahaj.zone 7 months agoI’m currently playing around with the Jan client, which uses the nitro engine. I think I need to read up on it more, because when I set the ngl value to 15 in order to offload 50% to GPU like the Jan guide says, nothing happens. Though that could be an issue specific to Jan.
Fisch@lemmy.ml 7 months ago
Maybe 50% GPU is already using too much VRAM and it crashes. You could try to set it to 0% GPU and see if that works.
miss_brainfarts@lemmy.blahaj.zone 7 months ago
I may need to lower it a bit more, yeah. Though when I try to to use offloading, I can see that vram usage doesn’t increase at all.
When I leave the setting at its default 100 value on the other hand, I see vram usage climb until it stops because there isn’t enough of it.
So I guess not all models support offloading?
General_Effort@lemmy.world 7 months ago
Most formats don’t support it. It has to be gguf format, afaik. You can usually find a conversion on huggingface. Prefer offerings by TheBloke for the detailed documentation, if nothing else.
Fisch@lemmy.ml 7 months ago
The models you have should be .gguf files right? I think those are the only ones where that’s supported
miss_brainfarts@lemmy.blahaj.zone 7 months ago
All of them are gguf, yeah