Comment on Have you tried LocalGPT PrivateGPT or other similar alternatives to ChatGPT?
exu@feditown.com 10 months agoIf you’re using llama.cpp, have a look at the GGUF models by TheBloke on huggingface. He puts approximate RAM required in the readme based on the quantisation level.
From personal experience I’d estimate 12G for 7B models based on how full RAM was with 16 gigs. For mixtral at least 32G.
ReallyActuallyFrankenstein@lemmynsfw.com 10 months ago
Thanks, appreciate it (I’m new to local text CPU models, I know it was a stupid question).