Comment on Have you tried LocalGPT PrivateGPT or other similar alternatives to ChatGPT?

<- View Parent
exu@feditown.com ⁨10⁩ ⁨months⁩ ago

If you’re using llama.cpp, have a look at the GGUF models by TheBloke on huggingface. He puts approximate RAM required in the readme based on the quantisation level.

From personal experience I’d estimate 12G for 7B models based on how full RAM was with 16 gigs. For mixtral at least 32G.

source
Sort:hotnewtop