If you get just the right gguf model (read the description when you download them to get the right K-optimization or whatever it’s called) and actually use multithreading (llamacpp supports multithreading so in theory gpt4all should too), then it’s reasonably fast. I’ve achieved roughly half the speed of ChatGPT just on an overclocked 8 core amd fx.
Comment on [deleted]
Ziggurat@sh.itjust.works 11 months ago
Have you tried GPT4All gpt4all.io/index.html ? It runs on CPU so is a bit slow, but it’s a way to run various LLM locally with an plug and play, easy to use solution. That said, LLM are huge, and perform better on GPU, provided you have a GPU big enough. Here is the trap. How much do you want to spend in a GPU ?
PeterPoopshit@lemmy.world 11 months ago
kindenough@kbin.social 11 months ago
On GPU it is okay. GTX-1080 with a R5 3700X.
It has just written a 24 page tourist info booklet about the town I live in and a bunch of it is very inaccurate or outdated on the places to go. Fun and impressive anyway. Took only a few minutes.