Ollama with Ollama WebUI is the best combo from my experience.
Comment on NVIDIA’s new AI chatbot runs locally on your PC
GenderNeutralBro@lemmy.sdf.org 7 months agoPretty much every LLM you can download already has CUDA support via PyTorch.
However, some of the easier to use frontends don’t use GPU acceleration because it’s a bit of a pain to configure across a wide range of hardware models and driver versions. IIRC GPT4All does not use GPU acceleration yet (might need outdated; I haven’t checked in a while).
If this makes local LLMs more accessible to people who are not familiar with setting up a CUDA development environment or Python venvs, that’s great news.
CeeBee@lemmy.world 7 months ago
Oha@lemmy.ohaa.xyz 7 months ago
Gpt4all somehow uses Gpu acceleration on my rx 6600xt
GenderNeutralBro@lemmy.sdf.org 7 months ago
Ooh nice. Looking at the change logs, looks like they added Vulkan acceleration back in September. Probably not as good as CUDA/Metal on supported hardware though.
Oha@lemmy.ohaa.xyz 7 months ago
getting around 44 iterations/s (or whatever that means) on my gpu
General_Effort@lemmy.world 7 months ago
I’d hope that this uses the hardware better than Pytorch. Otherwise, why the specific hardware demands? Well, it can always be marketing.
There are several alternatives that offer 1-click installers. EG in this thread:
AGPL-3.0 license: jan.ai
MIT license: ollama.com
MIT license: gpt4all.io/index.html
(There’s more.)