Ollama with Ollama WebUI is the best combo from my experience.
Comment on NVIDIA’s new AI chatbot runs locally on your PC
GenderNeutralBro@lemmy.sdf.org 10 months agoPretty much every LLM you can download already has CUDA support via PyTorch.
However, some of the easier to use frontends don’t use GPU acceleration because it’s a bit of a pain to configure across a wide range of hardware models and driver versions. IIRC GPT4All does not use GPU acceleration yet (might need outdated; I haven’t checked in a while).
If this makes local LLMs more accessible to people who are not familiar with setting up a CUDA development environment or Python venvs, that’s great news.
CeeBee@lemmy.world 10 months ago
Oha@lemmy.ohaa.xyz 10 months ago
Gpt4all somehow uses Gpu acceleration on my rx 6600xt
GenderNeutralBro@lemmy.sdf.org 10 months ago
Ooh nice. Looking at the change logs, looks like they added Vulkan acceleration back in September. Probably not as good as CUDA/Metal on supported hardware though.
Oha@lemmy.ohaa.xyz 10 months ago
getting around 44 iterations/s (or whatever that means) on my gpu
General_Effort@lemmy.world 10 months ago
I’d hope that this uses the hardware better than Pytorch. Otherwise, why the specific hardware demands? Well, it can always be marketing.
There are several alternatives that offer 1-click installers. EG in this thread:
AGPL-3.0 license: jan.ai
MIT license: ollama.com
MIT license: gpt4all.io/index.html
(There’s more.)