Comment

Comment on NVIDIA’s new AI chatbot runs locally on your PC

GenderNeutralBro@lemmy.sdf.org ⁨8⁩ ⁨months⁩ ago

Pretty much every LLM you can download already has CUDA support via PyTorch.

However, some of the easier to use frontends don’t use GPU acceleration because it’s a bit of a pain to configure across a wide range of hardware models and driver versions. IIRC GPT4All does not use GPU acceleration yet (might need outdated; I haven’t checked in a while).

If this makes local LLMs more accessible to people who are not familiar with setting up a CUDA development environment or Python venvs, that’s great news.

source

Sort:hotnew top

General_Effort@lemmy.world ⁨8⁩ ⁨months⁩ ago
I’d hope that this uses the hardware better than Pytorch. Otherwise, why the specific hardware demands? Well, it can always be marketing.

There are several alternatives that offer 1-click installers. EG in this thread:

AGPL-3.0 license: jan.ai

MIT license: ollama.com

MIT license: gpt4all.io/index.html

(There’s more.)

source
CeeBee@lemmy.world ⁨8⁩ ⁨months⁩ ago
Ollama with Ollama WebUI is the best combo from my experience.

source
Oha@lemmy.ohaa.xyz ⁨8⁩ ⁨months⁩ ago
Gpt4all somehow uses Gpu acceleration on my rx 6600xt

source
- GenderNeutralBro@lemmy.sdf.org ⁨8⁩ ⁨months⁩ ago
  Ooh nice. Looking at the change logs, looks like they added Vulkan acceleration back in September. Probably not as good as CUDA/Metal on supported hardware though.
  
  source
  - Oha@lemmy.ohaa.xyz ⁨8⁩ ⁨months⁩ ago
    getting around 44 iterations/s (or whatever that means) on my gpu
    
    source