Comment on Opera is testing letting you download LLMs for local use, a first for a major browser
Bandicoot_Academic@lemmy.one 7 months ago
Intresting. But I’m curious about the performance.
A bigger LLM (mixtral) already struggles to run on my mid-range gaming PC. Trying to run an LLM that isn’t terrible on a standard laptop wouldn’t be a good experience.
tal@lemmy.today 7 months ago
I have no idea how this is set up to work technically, but most of the heavy lifting is gonna be on the GPU. I’m not sure that it matters much whether the browser is what’s pushing data to the GPU or some other package.
Bandicoot_Academic@lemmy.one 7 months ago
Most people probably don’t have a dedicated GPU and an iGPU is probably not powerfull enough to run an LLM at decent speed. Also a decent model requires like 20GB of RAM which most people don’t have.
douglasg14b@lemmy.world 7 months ago
It doesn’t just require 20GB of RAM, it requires that in VRAM. Which is a much higher barrier to entry.
Hamartiogonic@sopuli.xyz 7 months ago
But what if you have an AMD APU. Doesn’t that use your normal RAM as VRAM?
T156@lemmy.world 7 months ago
Unlikely, at least on non-nvidia chips, and even on AMD, it’s only the latest four chips that support it. Anything older isn’t going to cut it.
You also need a fairly big amount of VRAM for models like that. (4 GB is the minimum for the common kinds, which is more than typical integrated systems, or 8 GB of system memory). You can get by with system RAM, but the performance will be quite bad, since you’re either relying on the CPU, or you’ll be adding the latency from data moving between them.