Ram is important but it has to be vram not system ram.
Only MacBooks can use the system ram because they have an integrated GPU rather than a dedicated one.
Comment on [deleted]
gnutard@sh.itjust.works 4 months agoI thought you need tons of RAM to run LLMs? I thought the newer models needed up to 64GB RAM?
april@lemmy.world 4 months ago
PumpkinEscobar@lemmy.world 4 months ago
Taking ollama for instance, either the whole model runs in vram and compute is done on the gpu, or it runs in system ram and compute is done on the cpu. Running models on CPU is horribly slow. You won’t want to do it for large models
LM studio and others allow you to run part of the model on GPU and part on CPU, splitting memory requirements but still pretty slow.
Even the smaller 7B parameter models run pretty slow in CPU and the huge models are orders of magnitude slower
So technically more system ram will let you run some larger models but you will quickly figure out you just don’t want to do it.
Findmysec@infosec.pub 4 months ago
They do, but VRAM. Unfortunately, the cards that do have that much of memory are used by OEMs/corporations and are insanely pricey
atzanteol@sh.itjust.works 4 months ago
VRAM. Not system RAM. LLMs run best entirely on the GPU.